Merge branch 'v1' into 64inlinemsgid

readthedocs/misc/changelog.rst

@@ -13,6 +13,49 @@ it can take advantage of new goodies!
 
 .. contents:: List of All Versions
 
+New Layer and housekeeping (v1.28)
+==================================
+
++------------------------+
+| Scheme layer used: 155 |
++------------------------+
+
+Plenty of stale issues closed, as well as improvements for some others.
+
+Additions
+~~~~~~~~~
+
+* New ``entity_cache_limit`` parameter in the ``TelegramClient`` constructor.
+  This should help a bit in keeping memory usage in check.
+
+Enhancements
+~~~~~~~~~~~~
+
+* ``progress_callback`` is now called when dealing with albums. See the
+  documentation on `client.send_file() <telethon.client.uploads.UploadMethods.send_file>`
+  for details.
+* Update state and entities are now periodically saved, so that the information
+  isn't lost in the case of crash or unexpected script terminations. You should
+  still be calling ``disconnect`` or using the context-manager, though.
+* The client should no longer unnecessarily call ``get_me`` every time it's started.
+
+Bug fixes
+~~~~~~~~~
+
+* Messages obtained via raw API could not be used in ``forward_messages``.
+* ``force_sms`` and ``sign_up`` have been deprecated. See `issue 4050`_ for details.
+  It is no longer possible for third-party applications, such as those made with
+  Telethon, to use those features.
+* ``events.ChatAction`` should now work in more cases in groups with hidden members.
+* Errors that occur at the connection level should now be properly propagated, so that
+  you can actually have a chance to handle them.
+* Update handling should be more resilient.
+* ``PhoneCodeExpiredError`` will correctly clear the stored hash if it occurs in ``sign_in``.
+
+
+.. _issue 4050: https://github.com/LonamiWebs/Telethon/issues/4050
+
+
 New Layer and some Bug fixes (v1.27)
 ====================================
 

readthedocs/quick-references/faq.rst

@@ -178,6 +178,69 @@ won't do unnecessary work unless you need to:
         sender = await event.get_sender()
 
 
+What does "Server sent a very new message with ID" mean?
+========================================================
+
+You may also see this error as "Server sent a very old message with ID".
+
+This is a security feature from Telethon that cannot be disabled and is
+meant to protect you against replay attacks.
+
+When this message is incorrectly reported as a "bug",
+the most common patterns seem to be:
+
+* Your system time is incorrect.
+* The proxy you're using may be interfering somehow.
+* The Telethon session is being used or has been used from somewhere else.
+  Make sure that you created the session from Telethon, and are not using the
+  same session anywhere else. If you need to use the same account from
+  multiple places, login and use a different session for each place you need.
+
+
+What does "Server replied with a wrong session ID" mean?
+========================================================
+
+This is a security feature from Telethon that cannot be disabled and is
+meant to protect you against unwanted session reuse.
+
+When this message is reported as a "bug", the most common patterns seem to be:
+
+* The proxy you're using may be interfering somehow.
+* The Telethon session is being used or has been used from somewhere else.
+  Make sure that you created the session from Telethon, and are not using the
+  same session anywhere else. If you need to use the same account from
+  multiple places, login and use a different session for each place you need.
+* You may be using multiple connections to the Telegram server, which seems
+  to confuse Telegram.
+
+Most of the time it should be safe to ignore this warning. If the library
+still doesn't behave correctly, make sure to check if any of the above bullet
+points applies in your case and try to work around it.
+
+If the issue persists and there is a way to reliably reproduce this error,
+please add a comment with any additional details you can provide to
+`issue 3759`_, and perhaps some additional investigation can be done
+(but it's unlikely, as Telegram *is* sending unexpected data).
+
+
+What does "Could not find a matching Constructor ID for the TLObject" mean?
+===========================================================================
+
+Telegram uses "layers", which you can think of as "versions" of the API they
+offer. When Telethon reads responses that the Telegram servers send, these
+need to be deserialized (into what Telethon calls "TLObjects").
+
+Every Telethon version understands a single Telegram layer. When Telethon
+connects to Telegram, both agree on the layer to use. If the layers don't
+match, Telegram may send certain objects which Telethon no longer understands.
+
+When this message is reported as a "bug", the most common patterns seem to be
+that he Telethon session is being used or has been used from somewhere else.
+Make sure that you created the session from Telethon, and are not using the
+same session anywhere else. If you need to use the same account from
+multiple places, login and use a different session for each place you need.
+
+
 What does "bases ChatGetter" mean?
 ==================================
 
@@ -268,4 +331,5 @@ file and run that, or use the normal ``python`` interpreter.
 .. _logging: https://docs.python.org/3/library/logging.html
 .. _@SpamBot: https://t.me/SpamBot
 .. _issue 297: https://github.com/LonamiWebs/Telethon/issues/297
+.. _issue 3759: https://github.com/LonamiWebs/Telethon/issues/3759
 .. _quart_login.py: https://github.com/LonamiWebs/Telethon/tree/v1/telethon_examples#quart_loginpy

telethon/_updates/entitycache.py

@@ -9,15 +9,17 @@ class EntityCache:
         self,
         hash_map: dict = _sentinel,
         self_id: int = None,
-        self_bot: bool = False
+        self_bot: bool = None
     ):
         self.hash_map = {} if hash_map is _sentinel else hash_map
         self.self_id = self_id
         self.self_bot = self_bot
 
-    def set_self_user(self, id, bot):
+    def set_self_user(self, id, bot, hash):
         self.self_id = id
         self.self_bot = bot
+        if hash:
+            self.hash_map[id] = (hash, EntityType.BOT if bot else EntityType.USER)
 
     def get(self, id):
         try:
@@ -52,3 +54,9 @@ class EntityCache:
 
     def put(self, entity):
         self.hash_map[entity.id] = (entity.hash, entity.ty)
+
+    def retain(self, filter):
+        self.hash_map = {k: v for k, v in self.hash_map.items() if filter(k)}
+
+    def __len__(self):
+        return len(self.hash_map)

telethon/_updates/messagebox.py

@@ -232,7 +232,7 @@ class MessageBox:
             self.map[ENTRY_SECRET] = State(pts=session_state.qts, deadline=deadline)
         self.map.update((s.channel_id, State(pts=s.pts, deadline=deadline)) for s in channel_states)
 
-        self.date = datetime.datetime.fromtimestamp(session_state.date).replace(tzinfo=datetime.timezone.utc)
+        self.date = datetime.datetime.fromtimestamp(session_state.date, tz=datetime.timezone.utc)
         self.seq = session_state.seq
         self.next_deadline = ENTRY_ACCOUNT
 

telethon/client/auth.py

@@ -352,7 +352,12 @@ class AuthMethods:
                 'and a password only if an RPCError was raised before.'
             )
 
-        result = await self(request)
+        try:
+            result = await self(request)
+        except errors.PhoneCodeExpiredError:
+            self._phone_code_hash.pop(phone, None)
+            raise
+
         if isinstance(result, types.auth.AuthorizationSignUpRequired):
             # Emulate pre-layer 104 behaviour
             self._tos = result.terms_of_service
@@ -380,8 +385,7 @@ class AuthMethods:
 
         Returns the input user parameter.
         """
-        self._bot = bool(user.bot)
+        self._mb_entity_cache.set_self_user(user.id, user.bot, user.access_hash)
-        self._self_input_peer = utils.get_input_peer(user, allow_self=False)
         self._authorized = True
 
         state = await self(functions.updates.GetStateRequest())
@@ -531,8 +535,7 @@ class AuthMethods:
         except errors.RPCError:
             return False
 
-        self._bot = None
+        self._mb_entity_cache.set_self_user(None, None, None)
-        self._self_input_peer = None
         self._authorized = False
 
         await self.disconnect()

telethon/client/telegrambaseclient.py

@@ -10,7 +10,6 @@ import datetime
 
 from .. import version, helpers, __name__ as __base_name__
 from ..crypto import rsa
-from ..entitycache import EntityCache
 from ..extensions import markdown
 from ..network import MTProtoSender, Connection, ConnectionTcpFull, TcpMTProxy
 from ..sessions import Session, SQLiteSession, MemorySession
@@ -209,6 +208,20 @@ class TelegramBaseClient(abc.ABC):
             so event handlers, conversations, and QR login will not work.
             However, certain scripts don't need updates, so this will reduce
             the amount of bandwidth used.
+
+        entity_cache_limit (`int`, optional):
+            How many users, chats and channels to keep in the in-memory cache
+            at most. This limit is checked against when processing updates.
+
+            When this limit is reached or exceeded, all entities that are not
+            required for update handling will be flushed to the session file.
+
+            Note that this implies that there is a lower bound to the amount
+            of entities that must be kept in memory.
+
+            Setting this limit too low will cause the library to attempt to
+            flush entities to the session file even if no entities can be
+            removed from the in-memory cache, which will degrade performance.
     """
 
     # Current TelegramClient version
@@ -246,7 +259,8 @@ class TelegramBaseClient(abc.ABC):
             loop: asyncio.AbstractEventLoop = None,
             base_logger: typing.Union[str, logging.Logger] = None,
             receive_updates: bool = True,
-            catch_up: bool = False
+            catch_up: bool = False,
+            entity_cache_limit: int = 5000
     ):
         if not api_id or not api_hash:
             raise ValueError(
@@ -300,7 +314,7 @@ class TelegramBaseClient(abc.ABC):
         self.flood_sleep_threshold = flood_sleep_threshold
 
         # TODO Use AsyncClassWrapper(session)
-        # ChatGetter and SenderGetter can use the in-memory _entity_cache
+        # ChatGetter and SenderGetter can use the in-memory _mb_entity_cache
         # to avoid network access and the need for await in session files.
         #
         # The session files only wants the entities to persist
@@ -308,7 +322,6 @@ class TelegramBaseClient(abc.ABC):
         # TODO Session should probably return all cached
         #      info of entities, not just the input versions
         self.session = session
-        self._entity_cache = EntityCache()
         self.api_id = int(api_id)
         self.api_hash = api_hash
 
@@ -422,10 +435,6 @@ class TelegramBaseClient(abc.ABC):
         self._phone = None
         self._tos = None
 
-        # Sometimes we need to know who we are, cache the self peer
-        self._self_input_peer = None
-        self._bot = None
-
         # A place to store if channels are a megagroup or not (see `edit_admin`)
         self._megagroup_cache = {}
 
@@ -433,8 +442,8 @@ class TelegramBaseClient(abc.ABC):
         self._catch_up = catch_up
         self._updates_queue = asyncio.Queue()
         self._message_box = MessageBox(self._log['messagebox'])
-        # This entity cache is tailored for the messagebox and is not used for absolutely everything like _entity_cache
         self._mb_entity_cache = MbEntityCache()  # required for proper update handling (to know when to getDifference)
+        self._entity_cache_limit = entity_cache_limit
 
         self._sender = MTProtoSender(
             self.session.auth_key,
@@ -540,6 +549,14 @@ class TelegramBaseClient(abc.ABC):
         self.session.auth_key = self._sender.auth_key
         self.session.save()
 
+        try:
+            # See comment when saving entities to understand this hack
+            self_id = self.session.get_input_entity(0).access_hash
+            self_user = self.session.get_input_entity(self_id)
+            self._mb_entity_cache.set_self_user(self_id, None, self_user.access_hash)
+        except ValueError:
+            pass
+
         if self._catch_up:
             ss = SessionState(0, 0, False, 0, 0, 0, 0, None)
             cs = []
@@ -654,6 +671,24 @@ class TelegramBaseClient(abc.ABC):
             else:
                 connection._proxy = proxy
 
+    def _save_states_and_entities(self: 'TelegramClient'):
+        entities = self._mb_entity_cache.get_all_entities()
+
+        # Piggy-back on an arbitrary TL type with users and chats so the session can understand to read the entities.
+        # It doesn't matter if we put users in the list of chats.
+        self.session.process_entities(types.contacts.ResolvedPeer(None, [e._as_input_peer() for e in entities], []))
+
+        # As a hack to not need to change the session files, save ourselves with ``id=0`` and ``access_hash`` of our ``id``.
+        # This way it is possible to determine our own ID by querying for 0. However, whether we're a bot is not saved.
+        if self._mb_entity_cache.self_id:
+            self.session.process_entities(types.contacts.ResolvedPeer(None, [types.InputPeerUser(0, self._mb_entity_cache.self_id)], []))
+
+        ss, cs = self._message_box.session_state()
+        self.session.set_update_state(0, types.updates.State(**ss, unread_count=0))
+        now = datetime.datetime.now()  # any datetime works; channels don't need it
+        for channel_id, pts in cs.items():
+            self.session.set_update_state(channel_id, types.updates.State(pts, 0, now, 0, unread_count=0))
+
     async def _disconnect_coro(self: 'TelegramClient'):
         if self.session is None:
             return  # already logged out and disconnected
@@ -684,17 +719,7 @@ class TelegramBaseClient(abc.ABC):
             await asyncio.wait(self._event_handler_tasks)
             self._event_handler_tasks.clear()
 
-        entities = self._mb_entity_cache.get_all_entities()
+        self._save_states_and_entities()
-
-        # Piggy-back on an arbitrary TL type with users and chats so the session can understand to read the entities.
-        # It doesn't matter if we put users in the list of chats.
-        self.session.process_entities(types.contacts.ResolvedPeer(None, [e._as_input_peer() for e in entities], []))
-
-        ss, cs = self._message_box.session_state()
-        self.session.set_update_state(0, types.updates.State(**ss, unread_count=0))
-        now = datetime.datetime.now()  # any datetime works; channels don't need it
-        for channel_id, pts in cs.items():
-            self.session.set_update_state(channel_id, types.updates.State(pts, 0, now, 0, unread_count=0))
 
         self.session.close()
 

telethon/client/updates.py

@@ -7,6 +7,7 @@ import time
 import traceback
 import typing
 import logging
+import warnings
 from collections import deque
 
 from .. import events, utils, errors
@@ -14,6 +15,7 @@ from ..events.common import EventBuilder, EventCommon
 from ..tl import types, functions
 from .._updates import GapError, PrematureEndReason
 from ..helpers import get_running_loop
+from ..version import __version__
 
 
 if typing.TYPE_CHECKING:
@@ -280,6 +282,24 @@ class UpdateMethods:
 
                     continue
 
+                if len(self._mb_entity_cache) >= self._entity_cache_limit:
+                    self._log[__name__].info(
+                        'In-memory entity cache limit reached (%s/%s), flushing to session',
+                        len(self._mb_entity_cache),
+                        self._entity_cache_limit
+                    )
+                    self._save_states_and_entities()
+                    self._mb_entity_cache.retain(lambda id: id == self._mb_entity_cache.self_id or id in self._message_box.map)
+                    if len(self._mb_entity_cache) >= self._entity_cache_limit:
+                        warnings.warn('in-memory entities exceed entity_cache_limit after flushing; consider setting a larger limit')
+
+                    self._log[__name__].info(
+                        'In-memory entity cache at %s/%s after flushing to session',
+                        len(self._mb_entity_cache),
+                        self._entity_cache_limit
+                    )
+
+
                 get_diff = self._message_box.get_difference()
                 if get_diff:
                     self._log[__name__].debug('Getting difference for account updates')
@@ -419,7 +439,7 @@ class UpdateMethods:
         except asyncio.CancelledError:
             pass
         except Exception as e:
-            self._log[__name__].exception('Fatal error handling updates (this is a bug in Telethon, please report it)')
+            self._log[__name__].exception(f'Fatal error handling updates (this is a bug in Telethon v{__version__}, please report it)')
             self._updates_error = e
             await self.disconnect()
 
@@ -467,16 +487,18 @@ class UpdateMethods:
             # inserted because this is a rather expensive operation
             # (default's sqlite3 takes ~0.1s to commit changes). Do
             # it every minute instead. No-op if there's nothing new.
+            self._save_states_and_entities()
+
             self.session.save()
 
     async def _dispatch_update(self: 'TelegramClient', update):
         # TODO only used for AlbumHack, and MessageBox is not really designed for this
         others = None
 
-        if not self._self_input_peer:
+        if not self._mb_entity_cache.self_id:
             # Some updates require our own ID, so we must make sure
             # that the event builder has offline access to it. Calling
-            # `get_me()` will cache it under `self._self_input_peer`.
+            # `get_me()` will cache it under `self._mb_entity_cache`.
             #
             # It will return `None` if we haven't logged in yet which is
             # fine, we will just retry next time anyway.

telethon/client/uploads.py

@@ -352,6 +352,11 @@ class UploadMethods:
         # First check if the user passed an iterable, in which case
         # we may want to send grouped.
         if utils.is_list_like(file):
+            sent_count = 0
+            used_callback = None if not progress_callback else (
+                lambda s, t: progress_callback(sent_count + s, len(file))
+            )
+
             if utils.is_list_like(caption):
                 captions = caption
             else:
@@ -361,25 +366,14 @@ class UploadMethods:
             while file:
                 result += await self._send_album(
                     entity, file[:10], caption=captions[:10],
-                    progress_callback=progress_callback, reply_to=reply_to,
+                    progress_callback=used_callback, reply_to=reply_to,
                     parse_mode=parse_mode, silent=silent, schedule=schedule,
                     supports_streaming=supports_streaming, clear_draft=clear_draft,
                     force_document=force_document, background=background,
                 )
                 file = file[10:]
                 captions = captions[10:]
-
+                sent_count += 10
-            for doc, cap in zip(file, captions):
-                result.append(await self.send_file(
-                    entity, doc, allow_cache=allow_cache,
-                    caption=cap, force_document=force_document,
-                    progress_callback=progress_callback, reply_to=reply_to,
-                    attributes=attributes, thumb=thumb, voice_note=voice_note,
-                    video_note=video_note, buttons=buttons, silent=silent,
-                    supports_streaming=supports_streaming, schedule=schedule,
-                    clear_draft=clear_draft, background=background,
-                    **kwargs
-                ))
 
             return result
 
@@ -436,16 +430,22 @@ class UploadMethods:
 
         reply_to = utils.get_message_id(reply_to)
 
+        used_callback = None if not progress_callback else (
+            # use an integer when sent matches total, to easily determine a file has been fully sent
+            lambda s, t: progress_callback(sent_count + 1 if s == t else sent_count + s / t, len(files))
+        )
+
         # Need to upload the media first, but only if they're not cached yet
         media = []
-        for file in files:
+        for sent_count, file in enumerate(files):
             # Albums want :tl:`InputMedia` which, in theory, includes
             # :tl:`InputMediaUploadedPhoto`. However using that will
             # make it `raise MediaInvalidError`, so we need to upload
             # it as media and then convert that to :tl:`InputMediaPhoto`.
             fh, fm, _ = await self._file_to_media(
                 file, supports_streaming=supports_streaming,
-                force_document=force_document, ttl=ttl)
+                force_document=force_document, ttl=ttl,
+                progress_callback=used_callback)
             if isinstance(fm, (types.InputMediaUploadedPhoto, types.InputMediaPhotoExternal)):
                 r = await self(functions.messages.UploadMediaRequest(
                     entity, media=fm
@@ -546,6 +546,13 @@ class UploadMethods:
                 A callback function accepting two parameters:
                 ``(sent bytes, total)``.
 
+                When sending an album, the callback will receive a number
+                between 0 and the amount of files as the "sent" parameter,
+                and the amount of files as the "total". Note that the first
+                parameter will be a floating point number to indicate progress
+                within a file (e.g. ``2.5`` means it has sent 50% of the third
+                file, because it's between 2 and 3).
+
         Returns
             :tl:`InputFileBig` if the file size is larger than 10MB,
             `InputSizedFile <telethon.tl.custom.inputsizedfile.InputSizedFile>`

telethon/client/users.py

@@ -72,7 +72,6 @@ class UserMethods:
                             results.append(None)
                             continue
                         self.session.process_entities(result)
-                        self._entity_cache.add(result)
                         exceptions.append(None)
                         results.append(result)
                         request_index += 1
@@ -83,7 +82,6 @@ class UserMethods:
                 else:
                     result = await future
                     self.session.process_entities(result)
-                    self._entity_cache.add(result)
                     return result
             except (errors.ServerError, errors.RpcCallFailError,
                     errors.RpcMcgetFailError, errors.InterdcCallErrorError,
@@ -154,20 +152,17 @@ class UserMethods:
                 me = await client.get_me()
                 print(me.username)
         """
-        if input_peer and self._self_input_peer:
+        if input_peer and self._mb_entity_cache.self_id:
-            return self._self_input_peer
+            return self._mb_entity_cache.get(self._mb_entity_cache.self_id)._as_input_peer()
 
         try:
             me = (await self(
                 functions.users.GetUsersRequest([types.InputUserSelf()])))[0]
 
-            self._bot = me.bot
+            if not self._mb_entity_cache.self_id:
-            if not self._self_input_peer:
+                self._mb_entity_cache.set_self_user(me.id, me.bot, me.access_hash)
-                self._self_input_peer = utils.get_input_peer(
-                    me, allow_self=False
-                )
 
-            return self._self_input_peer if input_peer else me
+            return utils.get_input_peer(me, allow_self=False) if input_peer else me
         except errors.UnauthorizedError:
             return None
 
@@ -179,7 +174,7 @@ class UserMethods:
         This property is used in every update, and some like `updateLoginToken`
         occur prior to login, so it gracefully handles when no ID is known yet.
         """
-        return self._self_input_peer.user_id if self._self_input_peer else None
+        return self._mb_entity_cache.self_id
 
     async def is_bot(self: 'TelegramClient') -> bool:
         """
@@ -193,10 +188,10 @@ class UserMethods:
                 else:
                     print('Hello')
         """
-        if self._bot is None:
+        if self._mb_entity_cache.self_bot is None:
-            self._bot = (await self.get_me()).bot
+            await self.get_me(input_peer=True)
 
-        return self._bot
+        return self._mb_entity_cache.self_bot
 
     async def is_user_authorized(self: 'TelegramClient') -> bool:
         """
@@ -417,8 +412,8 @@ class UserMethods:
         try:
             # 0x2d45687 == crc32(b'Peer')
             if isinstance(peer, int) or peer.SUBCLASS_OF_ID == 0x2d45687:
-                return self._entity_cache[peer]
+                return self._mb_entity_cache.get(utils.get_peer_id(peer, add_mark=False))._as_input_peer()
-        except (AttributeError, KeyError):
+        except AttributeError:
             pass
 
         # Then come known strings that take precedence

telethon/entitycache.py

@@ -1,147 +0,0 @@
-import inspect
-import itertools
-
-from . import utils
-from .tl import types
-
-# Which updates have the following fields?
-_has_field = {
-    ('user_id', int): [],
-    ('chat_id', int): [],
-    ('channel_id', int): [],
-    ('peer', 'TypePeer'): [],
-    ('peer', 'TypeDialogPeer'): [],
-    ('message', 'TypeMessage'): [],
-}
-
-# Note: We don't bother checking for some rare:
-# * `UpdateChatParticipantAdd.inviter_id` integer.
-# * `UpdateNotifySettings.peer` dialog peer.
-# * `UpdatePinnedDialogs.order` list of dialog peers.
-# * `UpdateReadMessagesContents.messages` list of messages.
-# * `UpdateChatParticipants.participants` list of participants.
-#
-# There are also some uninteresting `update.message` of type string.
-
-
-def _fill():
-    for name in dir(types):
-        update = getattr(types, name)
-        if getattr(update, 'SUBCLASS_OF_ID', None) == 0x9f89304e:
-            cid = update.CONSTRUCTOR_ID
-            sig = inspect.signature(update.__init__)
-            for param in sig.parameters.values():
-                vec = _has_field.get((param.name, param.annotation))
-                if vec is not None:
-                    vec.append(cid)
-
-    # Future-proof check: if the documentation format ever changes
-    # then we won't be able to pick the update types we are interested
-    # in, so we must make sure we have at least an update for each field
-    # which likely means we are doing it right.
-    if not all(_has_field.values()):
-        raise RuntimeError('FIXME: Did the init signature or updates change?')
-
-
-# We use a function to avoid cluttering the globals (with name/update/cid/doc)
-_fill()
-
-
-class EntityCache:
-    """
-    In-memory input entity cache, defaultdict-like behaviour.
-    """
-    def add(self, entities):
-        """
-        Adds the given entities to the cache, if they weren't saved before.
-        """
-        if not utils.is_list_like(entities):
-            # Invariant: all "chats" and "users" are always iterables,
-            # and "user" never is (so we wrap it inside a list).
-            entities = itertools.chain(
-                getattr(entities, 'chats', []),
-                getattr(entities, 'users', []),
-                (hasattr(entities, 'user') and [entities.user]) or []
-            )
-
-        for entity in entities:
-            try:
-                pid = utils.get_peer_id(entity)
-                if pid not in self.__dict__:
-                    # Note: `get_input_peer` already checks for `access_hash`
-                    self.__dict__[pid] = utils.get_input_peer(entity)
-            except TypeError:
-                pass
-
-    def __getitem__(self, item):
-        """
-        Gets the corresponding :tl:`InputPeer` for the given ID or peer,
-        or raises ``KeyError`` on any error (i.e. cannot be found).
-        """
-        if not isinstance(item, int) or item < 0:
-            try:
-                return self.__dict__[utils.get_peer_id(item)]
-            except TypeError:
-                raise KeyError('Invalid key will not have entity') from None
-
-        for cls in (types.PeerUser, types.PeerChat, types.PeerChannel):
-            result = self.__dict__.get(utils.get_peer_id(cls(item)))
-            if result:
-                return result
-
-        raise KeyError('No cached entity for the given key')
-
-    def clear(self):
-        """
-        Clear the entity cache.
-        """
-        self.__dict__.clear()
-
-    def ensure_cached(
-            self,
-            update,
-            has_user_id=frozenset(_has_field[('user_id', int)]),
-            has_chat_id=frozenset(_has_field[('chat_id', int)]),
-            has_channel_id=frozenset(_has_field[('channel_id', int)]),
-            has_peer=frozenset(_has_field[('peer', 'TypePeer')] + _has_field[('peer', 'TypeDialogPeer')]),
-            has_message=frozenset(_has_field[('message', 'TypeMessage')])
-    ):
-        """
-        Ensures that all the relevant entities in the given update are cached.
-        """
-        # This method is called pretty often and we want it to have the lowest
-        # overhead possible. For that, we avoid `isinstance` and constantly
-        # getting attributes out of `types.` by "caching" the constructor IDs
-        # in sets inside the arguments, and using local variables.
-        dct = self.__dict__
-        cid = update.CONSTRUCTOR_ID
-        if cid in has_user_id and \
-                update.user_id not in dct:
-            return False
-
-        if cid in has_chat_id and \
-                utils.get_peer_id(types.PeerChat(update.chat_id)) not in dct:
-            return False
-
-        if cid in has_channel_id and \
-                utils.get_peer_id(types.PeerChannel(update.channel_id)) not in dct:
-            return False
-
-        if cid in has_peer and \
-                utils.get_peer_id(update.peer) not in dct:
-            return False
-
-        if cid in has_message:
-            x = update.message
-            y = getattr(x, 'peer_id', None)  # handle MessageEmpty
-            if y and utils.get_peer_id(y) not in dct:
-                return False
-
-            y = getattr(x, 'from_id', None)
-            if y and utils.get_peer_id(y) not in dct:
-                return False
-
-            # We don't quite worry about entities anywhere else.
-            # This is enough.
-
-        return True

telethon/errors/common.py

@@ -19,8 +19,8 @@ class TypeNotFoundError(Exception):
     def __init__(self, invalid_constructor_id, remaining):
         super().__init__(
             'Could not find a matching Constructor ID for the TLObject '
-            'that was supposed to be read with ID {:08x}. Most likely, '
+            'that was supposed to be read with ID {:08x}. See the FAQ '
-            'a TLObject was trying to be read when it should not be read. '
+            'for more details. '
             'Remaining bytes: {!r}'.format(invalid_constructor_id, remaining))
 
         self.invalid_constructor_id = invalid_constructor_id

telethon/events/album.py

@@ -160,7 +160,7 @@ class Album(EventBuilder):
         def _set_client(self, client):
             super()._set_client(client)
             self._sender, self._input_sender = utils._get_entity_pair(
-                self.sender_id, self._entities, client._entity_cache)
+                self.sender_id, self._entities, client._mb_entity_cache)
 
             for msg in self.messages:
                 msg._finish_init(client, self._entities, None)

telethon/events/callbackquery.py

@@ -151,7 +151,7 @@ class CallbackQuery(EventBuilder):
         def _set_client(self, client):
             super()._set_client(client)
             self._sender, self._input_sender = utils._get_entity_pair(
-                self.sender_id, self._entities, client._entity_cache)
+                self.sender_id, self._entities, client._mb_entity_cache)
 
         @property
         def id(self):
@@ -208,8 +208,9 @@ class CallbackQuery(EventBuilder):
             if not getattr(self._input_sender, 'access_hash', True):
                 # getattr with True to handle the InputPeerSelf() case
                 try:
-                    self._input_sender = self._client._entity_cache[self._sender_id]
+                    self._input_sender = self._client._mb_entity_cache.get(
-                except KeyError:
+                        utils.resolve_id(self._sender_id)[0])._as_input_peer()
+                except AttributeError:
                     m = await self.get_message()
                     if m:
                         self._sender = m._sender

telethon/events/chataction.py

@@ -425,9 +425,10 @@ class ChatAction(EventBuilder):
 
                     # If missing, try from the entity cache
                     try:
-                        self._input_users.append(self._client._entity_cache[user_id])
+                        self._input_users.append(self._client._mb_entity_cache.get(
+                            utils.resolve_id(user_id)[0])._as_input_peer())
                         continue
-                    except KeyError:
+                    except AttributeError:
                         pass
 
             return self._input_users or []

telethon/events/common.py

@@ -154,7 +154,7 @@ class EventCommon(ChatGetter, abc.ABC):
         self._client = client
         if self._chat_peer:
             self._chat, self._input_chat = utils._get_entity_pair(
-                self.chat_id, self._entities, client._entity_cache)
+                self.chat_id, self._entities, client._mb_entity_cache)
         else:
             self._chat = self._input_chat = None
 

telethon/events/inlinequery.py

@@ -99,7 +99,7 @@ class InlineQuery(EventBuilder):
         def _set_client(self, client):
             super()._set_client(client)
             self._sender, self._input_sender = utils._get_entity_pair(
-                self.sender_id, self._entities, client._entity_cache)
+                self.sender_id, self._entities, client._mb_entity_cache)
 
         @property
         def id(self):

telethon/events/userupdate.py

@@ -95,7 +95,7 @@ class UserUpdate(EventBuilder):
         def _set_client(self, client):
             super()._set_client(client)
             self._sender, self._input_sender = utils._get_entity_pair(
-                self.sender_id, self._entities, client._entity_cache)
+                self.sender_id, self._entities, client._mb_entity_cache)
 
         @property
         def user(self):

telethon/network/mtprotostate.py

@@ -176,7 +176,7 @@ class MTProtoState:
         reader = BinaryReader(body)
         reader.read_long()  # remote_salt
         if reader.read_long() != self.id:
-            raise SecurityError('Server replied with a wrong session ID')
+            raise SecurityError('Server replied with a wrong session ID (see FAQ for details)')
 
         remote_msg_id = reader.read_long()
 
@@ -208,12 +208,12 @@ class MTProtoState:
             time_delta = now - remote_msg_time
 
             if time_delta > MSG_TOO_OLD_DELTA:
-                self._log.warning('Server sent a very old message with ID %d, ignoring', remote_msg_id)
+                self._log.warning('Server sent a very old message with ID %d, ignoring (see FAQ for details)', remote_msg_id)
                 self._count_ignored()
                 return None
 
             if -time_delta > MSG_TOO_NEW_DELTA:
-                self._log.warning('Server sent a very new message with ID %d, ignoring', remote_msg_id)
+                self._log.warning('Server sent a very new message with ID %d, ignoring (see FAQ for details)', remote_msg_id)
                 self._count_ignored()
                 return None
 

telethon/tl/custom/chatgetter.py

@@ -66,8 +66,9 @@ class ChatGetter(abc.ABC):
         """
         if self._input_chat is None and self._chat_peer and self._client:
             try:
-                self._input_chat = self._client._entity_cache[self._chat_peer]
+                self._input_chat = self._client._mb_entity_cache.get(
-            except KeyError:
+                        utils.get_peer_id(self._chat_peer, add_mark=False))._as_input_peer()
+            except AttributeError:
                 pass
 
         return self._input_chat

telethon/tl/custom/draft.py

@@ -5,7 +5,7 @@ from ..functions.messages import SaveDraftRequest
 from ..types import DraftMessage
 from ...errors import RPCError
 from ...extensions import markdown
-from ...utils import get_input_peer, get_peer
+from ...utils import get_input_peer, get_peer, get_peer_id
 
 
 class Draft:
@@ -53,8 +53,9 @@ class Draft:
         """
         if not self._input_entity:
             try:
-                self._input_entity = self._client._entity_cache[self._peer]
+                self._input_entity = self._client._mb_entity_cache.get(
-            except KeyError:
+                        get_peer_id(self._peer, add_mark=False))._as_input_peer()
+            except AttributeError:
                 pass
 
         return self._input_entity

telethon/tl/custom/forward.py

@@ -36,12 +36,12 @@ class Forward(ChatGetter, SenderGetter):
             if ty == helpers._EntityType.USER:
                 sender_id = utils.get_peer_id(original.from_id)
                 sender, input_sender = utils._get_entity_pair(
-                    sender_id, entities, client._entity_cache)
+                    sender_id, entities, client._mb_entity_cache)
 
             elif ty in (helpers._EntityType.CHAT, helpers._EntityType.CHANNEL):
                 peer = original.from_id
                 chat, input_chat = utils._get_entity_pair(
-                    utils.get_peer_id(peer), entities, client._entity_cache)
+                    utils.get_peer_id(peer), entities, client._mb_entity_cache)
 
         # This call resets the client
         ChatGetter.__init__(self, peer, chat=chat, input_chat=input_chat)

telethon/tl/custom/message.py

@@ -285,7 +285,7 @@ class Message(ChatGetter, SenderGetter, TLObject):
         if self.peer_id == types.PeerUser(client._self_id) and not self.fwd_from:
             self.out = True
 
-        cache = client._entity_cache
+        cache = client._mb_entity_cache
 
         self._sender, self._input_sender = utils._get_entity_pair(
             self.sender_id, entities, cache)
@@ -1138,8 +1138,9 @@ class Message(ChatGetter, SenderGetter, TLObject):
                         return bot
                     else:
                         try:
-                            return self._client._entity_cache[self.via_bot_id]
+                            return self._client._mb_entity_cache.get(
-                        except KeyError:
+                                utils.resolve_id(self.via_bot_id)[0])._as_input_peer()
+                        except AttributeError:
                             raise ValueError('No input sender') from None
 
     def _document_by_attribute(self, kind, condition=None):

telethon/tl/custom/sendergetter.py

@@ -1,5 +1,7 @@
 import abc
 
+from ... import utils
+
 
 class SenderGetter(abc.ABC):
     """
@@ -69,9 +71,9 @@ class SenderGetter(abc.ABC):
         """
         if self._input_sender is None and self._sender_id and self._client:
             try:
-                self._input_sender = \
+                self._input_sender = self._client._mb_entity_cache.get(
-                    self._client._entity_cache[self._sender_id]
+                        utils.resolve_id(self._sender_id)[0])._as_input_peer()
-            except KeyError:
+            except AttributeError:
                 pass
         return self._input_sender
 

telethon/utils.py

@@ -583,11 +583,14 @@ def _get_entity_pair(entity_id, entities, cache,
     """
     Returns ``(entity, input_entity)`` for the given entity ID.
     """
+    if not entity_id:
+        return None, None
+
     entity = entities.get(entity_id)
     try:
-        input_entity = cache[entity_id]
+        input_entity = cache.get(resolve_id(entity_id)[0])._as_input_peer()
-    except KeyError:
+    except AttributeError:
-        # KeyError is unlikely, so another TypeError won't hurt
+        # AttributeError is unlikely, so another TypeError won't hurt
         try:
             input_entity = get_input_peer(entity)
         except TypeError: