Update to layer 211

There no plans for this to ever be non-experimental in v1.

optional-requirements.txt

@@ -3,3 +3,4 @@ pysocks
 python-socks[asyncio]
 hachoir
 pillow
+isal

readthedocs/index.rst

@@ -103,7 +103,6 @@ You can also use the menu on the left to quickly skip over sections.
     :caption: Miscellaneous
 
     misc/changelog
-    misc/wall-of-shame.rst
     misc/compatibility-and-convenience
 
 .. toctree::

readthedocs/misc/changelog.rst

@@ -13,6 +13,85 @@ it can take advantage of new goodies!
 
 .. contents:: List of All Versions
 
+New layer (v1.40)
+=================
+
++------------------------+
+| Scheme layer used: 201 |
++------------------------+
+
+`View new and changed raw API methods <https://diff.telethon.dev/?from=199&to=201>`__.
+
+Additions
+~~~~~~~~~
+
+* ``send_as`` and ``effect`` added to ``send_message`` and related methods.
+* :tl:`MessageMediaGeoLive` is now recognized for auto-input conversion.
+
+Enhancements
+~~~~~~~~~~~~
+
+* Improved wording when using a likely unintended session file.
+* Improved behaviour for matching Markdown links.
+* A truly clean update-state is now fetched upon login. This was most notably important for bots.
+* Time offset is now updated more reliably after connecting. This should fix legitimate "message too old/new" issues.
+
+Bug fixes
+~~~~~~~~~
+
+* :tl:`ChannelParticipantLeft` is now skipped in ``iter_participants``.
+* ``spoiler`` flag was lost on :tl:`MessageMediaPhoto` auto-input conversion.
+* :tl:`KeyboardButtonCopy` is now recognized as an inline button.
+* Downloading web-documents should now work again. Note that this still fetches the file from the original server.
+
+
+New layer (v1.39)
+=================
+
++------------------------+
+| Scheme layer used: 199 |
++------------------------+
+
+`View new and changed raw API methods <https://diff.telethon.dev/?from=193&to=199>`__.
+
+Additions
+~~~~~~~~~
+
+* ``drop_media_captions`` added to ``forward_messages``, and documented together with ``drop_author``.
+* :tl:`InputMediaDocumentExternal` is now recognized when sending albums.
+
+Enhancements
+~~~~~~~~~~~~
+
+* ``receive_updates=False`` now covers more cases, however, Telegram is still free to ignore it.
+* Better type-hints in several methods.
+* Markdown parsing of inline links should cover more cases.
+* ``range`` is now considered "list-like" and can be used on e.g. ``ids`` parameters.
+
+Bug fixes
+~~~~~~~~~
+
+* Session is now saved after setting the DC.
+* Fixed rare crash in entity cache handling when iterating through dialogs.
+* Fixed IOError that could occur during automatic resizing of some photos.
+
+
+New layer (v1.38)
+=================
+
++------------------------+
+| Scheme layer used: 193 |
++------------------------+
+
+`View new and changed raw API methods <https://diff.telethon.dev/?from=188&to=193>`__.
+
+Bug fixes
+~~~~~~~~~
+
+* Formatting entities misbehaved with albums.
+* Sending a Message object with a file did not use the new file.
+
+
 New layer (v1.37)
 =================
 

readthedocs/misc/wall-of-shame.rst

@@ -1,65 +0,0 @@
-=============
-Wall of Shame
-=============
-
-
-This project has an
-`issues <https://github.com/LonamiWebs/Telethon/issues>`__ section for
-you to file **issues** whenever you encounter any when working with the
-library. Said section is **not** for issues on *your* program but rather
-issues with Telethon itself.
-
-If you have not made the effort to 1. read through the docs and 2.
-`look for the method you need <https://tl.telethon.dev/>`__,
-you will end up on the `Wall of
-Shame <https://github.com/LonamiWebs/Telethon/issues?q=is%3Aissue+label%3ARTFM+is%3Aclosed>`__,
-i.e. all issues labeled
-`"RTFM" <http://www.urbandictionary.com/define.php?term=RTFM>`__:
-
-       **rtfm**
-       Literally "Read The F--king Manual"; a term showing the
-       frustration of being bothered with questions so trivial that the asker
-       could have quickly figured out the answer on their own with minimal
-       effort, usually by reading readily-available documents. People who
-       say"RTFM!" might be considered rude, but the true rude ones are the
-       annoying people who take absolutely no self-responibility and expect to
-       have all the answers handed to them personally.
-
-       *"Damn, that's the twelveth time that somebody posted this question
-       to the messageboard today! RTFM, already!"*
-
-       *by Bill M. July 27, 2004*
-
-If you have indeed read the docs, and have tried looking for the method,
-and yet you didn't find what you need, **that's fine**. Telegram's API
-can have some obscure names at times, and for this reason, there is a
-`"question"
-label <https://github.com/LonamiWebs/Telethon/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20label%3Aquestion%20>`__
-with questions that are okay to ask. Just state what you've tried so
-that we know you've made an effort, or you'll go to the Wall of Shame.
-
-Of course, if the issue you're going to open is not even a question but
-a real issue with the library (thankfully, most of the issues have been
-that!), you won't end up here. Don't worry.
-
-Current winner
---------------
-
-The current winner is `issue
-213 <https://github.com/LonamiWebs/Telethon/issues/213>`__:
-
-**Issue:**
-
-.. figure:: https://user-images.githubusercontent.com/6297805/29822978-9a9a6ef0-8ccd-11e7-9ec5-934ea0f57681.jpg
-
-:alt: Winner issue
-
-Winner issue
-
-**Answer:**
-
-.. figure:: https://user-images.githubusercontent.com/6297805/29822983-9d523402-8ccd-11e7-9fb1-5783740ee366.jpg
-
-:alt: Winner issue answer
-
-Winner issue answer

readthedocs/quick-references/faq.rst

@@ -20,7 +20,7 @@ To enable logging, at the following code to the top of your main file:
 .. code-block:: python
 
     import logging
-    logging.basicConfig(format='[%(levelname) 5s/%(asctime)s] %(name)s: %(message)s',
+    logging.basicConfig(format='[%(levelname) %(asctime)s] %(name)s: %(message)s',
                         level=logging.WARNING)
 
 You can change the logging level to be something different, from less to more information:
@@ -272,7 +272,7 @@ connects to Telegram, both agree on the layer to use. If the layers don't
 match, Telegram may send certain objects which Telethon no longer understands.
 
 When this message is reported as a "bug", the most common patterns seem to be
-that he Telethon session is being used or has been used from somewhere else.
+that the Telethon session is being used or has been used from somewhere else.
 Make sure that you created the session from Telethon, and are not using the
 same session anywhere else. If you need to use the same account from
 multiple places, login and use a different session for each place you need.

telethon/_updates/entitycache.py

@@ -49,9 +49,6 @@ class EntityCache:
             if getattr(c, 'access_hash', None) and not getattr(c, 'min', None)
         )
 
-    def get_all_entities(self):
-        return [Entity(ty, id, hash) for id, (hash, ty) in self.hash_map.items()]
-
     def put(self, entity):
         self.hash_map[entity.id] = (entity.hash, entity.ty)
 

telethon/_updates/messagebox.py

@@ -295,6 +295,8 @@ class MessageBox:
     #
     # It also updates the next deadline time to reflect the new closest deadline.
     def reset_deadlines(self, entries, deadline):
+        if not entries:
+            return
         for entry in entries:
             if entry not in self.map:
                 raise RuntimeError('Called reset_deadline on an entry for which we do not have state')
@@ -458,30 +460,14 @@ class MessageBox:
             return pts.pts - pts.pts_count if pts else 0
 
         reset_deadlines = set()  # temporary buffer
-        any_pts_applied = [False]  # using a list to pass "by reference"
 
         result.extend(filter(None, (
-            self.apply_pts_info(u, reset_deadlines=reset_deadlines, any_pts_applied=any_pts_applied)
+            self.apply_pts_info(u, reset_deadlines=reset_deadlines)
             # Telegram can send updates out of order (e.g. ReadChannelInbox first
             # and then NewChannelMessage, both with the same pts, but the count is
             # 0 and 1 respectively), so we sort them first.
             for u in sorted(updates, key=_sort_gaps))))
 
-        # > If the updates were applied, local *Updates* state must be updated
-        # > with `seq` (unless it's 0) and `date` from the constructor.
-        #
-        # By "were applied", we assume it means "some other pts was applied".
-        # Updates which can be applied in any order, such as `UpdateChat`,
-        # should not cause `seq` to be updated (or upcoming updates such as
-        # `UpdateChatParticipant` could be missed).
-        if any_pts_applied[0]:
-            if __debug__:
-                self._trace('Updating seq as local pts was updated too')
-            if date != epoch():
-                self.date = date
-            if seq != NO_SEQ:
-                self.seq = seq
-
         self.reset_deadlines(reset_deadlines, next_updates_deadline())
 
         if self.possible_gaps:
@@ -508,6 +494,16 @@ class MessageBox:
 
         real_result.extend(u for u in result if not u._self_outgoing)
 
+        if result and not self.possible_gaps:
+            # > If the updates were applied, local *Updates* state must be updated
+            # > with `seq` (unless it's 0) and `date` from the constructor.
+            if __debug__:
+                self._trace('Updating seq as all updates were applied')
+            if date != epoch():
+                self.date = date
+            if seq != NO_SEQ:
+                self.seq = seq
+
         return (users, chats)
 
     # Tries to apply the input update if its `PtsInfo` follows the correct order.
@@ -520,7 +516,6 @@ class MessageBox:
         update,
         *,
         reset_deadlines,
-        any_pts_applied=[True],  # mutable default is fine as it's write-only
     ):
         # This update means we need to call getChannelDifference to get the updates from the channel
         if isinstance(update, tl.UpdateChannelTooLong):
@@ -572,7 +567,6 @@ class MessageBox:
                 return None
             else:
                 # Apply
-                any_pts_applied[0] = True
                 if __debug__:
                     self._trace('Applying update pts since local pts %r = %r: %s', local_pts, pts, update)
 

telethon/_updates/session.py

@@ -1,7 +1,7 @@
 from typing import Optional, Tuple
 from enum import IntEnum
 from ..tl.types import InputPeerUser, InputPeerChat, InputPeerChannel
-
+import struct
 
 class SessionState:
     """
@@ -173,7 +173,7 @@ class Entity:
         try:
             ty, id, hash = struct.unpack('<Bqq', blob)
         except struct.error:
-            raise ValueError(f'malformed entity data, got {string!r}') from None
+            raise ValueError(f'malformed entity data, got {blob!r}') from None
 
         return cls(EntityType(ty), id, hash)
 

telethon/client/auth.py

@@ -147,14 +147,16 @@ class AuthMethods:
                 if bot_token[:bot_token.find(':')] != str(me.id):
                     warnings.warn(
                         'the session already had an authorized user so it did '
-                        'not login to the bot account using the provided '
-                        'bot_token (it may not be using the user you expect)'
+                        'not login to the bot account using the provided bot_token; '
+                        'if you were expecting a different user, check whether '
+                        'you are accidentally reusing an existing session'
                     )
             elif phone and not callable(phone) and utils.parse_phone(phone) != me.phone:
                 warnings.warn(
                     'the session already had an authorized user so it did '
-                    'not login to the user account using the provided '
-                    'phone (it may not be using the user you expect)'
+                    'not login to the user account using the provided phone; '
+                    'if you were expecting a different user, check whether '
+                    'you are accidentally reusing an existing session'
                 )
 
             return self
@@ -390,6 +392,16 @@ class AuthMethods:
         self._authorized = True
 
         state = await self(functions.updates.GetStateRequest())
+        # the server may send an old qts in getState
+        difference = await self(functions.updates.GetDifferenceRequest(pts=state.pts, date=state.date, qts=state.qts))
+
+        if isinstance(difference, types.updates.Difference):
+            state = difference.state
+        elif isinstance(difference, types.updates.DifferenceSlice):
+            state = difference.intermediate_state
+        elif isinstance(difference, types.updates.DifferenceTooLong):
+            state.pts = difference.pts
+
         self._message_box.load(SessionState(0, 0, 0, state.pts, state.qts, int(state.date.timestamp()), state.seq, 0), [])
 
         return user
@@ -540,7 +552,7 @@ class AuthMethods:
         self._authorized = False
 
         await self.disconnect()
-        self.session.delete()
+        await utils.maybe_async(self.session.delete())
         self.session = None
         return True
 

telethon/client/buttons.py

@@ -7,8 +7,8 @@ from ..tl import types, custom
 class ButtonMethods:
     @staticmethod
     def build_reply_markup(
-            buttons: 'typing.Optional[hints.MarkupLike]',
-            inline_only: bool = False) -> 'typing.Optional[types.TypeReplyMarkup]':
+            buttons: 'typing.Optional[hints.MarkupLike]'
+    ) -> 'typing.Optional[types.TypeReplyMarkup]':
         """
         Builds a :tl:`ReplyInlineMarkup` or :tl:`ReplyKeyboardMarkup` for
         the given buttons.
@@ -26,9 +26,6 @@ class ButtonMethods:
                 The button, list of buttons, array of buttons or markup
                 to convert into a markup.
 
-            inline_only (`bool`, optional):
-                Whether the buttons **must** be inline buttons only or not.
-
         Example
             .. code-block:: python
 
@@ -42,8 +39,8 @@ class ButtonMethods:
             return None
 
         try:
-            if buttons.SUBCLASS_OF_ID == 0xe2e10ef2:
-                return buttons  # crc32(b'ReplyMarkup'):
+            if buttons.SUBCLASS_OF_ID == 0xe2e10ef2:  # crc32(b'ReplyMarkup'):
+                return buttons
         except AttributeError:
             pass
 
@@ -57,6 +54,8 @@ class ButtonMethods:
         resize = None
         single_use = None
         selective = None
+        persistent = None
+        placeholder = None
 
         rows = []
         for row in buttons:
@@ -69,6 +68,10 @@ class ButtonMethods:
                         single_use = button.single_use
                     if button.selective is not None:
                         selective = button.selective
+                    if button.persistent is not None:
+                        persistent = button.persistent
+                    if button.placeholder is not None:
+                        placeholder = button.placeholder
 
                     button = button.button
                 elif isinstance(button, custom.MessageButton):
@@ -78,19 +81,21 @@ class ButtonMethods:
                 is_inline |= inline
                 is_normal |= not inline
 
-                if button.SUBCLASS_OF_ID == 0xbad74a3:
-                    # 0xbad74a3 == crc32(b'KeyboardButton')
+                if button.SUBCLASS_OF_ID == 0xbad74a3:  # crc32(b'KeyboardButton')
                     current.append(button)
 
             if current:
                 rows.append(types.KeyboardButtonRow(current))
 
-        if inline_only and is_normal:
-            raise ValueError('You cannot use non-inline buttons here')
-        elif is_inline == is_normal and is_normal:
+        if is_inline and is_normal:
             raise ValueError('You cannot mix inline with normal buttons')
         elif is_inline:
             return types.ReplyInlineMarkup(rows)
-        # elif is_normal:
         return types.ReplyKeyboardMarkup(
-            rows, resize=resize, single_use=single_use, selective=selective)
+            rows=rows,
+            resize=resize,
+            single_use=single_use,
+            selective=selective,
+            persistent=persistent,
+            placeholder=placeholder
+        )

telethon/client/chats.py

@@ -218,8 +218,10 @@ class _ParticipantsIter(RequestIter):
         self.requests.offset += len(participants.participants)
         users = {user.id: user for user in participants.users}
         for participant in participants.participants:
-
-            if isinstance(participant, types.ChannelParticipantBanned):
+            if isinstance(participant, types.ChannelParticipantLeft):
+                # See issue #3231 to learn why this is ignored.
+                continue
+            elif isinstance(participant, types.ChannelParticipantBanned):
                 if not isinstance(participant.peer, types.PeerUser):
                     # May have the entire channel banned. See #3105.
                     continue

telethon/client/downloads.py

@@ -63,9 +63,12 @@ class _DirectDownloadIter(RequestIter):
                 config = await self.client(functions.help.GetConfigRequest())
                 for option in config.dc_options:
                     if option.ip_address == self.client.session.server_address:
+                        await utils.maybe_async(
                             self.client.session.set_dc(
-                            option.id, option.ip_address, option.port)
-                        self.client.session.save()
+                                option.id, option.ip_address, option.port
+                            )
+                        )
+                        await utils.maybe_async(self.client.session.save())
                         break
 
                 # TODO Figure out why the session may have the wrong DC ID
@@ -992,8 +995,8 @@ class DownloadMethods:
             )
 
         # TODO Better way to get opened handles of files and auto-close
-        kind, possible_names = self._get_kind_and_names(web.attributes)
-        file = self._get_proper_filename(
+        kind, possible_names = cls._get_kind_and_names(web.attributes)
+        file = cls._get_proper_filename(
             file, kind, utils.get_extension(web),
             possible_names=possible_names
         )

telethon/client/messages.py

@@ -644,6 +644,8 @@ class MessageMethods:
             schedule: 'hints.DateLike' = None,
             comment_to: 'typing.Union[int, types.Message]' = None,
             nosound_video: bool = None,
+            send_as: typing.Optional['hints.EntityLike'] = None,
+            message_effect_id: typing.Optional[int] = None
     ) -> 'types.Message':
         """
         Sends a message to the specified user, chat or channel.
@@ -766,6 +768,16 @@ class MessageMethods:
                 on non-video files. This is set to ``True`` for albums, as gifs
                 cannot be sent in albums.
 
+            send_as (`entity`):
+                Unique identifier (int) or username (str) of the chat or channel to send the message as.
+                You can use this to send the message on behalf of a chat or channel where you have appropriate permissions.
+                Use the GetSendAs to return the list of message sender identifiers, which can be used to send messages in the chat,
+                This setting applies to the current message and will remain effective for future messages unless explicitly changed.
+                To set this behavior permanently for all messages, use SaveDefaultSendAs.
+
+            message_effect_id (`int`, optional):
+                Unique identifier of the message effect to be added to the message; for private chats only
+
         Returns
             The sent `custom.Message <telethon.tl.custom.message.Message>`.
 
@@ -826,6 +838,9 @@ class MessageMethods:
                 await client.send_message(chat, 'Hi, future!', schedule=timedelta(minutes=5))
         """
         if file is not None:
+            if isinstance(message, types.Message):
+                formatting_entities = formatting_entities or message.entities
+                message = message.message
             return await self.send_file(
                 entity, file, caption=message, reply_to=reply_to,
                 attributes=attributes, parse_mode=parse_mode,
@@ -835,6 +850,7 @@ class MessageMethods:
                 formatting_entities=formatting_entities,
                 comment_to=comment_to, background=background,
                 nosound_video=nosound_video,
+                send_as=send_as, message_effect_id=message_effect_id
             )
 
         entity = await self.get_input_entity(entity)
@@ -864,7 +880,8 @@ class MessageMethods:
                     buttons=markup,
                     formatting_entities=message.entities,
                     parse_mode=None,  # explicitly disable parse_mode to force using even empty formatting_entities
-                    schedule=schedule
+                    schedule=schedule,
+                    send_as=send_as, message_effect_id=message_effect_id
                 )
 
             request = functions.messages.SendMessageRequest(
@@ -878,7 +895,9 @@ class MessageMethods:
                 clear_draft=clear_draft,
                 no_webpage=not isinstance(
                     message.media, types.MessageMediaWebPage),
-                schedule_date=schedule
+                schedule_date=schedule,
+                send_as=await self.get_input_entity(send_as) if send_as else None,
+                effect=message_effect_id
             )
             message = message.message
         else:
@@ -899,7 +918,9 @@ class MessageMethods:
                 silent=silent,
                 background=background,
                 reply_markup=self.build_reply_markup(buttons),
-                schedule_date=schedule
+                schedule_date=schedule,
+                send_as=await self.get_input_entity(send_as) if send_as else None,
+                effect=message_effect_id
             )
 
         result = await self(request)
@@ -933,6 +954,7 @@ class MessageMethods:
             as_album: bool = None,
             schedule: 'hints.DateLike' = None,
             drop_author: bool = None,
+            drop_media_captions: bool = None,
     ) -> 'typing.Sequence[types.Message]':
         """
         Forwards the given messages to the specified entity.
@@ -976,6 +998,12 @@ class MessageMethods:
                 instead they will be scheduled to be automatically sent
                 at a later time.
 
+            drop_author (`bool`, optional):
+                Whether to forward messages without quoting the original author.
+
+            drop_media_captions (`bool`, optional):
+                Whether to strip captions from media. Setting this to `True` requires that `drop_author` also be set to `True`.
+
         Returns
             The list of forwarded `Message <telethon.tl.custom.message.Message>`,
             or a single one if a list wasn't provided as input.
@@ -1045,7 +1073,8 @@ class MessageMethods:
                 background=background,
                 with_my_score=with_my_score,
                 schedule_date=schedule,
-                drop_author=drop_author
+                drop_author=drop_author,
+                drop_media_captions=drop_media_captions
             )
             result = await self(req)
             sent.extend(self._get_response_message(req, result, entity))
@@ -1055,7 +1084,7 @@ class MessageMethods:
     async def edit_message(
             self: 'TelegramClient',
             entity: 'typing.Union[hints.EntityLike, types.Message]',
-            message: 'hints.MessageLike' = None,
+            message: 'typing.Union[int, types.Message, types.InputMessageID, str]' = None,
             text: str = None,
             *,
             parse_mode: str = (),
@@ -1085,7 +1114,7 @@ class MessageMethods:
                 which is the only way to edit messages that were sent
                 after the user selects an inline query result.
 
-            message (`int` | `Message <telethon.tl.custom.message.Message>` | `str`):
+            message (`int` | `Message <telethon.tl.custom.message.Message>` | :tl:`InputMessageID` | `str`):
                 The ID of the message (or `Message
                 <telethon.tl.custom.message.Message>` itself) to be edited.
                 If the `entity` was a `Message

telethon/client/telegrambaseclient.py

@@ -1,4 +1,5 @@
 import abc
+import inspect
 import re
 import asyncio
 import collections
@@ -9,7 +10,7 @@ import typing
 import datetime
 import pathlib
 
-from .. import version, helpers, __name__ as __base_name__
+from .. import utils, version, helpers, __name__ as __base_name__
 from ..crypto import rsa
 from ..extensions import markdown
 from ..network import MTProtoSender, Connection, ConnectionTcpFull, TcpMTProxy
@@ -305,15 +306,6 @@ class TelegramBaseClient(abc.ABC):
                 'The given session must be a str or a Session instance.'
             )
 
-        # ':' in session.server_address is True if it's an IPv6 address
-        if (not session.server_address or
-                (':' in session.server_address) != use_ipv6):
-            session.set_dc(
-                DEFAULT_DC_ID,
-                DEFAULT_IPV6_IP if self._use_ipv6 else DEFAULT_IPV4_IP,
-                DEFAULT_PORT
-            )
-
         self.flood_sleep_threshold = flood_sleep_threshold
 
         # TODO Use AsyncClassWrapper(session)
@@ -545,6 +537,18 @@ class TelegramBaseClient(abc.ABC):
         elif self._loop != helpers.get_running_loop():
             raise RuntimeError('The asyncio event loop must not change after connection (see the FAQ for details)')
 
+        # ':' in session.server_address is True if it's an IPv6 address
+        if (not self.session.server_address or
+                (':' in self.session.server_address) != self._use_ipv6):
+            await utils.maybe_async(
+                self.session.set_dc(
+                    DEFAULT_DC_ID,
+                    DEFAULT_IPV6_IP if self._use_ipv6 else DEFAULT_IPV4_IP,
+                    DEFAULT_PORT
+                )
+            )
+            await utils.maybe_async(self.session.save())
+
         if not await self._sender.connect(self._connection(
             self.session.server_address,
             self.session.port,
@@ -557,12 +561,13 @@ class TelegramBaseClient(abc.ABC):
             return
 
         self.session.auth_key = self._sender.auth_key
-        self.session.save()
+        await utils.maybe_async(self.session.save())
 
         try:
             # See comment when saving entities to understand this hack
-            self_id = self.session.get_input_entity(0).access_hash
-            self_user = self.session.get_input_entity(self_id)
+            self_entity = await utils.maybe_async(self.session.get_input_entity(0))
+            self_id = self_entity.access_hash
+            self_user = await utils.maybe_async(self.session.get_input_entity(self_id))
             self._mb_entity_cache.set_self_user(self_id, None, self_user.access_hash)
         except ValueError:
             pass
@@ -571,7 +576,8 @@ class TelegramBaseClient(abc.ABC):
             ss = SessionState(0, 0, False, 0, 0, 0, 0, None)
             cs = []
 
-            for entity_id, state in self.session.get_update_states():
+            update_states = await utils.maybe_async(self.session.get_update_states())
+            for entity_id, state in update_states:
                 if entity_id == 0:
                     # TODO current session doesn't store self-user info but adding that is breaking on downstream session impls
                     ss = SessionState(0, 0, False, state.pts, state.qts, int(state.date.timestamp()), state.seq, None)
@@ -581,7 +587,7 @@ class TelegramBaseClient(abc.ABC):
             self._message_box.load(ss, cs)
             for state in cs:
                 try:
-                    entity = self.session.get_input_entity(state.channel_id)
+                    entity = await utils.maybe_async(self.session.get_input_entity(state.channel_id))
                 except ValueError:
                     self._log[__name__].warning(
                         'No access_hash in cache for channel %s, will not catch up', state.channel_id)
@@ -685,23 +691,27 @@ class TelegramBaseClient(abc.ABC):
             else:
                 connection._proxy = proxy
 
-    def _save_states_and_entities(self: 'TelegramClient'):
-        entities = self._mb_entity_cache.get_all_entities()
-
-        # Piggy-back on an arbitrary TL type with users and chats so the session can understand to read the entities.
-        # It doesn't matter if we put users in the list of chats.
-        self.session.process_entities(types.contacts.ResolvedPeer(None, [e._as_input_peer() for e in entities], []))
-
+    async def _save_states_and_entities(self: 'TelegramClient'):
         # As a hack to not need to change the session files, save ourselves with ``id=0`` and ``access_hash`` of our ``id``.
         # This way it is possible to determine our own ID by querying for 0. However, whether we're a bot is not saved.
+        # Piggy-back on an arbitrary TL type with users and chats so the session can understand to read the entities.
+        # It doesn't matter if we put users in the list of chats.
         if self._mb_entity_cache.self_id:
-            self.session.process_entities(types.contacts.ResolvedPeer(None, [types.InputPeerUser(0, self._mb_entity_cache.self_id)], []))
+            await utils.maybe_async(
+                self.session.process_entities(
+                    types.contacts.ResolvedPeer(None, [types.InputPeerUser(0, self._mb_entity_cache.self_id)], [])
+                )
+            )
 
         ss, cs = self._message_box.session_state()
-        self.session.set_update_state(0, types.updates.State(**ss, unread_count=0))
+        await utils.maybe_async(self.session.set_update_state(0, types.updates.State(**ss, unread_count=0)))
         now = datetime.datetime.now()  # any datetime works; channels don't need it
         for channel_id, pts in cs.items():
-            self.session.set_update_state(channel_id, types.updates.State(pts, 0, now, 0, unread_count=0))
+            await utils.maybe_async(
+                self.session.set_update_state(
+                    channel_id, types.updates.State(pts, 0, now, 0, unread_count=0)
+                )
+            )
 
     async def _disconnect_coro(self: 'TelegramClient'):
         if self.session is None:
@@ -733,9 +743,9 @@ class TelegramBaseClient(abc.ABC):
             await asyncio.wait(self._event_handler_tasks)
             self._event_handler_tasks.clear()
 
-        self._save_states_and_entities()
+        await self._save_states_and_entities()
 
-        self.session.close()
+        await utils.maybe_async(self.session.close())
 
     async def _disconnect(self: 'TelegramClient'):
         """
@@ -756,22 +766,22 @@ class TelegramBaseClient(abc.ABC):
         self._log[__name__].info('Reconnecting to new data center %s', new_dc)
         dc = await self._get_dc(new_dc)
 
-        self.session.set_dc(dc.id, dc.ip_address, dc.port)
+        await utils.maybe_async(self.session.set_dc(dc.id, dc.ip_address, dc.port))
         # auth_key's are associated with a server, which has now changed
         # so it's not valid anymore. Set to None to force recreating it.
         self._sender.auth_key.key = None
         self.session.auth_key = None
-        self.session.save()
+        await utils.maybe_async(self.session.save())
         await self._disconnect()
         return await self.connect()
 
-    def _auth_key_callback(self: 'TelegramClient', auth_key):
+    async def _auth_key_callback(self: 'TelegramClient', auth_key):
         """
         Callback from the sender whenever it needed to generate a
         new authorization key. This means we are not authorized.
         """
         self.session.auth_key = auth_key
-        self.session.save()
+        await utils.maybe_async(self.session.save())
 
     # endregion
 
@@ -897,8 +907,8 @@ class TelegramBaseClient(abc.ABC):
         session = self._exported_sessions.get(cdn_redirect.dc_id)
         if not session:
             dc = await self._get_dc(cdn_redirect.dc_id, cdn=True)
-            session = self.session.clone()
-            session.set_dc(dc.id, dc.ip_address, dc.port)
+            session = await utils.maybe_async(self.session.clone())
+            await utils.maybe_async(session.set_dc(dc.id, dc.ip_address, dc.port))
             self._exported_sessions[cdn_redirect.dc_id] = session
 
         self._log[__name__].info('Creating new CDN client')

telethon/client/updates.py

@@ -289,7 +289,7 @@ class UpdateMethods:
                         len(self._mb_entity_cache),
                         self._entity_cache_limit
                     )
-                    self._save_states_and_entities()
+                    await self._save_states_and_entities()
                     self._mb_entity_cache.retain(lambda id: id == self._mb_entity_cache.self_id or id in self._message_box.map)
                     if len(self._mb_entity_cache) >= self._entity_cache_limit:
                         warnings.warn('in-memory entities exceed entity_cache_limit after flushing; consider setting a larger limit')
@@ -343,7 +343,8 @@ class UpdateMethods:
                     if updates:
                         self._log[__name__].info('Got difference for account updates')
 
-                    updates_to_dispatch.extend(self._preprocess_updates(updates, users, chats))
+                    _preprocess_updates = await utils.maybe_async(self._preprocess_updates(updates, users, chats))
+                    updates_to_dispatch.extend(_preprocess_updates)
                     continue
 
                 get_diff = self._message_box.get_channel_difference(self._mb_entity_cache)
@@ -441,7 +442,8 @@ class UpdateMethods:
                     if updates:
                         self._log[__name__].info('Got difference for channel %d updates', get_diff.channel.channel_id)
 
-                    updates_to_dispatch.extend(self._preprocess_updates(updates, users, chats))
+                    _preprocess_updates = await utils.maybe_async(self._preprocess_updates(updates, users, chats))
+                    updates_to_dispatch.extend(_preprocess_updates)
                     continue
 
                 deadline = self._message_box.check_deadlines()
@@ -462,7 +464,8 @@ class UpdateMethods:
                 except GapError:
                     continue  # get(_channel)_difference will start returning requests
 
-                updates_to_dispatch.extend(self._preprocess_updates(processed, users, chats))
+                _preprocess_updates = await utils.maybe_async(self._preprocess_updates(processed, users, chats))
+                updates_to_dispatch.extend(_preprocess_updates)
         except asyncio.CancelledError:
             pass
         except Exception as e:
@@ -470,8 +473,9 @@ class UpdateMethods:
             self._updates_error = e
             await self.disconnect()
 
-    def _preprocess_updates(self, updates, users, chats):
+    async def _preprocess_updates(self, updates, users, chats):
         self._mb_entity_cache.extend(users, chats)
+        await utils.maybe_async(self.session.process_entities(types.contacts.ResolvedPeer(None, users, chats)))
         entities = {utils.get_peer_id(x): x
                     for x in itertools.chain(users, chats)}
         for u in updates:
@@ -514,9 +518,9 @@ class UpdateMethods:
             # inserted because this is a rather expensive operation
             # (default's sqlite3 takes ~0.1s to commit changes). Do
             # it every minute instead. No-op if there's nothing new.
-            self._save_states_and_entities()
+            await self._save_states_and_entities()
 
-            self.session.save()
+            await utils.maybe_async(self.session.save())
 
     async def _dispatch_update(self: 'TelegramClient', update):
         # TODO only used for AlbumHack, and MessageBox is not really designed for this

telethon/client/uploads.py

@@ -18,7 +18,6 @@ try:
 except ImportError:
     PIL = None
 
-
 if typing.TYPE_CHECKING:
     from .telegramclient import TelegramClient
 
@@ -68,24 +67,31 @@ def _resize_photo_if_needed(
         except KeyError:
             kwargs = {}
 
+        if image.mode == 'RGB':
             # Check if image is within acceptable bounds, if so, check if the image is at or below 10 MB, or assume it isn't if size is None or 0
             if image.width <= width and image.height <= height and (before <= 10000000 if before else False):
                 return file
 
+            # If the image is already RGB, don't convert it
+            # certain modes such as 'P' have no alpha index but can't be saved as JPEG directly
             image.thumbnail((width, height), PIL.Image.LANCZOS)
-
-        alpha_index = image.mode.find('A')
-        if alpha_index == -1:
-            # If the image mode doesn't have alpha
-            # channel then don't bother masking it away.
             result = image
         else:
             # We could save the resized image with the original format, but
             # JPEG often compresses better -> smaller size -> faster upload
             # We need to mask away the alpha channel ([3]), since otherwise
             # IOError is raised when trying to save alpha channels in JPEG.
+            image.thumbnail((width, height), PIL.Image.LANCZOS)
             result = PIL.Image.new('RGB', image.size, background)
-            result.paste(image, mask=image.split()[alpha_index])
+            mask = None
+
+            if image.has_transparency_data:
+                if image.mode == 'RGBA':
+                    mask = image.getchannel('A')
+                else:
+                    mask = image.convert('RGBA').getchannel('A')
+
+            result.paste(image, mask=mask)
 
         buffer = io.BytesIO()
         result.save(buffer, 'JPEG', progressive=True, **kwargs)
@@ -111,6 +117,7 @@ class UploadMethods:
             *,
             caption: typing.Union[str, typing.Sequence[str]] = None,
             force_document: bool = False,
+            mime_type: str = None,
             file_size: int = None,
             clear_draft: bool = False,
             progress_callback: 'hints.ProgressCallback' = None,
@@ -119,7 +126,11 @@ class UploadMethods:
             thumb: 'hints.FileLike' = None,
             allow_cache: bool = True,
             parse_mode: str = (),
-            formatting_entities: typing.Optional[typing.List[types.TypeMessageEntity]] = None,
+            formatting_entities: typing.Optional[
+                typing.Union[
+                    typing.List[types.TypeMessageEntity], typing.List[typing.List[types.TypeMessageEntity]]
+                ]
+            ] = None,
             voice_note: bool = False,
             video_note: bool = False,
             buttons: typing.Optional['hints.MarkupLike'] = None,
@@ -130,7 +141,9 @@ class UploadMethods:
             comment_to: 'typing.Union[int, types.Message]' = None,
             ttl: int = None,
             nosound_video: bool = None,
-            **kwargs) -> 'types.Message':
+            send_as: typing.Optional['hints.EntityLike'] = None,
+            message_effect_id: typing.Optional[int] = None,
+            **kwargs) -> typing.Union[typing.List[typing.Any], typing.Any]:
         """
         Sends message with the given file to the specified entity.
 
@@ -197,6 +210,13 @@ class UploadMethods:
                 the extension of an image file or a video file, it will be
                 sent as such. Otherwise always as a document.
 
+            mime_type (`str`, optional):
+                Custom mime type to use for the file to be sent (for example,
+                ``audio/mpeg``, ``audio/x-vorbis+ogg``, etc.).
+                It can change the type of files displayed.
+                If not set to any value, the mime type will be determined
+                automatically based on the file's extension.
+
             file_size (`int`, optional):
                 The size of the file to be uploaded if it needs to be uploaded,
                 which will be determined automatically if not specified.
@@ -243,7 +263,11 @@ class UploadMethods:
                 default.
 
             formatting_entities (`list`, optional):
-                A list of message formatting entities. When provided, the ``parse_mode`` is ignored.
+                Optional formatting entities for the sent media message. When sending an album,
+                `formatting_entities` can be a list of lists, where each inner list contains
+                `types.TypeMessageEntity`. Each inner list will be assigned to the corresponding
+                file in a pairwise manner with the caption. If provided, the ``parse_mode``
+                parameter will be ignored.
 
             voice_note (`bool`, optional):
                 If `True` the audio will be sent as a voice note.
@@ -308,6 +332,16 @@ class UploadMethods:
                 on non-video files. This is set to ``True`` for albums, as gifs
                 cannot be sent in albums.
 
+            send_as (`entity`):
+                Unique identifier (int) or username (str) of the chat or channel to send the message as.
+                You can use this to send the message on behalf of a chat or channel where you have appropriate permissions.
+                Use the GetSendAs to return the list of message sender identifiers, which can be used to send messages in the chat,
+                This setting applies to the current message and will remain effective for future messages unless explicitly changed.
+                To set this behavior permanently for all messages, use SaveDefaultSendAs.
+
+            message_effect_id (`int`, optional):
+                Unique identifier of the message effect to be added to the message; for private chats only
+
         Returns
             The `Message <telethon.tl.custom.message.Message>` (or messages)
             containing the sent file, or messages if a list of them was passed.
@@ -365,6 +399,9 @@ class UploadMethods:
         if not caption:
             caption = ''
 
+        if not formatting_entities:
+            formatting_entities = []
+
         entity = await self.get_input_entity(entity)
         if comment_to is not None:
             entity, reply_to = await self._get_comment_data(entity, comment_to)
@@ -384,22 +421,36 @@ class UploadMethods:
             else:
                 captions = [caption]
 
+            # Check that formatting_entities list is valid
+            if all(utils.is_list_like(obj) for obj in formatting_entities):
+                formatting_entities = formatting_entities
+            elif utils.is_list_like(formatting_entities):
+                formatting_entities = [formatting_entities]
+            else:
+                raise TypeError('The formatting_entities argument must be a list or a sequence of lists')
+
+            # Check that all entities in all lists are of the correct type
+            if not all(isinstance(ent, types.TypeMessageEntity) for sublist in formatting_entities for ent in sublist):
+                raise TypeError('All entities must be instances of <types.TypeMessageEntity>')
+
             result = []
             while file:
                 result += await self._send_album(
-                    entity, file[:10], caption=captions[:10],
+                    entity, file[:10], caption=captions[:10], formatting_entities=formatting_entities[:10],
                     progress_callback=used_callback, reply_to=reply_to,
                     parse_mode=parse_mode, silent=silent, schedule=schedule,
                     supports_streaming=supports_streaming, clear_draft=clear_draft,
                     force_document=force_document, background=background,
+                    send_as=send_as, message_effect_id=message_effect_id
                 )
                 file = file[10:]
                 captions = captions[10:]
+                formatting_entities = formatting_entities[10:]
                 sent_count += 10
 
             return result
 
-        if formatting_entities is not None:
+        if formatting_entities:
             msg_entities = formatting_entities
         else:
             caption, msg_entities =\
@@ -407,6 +458,7 @@ class UploadMethods:
 
         file_handle, media, image = await self._file_to_media(
             file, force_document=force_document,
+            mime_type=mime_type,
             file_size=file_size,
             progress_callback=progress_callback,
             attributes=attributes, allow_cache=allow_cache, thumb=thumb,
@@ -425,15 +477,20 @@ class UploadMethods:
             entity, media, reply_to=reply_to, message=caption,
             entities=msg_entities, reply_markup=markup, silent=silent,
             schedule_date=schedule, clear_draft=clear_draft,
-            background=background
+            background=background,
+            send_as=await self.get_input_entity(send_as) if send_as else None,
+            effect=message_effect_id
         )
         return self._get_response_message(request, await self(request), entity)
 
     async def _send_album(self: 'TelegramClient', entity, files, caption='',
+                          formatting_entities=None,
                           progress_callback=None, reply_to=None,
                           parse_mode=(), silent=None, schedule=None,
                           supports_streaming=None, clear_draft=None,
-                          force_document=False, background=None, ttl=None):
+                          force_document=False, background=None, ttl=None,
+                          send_as: typing.Optional['hints.EntityLike'] = None,
+                          message_effect_id: typing.Optional[int] = None):
         """Specialized version of .send_file for albums"""
         # We don't care if the user wants to avoid cache, we will use it
         # anyway. Why? The cached version will be exactly the same thing
@@ -441,14 +498,23 @@ class UploadMethods:
         # cache only makes a difference for documents where the user may
         # want the attributes used on them to change.
         #
-        # In theory documents can be sent inside the albums but they appear
+        # In theory documents can be sent inside the albums, but they appear
         # as different messages (not inside the album), and the logic to set
         # the attributes/avoid cache is already written in .send_file().
         entity = await self.get_input_entity(entity)
         if not utils.is_list_like(caption):
             caption = (caption,)
+        if not all(isinstance(obj, list) for obj in formatting_entities):
+            formatting_entities = (formatting_entities,)
 
         captions = []
+        # If the formatting_entities argument is provided, we don't use parse_mode
+        if formatting_entities:
+            # Pop from the end (so reverse)
+            capt_with_ent = itertools.zip_longest(reversed(caption), reversed(formatting_entities), fillvalue=None)
+            for msg_caption, msg_entities in capt_with_ent:
+                captions.append((msg_caption, msg_entities))
+        else:
             for c in reversed(caption):  # Pop from the end (so reverse)
                 captions.append(await self._parse_message_text(c or '', parse_mode))
 
@@ -476,7 +542,7 @@ class UploadMethods:
                 ))
 
                 fm = utils.get_input_media(r.photo)
-            elif isinstance(fm, types.InputMediaUploadedDocument):
+            elif isinstance(fm, (types.InputMediaUploadedDocument, types.InputMediaDocumentExternal)):
                 r = await self(functions.messages.UploadMediaRequest(
                     entity, media=fm
                 ))
@@ -499,7 +565,9 @@ class UploadMethods:
         request = functions.messages.SendMultiMediaRequest(
             entity, reply_to=None if reply_to is None else types.InputReplyToMessage(reply_to), multi_media=media,
             silent=silent, schedule_date=schedule, clear_draft=clear_draft,
-            background=background
+            background=background,
+            send_as=await self.get_input_entity(send_as) if send_as else None,
+            effect=message_effect_id
         )
         result = await self(request)
 
@@ -712,7 +780,7 @@ class UploadMethods:
 
         # `aiofiles` do not base `io.IOBase` but do have `read`, so we
         # just check for the read attribute to see if it's file-like.
-        if not isinstance(file, (str, bytes, types.InputFile, types.InputFileBig))\
+        if not isinstance(file, (str, bytes, types.InputFile, types.InputFileBig)) \
                 and not hasattr(file, 'read'):
             # The user may pass a Message containing media (or the media,
             # or anything similar) that should be treated as a file. Try

telethon/client/users.py

@@ -36,8 +36,9 @@ class UserMethods:
 
         if flood_sleep_threshold is None:
             flood_sleep_threshold = self.flood_sleep_threshold
-        requests = (request if utils.is_list_like(request) else (request,))
-        for r in requests:
+        requests = list(request) if utils.is_list_like(request) else [request]
+        request = list(request) if utils.is_list_like(request) else request
+        for i, r in enumerate(requests):
             if not isinstance(r, TLRequest):
                 raise _NOT_A_REQUEST()
             await r.resolve(self, utils)
@@ -56,7 +57,11 @@ class UserMethods:
                     raise errors.FloodWaitError(request=r, capture=diff)
 
             if self._no_updates:
-                r = functions.InvokeWithoutUpdatesRequest(r)
+                if utils.is_list_like(request):
+                    request[i] = functions.InvokeWithoutUpdatesRequest(r)
+                else:
+                    # This should only run once as requests should be a list of 1 item
+                    request = functions.InvokeWithoutUpdatesRequest(r)
 
         request_index = 0
         last_error = None
@@ -75,7 +80,7 @@ class UserMethods:
                             exceptions.append(e)
                             results.append(None)
                             continue
-                        self.session.process_entities(result)
+                        await utils.maybe_async(self.session.process_entities(result))
                         exceptions.append(None)
                         results.append(result)
                         request_index += 1
@@ -85,7 +90,7 @@ class UserMethods:
                         return results
                 else:
                     result = await future
-                    self.session.process_entities(result)
+                    await utils.maybe_async(self.session.process_entities(result))
                     return result
             except (errors.ServerError, errors.RpcCallFailError,
                     errors.RpcMcgetFailError, errors.InterdcCallErrorError,
@@ -430,7 +435,8 @@ class UserMethods:
 
         # No InputPeer, cached peer, or known string. Fetch from disk cache
         try:
-            return self.session.get_input_entity(peer)
+            input_entity = await utils.maybe_async(self.session.get_input_entity(peer))
+            return input_entity
         except ValueError:
             pass
 
@@ -569,8 +575,8 @@ class UserMethods:
                     pass
             try:
                 # Nobody with this username, maybe it's an exact name/title
-                return await self.get_entity(
-                    self.session.get_input_entity(string))
+                input_entity = await utils.maybe_async(self.session.get_input_entity(string))
+                return await self.get_entity(input_entity)
             except ValueError:
                 pass
 

telethon/extensions/binaryreader.py

@@ -1,11 +1,9 @@
 """
 This module contains the BinaryReader utility class.
 """
-import os
+import struct
 import time
-from datetime import datetime, timezone, timedelta
-from io import BytesIO
-from struct import unpack
+from datetime import datetime, timedelta, timezone
 
 from ..errors import TypeNotFoundError
 from ..tl.alltlobjects import tlobjects
@@ -21,7 +19,8 @@ class BinaryReader:
     """
 
     def __init__(self, data):
-        self.stream = BytesIO(data)
+        self.stream = data or b''
+        self.position = 0
         self._last = None  # Should come in handy to spot -404 errors
 
     # region Reading
@@ -30,23 +29,35 @@ class BinaryReader:
     # https://core.telegram.org/mtproto
     def read_byte(self):
         """Reads a single byte value."""
-        return self.read(1)[0]
+        value, = struct.unpack_from("<B", self.stream, self.position)
+        self.position += 1
+        return value
 
     def read_int(self, signed=True):
         """Reads an integer (4 bytes) value."""
-        return int.from_bytes(self.read(4), byteorder='little', signed=signed)
+        fmt = '<i' if signed else '<I'
+        value, = struct.unpack_from(fmt, self.stream, self.position)
+        self.position += 4
+        return value
 
     def read_long(self, signed=True):
         """Reads a long integer (8 bytes) value."""
-        return int.from_bytes(self.read(8), byteorder='little', signed=signed)
+        fmt = '<q' if signed else '<Q'
+        value, = struct.unpack_from(fmt, self.stream, self.position)
+        self.position += 8
+        return value
 
     def read_float(self):
         """Reads a real floating point (4 bytes) value."""
-        return unpack('<f', self.read(4))[0]
+        value, = struct.unpack_from("<f", self.stream, self.position)
+        self.position += 4
+        return value
 
     def read_double(self):
         """Reads a real floating point (8 bytes) value."""
-        return unpack('<d', self.read(8))[0]
+        value, = struct.unpack_from("<d", self.stream, self.position)
+        self.position += 8
+        return value
 
     def read_large_int(self, bits, signed=True):
         """Reads a n-bits long integer value."""
@@ -55,7 +66,12 @@ class BinaryReader:
 
     def read(self, length=-1):
         """Read the given amount of bytes, or -1 to read all remaining."""
-        result = self.stream.read(length)
+        if length >= 0:
+            result = self.stream[self.position:self.position + length]
+            self.position += length
+        else:
+            result = self.stream[self.position:]
+            self.position += len(result)
         if (length >= 0) and (len(result) != length):
             raise BufferError(
                 'No more data left to read (need {}, got {}: {}); last read {}'
@@ -67,7 +83,7 @@ class BinaryReader:
 
     def get_bytes(self):
         """Gets the byte array representing the current buffer as a whole."""
-        return self.stream.getvalue()
+        return self.stream
 
     # endregion
 
@@ -153,24 +169,24 @@ class BinaryReader:
 
     def close(self):
         """Closes the reader, freeing the BytesIO stream."""
-        self.stream.close()
+        self.stream = b''
 
     # region Position related
 
     def tell_position(self):
         """Tells the current position on the stream."""
-        return self.stream.tell()
+        return self.position
 
     def set_position(self, position):
         """Sets the current position on the stream."""
-        self.stream.seek(position)
+        self.position = position
 
     def seek(self, offset):
         """
         Seeks the stream position given an offset from the current position.
         The offset may be negative.
         """
-        self.stream.seek(offset, os.SEEK_CUR)
+        self.position += offset
 
     # endregion
 

telethon/extensions/html.py

@@ -13,7 +13,7 @@ from ..tl.types import (
     MessageEntityPre, MessageEntityEmail, MessageEntityUrl,
     MessageEntityTextUrl, MessageEntityMentionName,
     MessageEntityUnderline, MessageEntityStrike, MessageEntityBlockquote,
-    TypeMessageEntity
+    MessageEntityCustomEmoji, TypeMessageEntity
 )
 
 
@@ -78,6 +78,14 @@ class HTMLToTelegramParser(HTMLParser):
                     url = None
             self._open_tags_meta.popleft()
             self._open_tags_meta.appendleft(url)
+        elif tag == 'tg-emoji':
+            try:
+                emoji_id = int(attrs['emoji-id'])
+            except (KeyError, ValueError):
+                return
+
+            EntityType = MessageEntityCustomEmoji
+            args['document_id'] = emoji_id
             
         if EntityType and tag not in self._building_entities:
             self._building_entities[tag] = EntityType(
@@ -146,6 +154,7 @@ ENTITY_TO_FORMATTER = {
     MessageEntityUrl: lambda _, t: ('<a href="{}">'.format(t), '</a>'),
     MessageEntityTextUrl: lambda e, _: ('<a href="{}">'.format(escape(e.url)), '</a>'),
     MessageEntityMentionName: lambda e, _: ('<a href="tg://user?id={}">'.format(e.user_id), '</a>'),
+    MessageEntityCustomEmoji: lambda e, _: ('<tg-emoji emoji-id="{}">'.format(e.document_id), '</tg-emoji>'),
 }
 
 

telethon/extensions/markdown.py

@@ -22,14 +22,10 @@ DEFAULT_DELIMITERS = {
     '```': MessageEntityPre
 }
 
-DEFAULT_URL_RE = re.compile(r'\[([^\]]+)\]\(([^)]+)\)')
+DEFAULT_URL_RE = re.compile(r'\[([^]]*?)\]\(([\s\S]*?)\)')
 DEFAULT_URL_FORMAT = '[{0}]({1})'
 
 
-def overlap(a, b, x, y):
-    return max(a, x) < min(b, y)
-
-
 def parse(message, delimiters=None, url_re=None):
     """
     Parses the given markdown message and returns its stripped representation
@@ -90,8 +86,8 @@ def parse(message, delimiters=None, url_re=None):
                 for ent in result:
                     # If the end is after our start, it is affected
                     if ent.offset + ent.length > i:
-                        # If the old start is also before ours, it is fully enclosed
-                        if ent.offset <= i:
+                        # If the old start is before ours and the old end is after ours, we are fully enclosed
+                        if ent.offset <= i and ent.offset + ent.length >= end + len(delim):
                             ent.length -= len(delim) * 2
                         else:
                             ent.length -= len(delim)
@@ -119,7 +115,7 @@ def parse(message, delimiters=None, url_re=None):
                     message[m.end():]
                 ))
 
-                delim_size = m.end() - m.start() - len(m.group())
+                delim_size = m.end() - m.start() - len(m.group(1))
                 for ent in result:
                     # If the end is after our start, it is affected
                     if ent.offset + ent.length > m.start():

telethon/hints.py

@@ -44,7 +44,12 @@ FileLike = typing.Union[
     typing.BinaryIO,
     types.TypeMessageMedia,
     types.TypeInputFile,
-    types.TypeInputFileLocation
+    types.TypeInputFileLocation,
+    types.TypeInputMedia,
+    types.TypePhoto,
+    types.TypeInputPhoto,
+    types.TypeDocument,
+    types.TypeInputDocument
 ]
 
 # Can't use `typing.Type` in Python 3.5.2

telethon/network/mtprotosender.py

@@ -302,7 +302,7 @@ class MTProtoSender:
             # notify whenever we change it. This is crucial when we
             # switch to different data centers.
             if self._auth_key_callback:
-                self._auth_key_callback(self.auth_key)
+                await self._auth_key_callback(self.auth_key)
 
             self._log.debug('auth_key generation success!')
             return True

telethon/network/mtprotostate.py

@@ -152,7 +152,7 @@ class MTProtoState:
         """
         Inverse of `encrypt_message_data` for incoming server messages.
         """
-        now = time.time() + self.time_offset  # get the time as early as possible, even if other checks make it go unused
+        now = time.time()  # get the time as early as possible, even if other checks make it go unused
 
         if len(body) < 8:
             raise InvalidBufferError(body)
@@ -203,9 +203,15 @@ class MTProtoState:
         # messages to change server_salt and notifications about invalid time on the client."
         #
         # This means we skip the time check for certain types of messages.
-        if obj.CONSTRUCTOR_ID not in (BadServerSalt.CONSTRUCTOR_ID, BadMsgNotification.CONSTRUCTOR_ID):
+        if obj.CONSTRUCTOR_ID in (BadServerSalt.CONSTRUCTOR_ID, BadMsgNotification.CONSTRUCTOR_ID):
+            if not self._highest_remote_id and not self.time_offset:
+                # If the first message we receive is a bad notification, take this opportunity
+                # to adjust the time offset. Assume it will remain stable afterwards. Updating
+                # the offset unconditionally would make the next checks pointless.
+                self.update_time_offset(remote_msg_id)
+        else:
             remote_msg_time = remote_msg_id >> 32
-            time_delta = now - remote_msg_time
+            time_delta = (now + self.time_offset) - remote_msg_time
 
             if time_delta > MSG_TOO_OLD_DELTA:
                 self._log.warning('Server sent a very old message with ID %d, ignoring (see FAQ for details)', remote_msg_id)

telethon/tl/core/gzippacked.py

@@ -1,4 +1,7 @@
-import gzip
+try:
+    from isal import igzip as gzip
+except ImportError:
+    import gzip
 import struct
 
 from .. import TLObject

telethon/tl/custom/button.py

@@ -37,11 +37,14 @@ class Button:
     to 128 characters and add the ellipsis (…) character as
     the 129.
     """
-    def __init__(self, button, *, resize, single_use, selective):
+    def __init__(self, button, *, resize, single_use, selective,
+                 persistent, placeholder):
         self.button = button
         self.resize = resize
         self.single_use = single_use
         self.selective = selective
+        self.persistent = persistent
+        self.placeholder = placeholder
 
     @staticmethod
     def _is_inline(button):
@@ -49,6 +52,7 @@ class Button:
         Returns `True` if the button belongs to an inline keyboard.
         """
         return isinstance(button, (
+            types.KeyboardButtonCopy,
             types.KeyboardButtonBuy,
             types.KeyboardButtonCallback,
             types.KeyboardButtonGame,
@@ -167,11 +171,15 @@ class Button:
         )
 
     @classmethod
-    def text(cls, text, *, resize=None, single_use=None, selective=None):
+    def text(cls, text, *, resize=None, single_use=None, selective=None,
+             persistent=None, placeholder=None):
         """
         Creates a new keyboard button with the given text.
 
         Args:
+            text (`str`):
+                The title of the button.
+
             resize (`bool`):
                 If present, the entire keyboard will be reconfigured to
                 be resized and be smaller if there are not many buttons.
@@ -186,48 +194,77 @@ class Button:
                 users. It will target users that are @mentioned in the text
                 of the message or to the sender of the message you reply to.
 
+            persistent (`bool`):
+                If present, always show the keyboard when the regular keyboard
+                is hidden. Defaults to false, in which case the custom keyboard
+                can be hidden and revealed via the keyboard icon.
+
+            placeholder (`str`):
+                The placeholder to be shown in the input field when the keyboard is active;
+                1-64 characters
+
         When the user clicks this button, a text message with the same text
         as the button will be sent, and can be handled with `events.NewMessage
         <telethon.events.newmessage.NewMessage>`. You cannot distinguish
         between a button press and the user typing and sending exactly the
         same text on their own.
         """
-        return cls(types.KeyboardButton(text),
-                   resize=resize, single_use=single_use, selective=selective)
+        return cls(
+            types.KeyboardButton(text),
+            resize=resize,
+            single_use=single_use,
+            selective=selective,
+            persistent=persistent,
+            placeholder=placeholder
+        )
 
     @classmethod
-    def request_location(cls, text, *,
-                         resize=None, single_use=None, selective=None):
+    def request_location(cls, text, *, resize=None, single_use=None, selective=None,
+                         persistent=None, placeholder=None):
         """
         Creates a new keyboard button to request the user's location on click.
 
-        ``resize``, ``single_use`` and ``selective`` are documented in `text`.
+        ``resize``, ``single_use``, ``selective``, ``persistent`` and ``placeholder``
+         are documented in `text`.
 
         When the user clicks this button, a confirmation box will be shown
         to the user asking whether they want to share their location with the
         bot, and if confirmed a message with geo media will be sent.
         """
-        return cls(types.KeyboardButtonRequestGeoLocation(text),
-                   resize=resize, single_use=single_use, selective=selective)
+        return cls(
+            types.KeyboardButtonRequestGeoLocation(text),
+            resize=resize,
+            single_use=single_use,
+            selective=selective,
+            persistent=persistent,
+            placeholder=placeholder
+        )
 
     @classmethod
-    def request_phone(cls, text, *,
-                      resize=None, single_use=None, selective=None):
+    def request_phone(cls, text, *, resize=None, single_use=None,
+                      selective=None, persistent=None, placeholder=None):
         """
         Creates a new keyboard button to request the user's phone on click.
 
-        ``resize``, ``single_use`` and ``selective`` are documented in `text`.
+        ``resize``, ``single_use``, ``selective``, ``persistent`` and ``placeholder``
+         are documented in `text`.
 
         When the user clicks this button, a confirmation box will be shown
         to the user asking whether they want to share their phone with the
         bot, and if confirmed a message with contact media will be sent.
         """
-        return cls(types.KeyboardButtonRequestPhone(text),
-                   resize=resize, single_use=single_use, selective=selective)
+        return cls(
+            types.KeyboardButtonRequestPhone(text),
+            resize=resize,
+            single_use=single_use,
+            selective=selective,
+            placeholder=placeholder,
+            persistent=persistent
+        )
 
     @classmethod
-    def request_poll(cls, text, *, force_quiz=False,
-                     resize=None, single_use=None, selective=None):
+    def request_poll(cls, text, *, force_quiz=False, resize=None, single_use=None,
+                     selective=None, persistent=None, placeholder=None):
         """
         Creates a new keyboard button to request the user to create a poll.
 
@@ -239,13 +276,20 @@ class Button:
         the votes cannot be retracted. Otherwise, users can vote and retract
         the vote, and the pol might be multiple choice.
 
-        ``resize``, ``single_use`` and ``selective`` are documented in `text`.
+        ``resize``, ``single_use``, ``selective``, ``persistent`` and ``placeholder``
+         are documented in `text`.
 
         When the user clicks this button, a screen letting the user create a
         poll will be shown, and if they do create one, the poll will be sent.
         """
-        return cls(types.KeyboardButtonRequestPoll(text, quiz=force_quiz),
-                   resize=resize, single_use=single_use, selective=selective)
+        return cls(
+            types.KeyboardButtonRequestPoll(text, quiz=force_quiz),
+            resize=resize,
+            single_use=single_use,
+            selective=selective,
+            persistent=persistent,
+            placeholder=placeholder
+        )
 
     @staticmethod
     def clear(selective=None):
@@ -264,15 +308,8 @@ class Button:
         Forces a reply to the message with this markup. If used,
         no other button should be present or it will be ignored.
 
-        ``single_use`` and ``selective`` are as documented in `text`.
+        ``single_use``, ``selective`` and ``placeholder`` are as documented in `text`.
 
-        Args:
-            placeholder (str):
-                text to show the user at typing place of message.
-
-                If the placeholder is too long, Telegram applications will
-                crop the text (for example, to 64 characters and adding an
-                ellipsis (…) character as the 65th).
         """
         return types.ReplyKeyboardForceReply(
             single_use=single_use,

telethon/tl/custom/inlinebuilder.py

@@ -391,7 +391,7 @@ class InlineBuilder:
                     'text geo contact game'.split(), args) if x[1]) or 'none')
             )
 
-        markup = self._client.build_reply_markup(buttons, inline_only=True)
+        markup = self._client.build_reply_markup(buttons)
         if text is not None:
             text, msg_entities = await self._client._parse_message_text(
                 text, parse_mode

telethon/tl/custom/message.py

@@ -96,7 +96,7 @@ class Message(ChatGetter, SenderGetter, TLObject):
             The ID of the bot used to send this message
             through its inline mode (e.g. "via @like").
 
-        reply_to (:tl:`MessageReplyHeader`):
+        reply_to (:tl:`MessageReplyHeader` | :tl:`MessageReplyStoryHeader`):
             The original reply header if this message is replying to another.
 
         date (`datetime`):
@@ -174,12 +174,54 @@ class Message(ChatGetter, SenderGetter, TLObject):
 
     def __init__(
         self,
-            id: int, peer_id: types.TypePeer,
-            date: Optional[datetime]=None, message: Optional[str]=None,
+        id: int,
+        peer_id: types.TypePeer,
+        date: Optional[datetime] = None,
+        message: Optional[str] = None,
         # Copied from Message.__init__ signature
-            out: Optional[bool]=None, mentioned: Optional[bool]=None, media_unread: Optional[bool]=None, silent: Optional[bool]=None, post: Optional[bool]=None, from_scheduled: Optional[bool]=None, legacy: Optional[bool]=None, edit_hide: Optional[bool]=None, pinned: Optional[bool]=None, noforwards: Optional[bool]=None, invert_media: Optional[bool]=None, offline: Optional[bool]=None, from_id: Optional[types.TypePeer]=None, from_boosts_applied: Optional[int]=None, saved_peer_id: Optional[types.TypePeer]=None, fwd_from: Optional[types.TypeMessageFwdHeader]=None, via_bot_id: Optional[int]=None, via_business_bot_id: Optional[int]=None, reply_to: Optional[types.TypeMessageReplyHeader]=None, media: Optional[types.TypeMessageMedia]=None, reply_markup: Optional[types.TypeReplyMarkup]=None, entities: Optional[List[types.TypeMessageEntity]]=None, views: Optional[int]=None, forwards: Optional[int]=None, replies: Optional[types.TypeMessageReplies]=None, edit_date: Optional[datetime]=None, post_author: Optional[str]=None, grouped_id: Optional[int]=None, reactions: Optional[types.TypeMessageReactions]=None, restriction_reason: Optional[List[types.TypeRestrictionReason]]=None, ttl_period: Optional[int]=None, quick_reply_shortcut_id: Optional[int]=None, effect: Optional[int]=None, factcheck: Optional[types.TypeFactCheck]=None,
+        out: Optional[bool] = None,
+        mentioned: Optional[bool] = None,
+        media_unread: Optional[bool] = None,
+        silent: Optional[bool] = None,
+        post: Optional[bool] = None,
+        from_scheduled: Optional[bool] = None,
+        legacy: Optional[bool] = None,
+        edit_hide: Optional[bool] = None,
+        pinned: Optional[bool] = None,
+        noforwards: Optional[bool] = None,
+        invert_media: Optional[bool] = None,
+        offline: Optional[bool] = None,
+        video_processing_pending: Optional[bool] = None,
+        paid_suggested_post_stars: Optional[bool] = None,
+        paid_suggested_post_ton: Optional[bool] = None,
+        from_id: Optional[types.TypePeer] = None,
+        from_boosts_applied: Optional[int] = None,
+        saved_peer_id: Optional[types.TypePeer] = None,
+        fwd_from: Optional[types.TypeMessageFwdHeader] = None,
+        via_bot_id: Optional[int] = None,
+        via_business_bot_id: Optional[int] = None,
+        reply_to: Optional[types.TypeMessageReplyHeader] = None,
+        media: Optional[types.TypeMessageMedia] = None,
+        reply_markup: Optional[types.TypeReplyMarkup] = None,
+        entities: Optional[List[types.TypeMessageEntity]] = None,
+        views: Optional[int] = None,
+        forwards: Optional[int] = None,
+        replies: Optional[types.TypeMessageReplies] = None,
+        edit_date: Optional[datetime] = None,
+        post_author: Optional[str] = None,
+        grouped_id: Optional[int] = None,
+        reactions: Optional[types.TypeMessageReactions] = None,
+        restriction_reason: Optional[List[types.TypeRestrictionReason]] = None,
+        ttl_period: Optional[int] = None,
+        quick_reply_shortcut_id: Optional[int] = None,
+        effect: Optional[int] = None,
+        factcheck: Optional[types.TypeFactCheck] = None,
+        report_delivery_until_date: Optional[datetime] = None,
+        paid_message_stars: Optional[int] = None,
+        suggested_post: Optional[types.TypeSuggestedPost] = None,
         # Copied from MessageService.__init__ signature
-            action: Optional[types.TypeMessageAction]=None
+        action: Optional[types.TypeMessageAction] = None,
+        reactions_are_possible: Optional[bool] = None,
     ):
         # Copied from Message.__init__ body
         self.id = id
@@ -198,6 +240,9 @@ class Message(ChatGetter, SenderGetter, TLObject):
         self.noforwards = noforwards
         self.invert_media = invert_media
         self.offline = offline
+        self.video_processing_pending = video_processing_pending
+        self.paid_suggested_post_stars = paid_suggested_post_stars
+        self.paid_suggested_post_ton = paid_suggested_post_ton
         self.from_id = from_id
         self.from_boosts_applied = from_boosts_applied
         self.saved_peer_id = saved_peer_id
@@ -220,8 +265,12 @@ class Message(ChatGetter, SenderGetter, TLObject):
         self.quick_reply_shortcut_id = quick_reply_shortcut_id
         self.effect = effect
         self.factcheck = factcheck
+        self.report_delivery_until_date = report_delivery_until_date
+        self.paid_message_stars = paid_message_stars
+        self.suggested_post = suggested_post
         # Copied from MessageService.__init__ body
         self.action = action
+        self.reactions_are_possible = reactions_are_possible
 
         # Convenient storage for custom functions
         # TODO This is becoming a bit of bloat
@@ -375,10 +424,11 @@ class Message(ChatGetter, SenderGetter, TLObject):
     @property
     def is_reply(self):
         """
-        `True` if the message is a reply to some other message.
+        `True` if the message is a reply to some other message or story.
 
-        Remember that you can access the ID of the message
-        this one is replying to through `reply_to.reply_to_msg_id`,
+        Remember that if the replied-to is a message, 
+        you can access the ID of the message this one is 
+        replying to through `reply_to.reply_to_msg_id`,
         and the `Message` object with `get_reply_message()`.
         """
         return self.reply_to is not None
@@ -667,7 +717,11 @@ class Message(ChatGetter, SenderGetter, TLObject):
         Returns the message ID this message is replying to, if any.
         This is equivalent to accessing ``.reply_to.reply_to_msg_id``.
         """
-        return self.reply_to.reply_to_msg_id if self.reply_to else None
+        return (
+            self.reply_to.reply_to_msg_id
+            if isinstance(self.reply_to, types.MessageReplyHeader)
+            else None
+        )
 
     @property
     def to_id(self):
@@ -733,7 +787,7 @@ class Message(ChatGetter, SenderGetter, TLObject):
         The result will be cached after its first use.
         """
         if self._reply_message is None and self._client:
-            if not self.reply_to:
+            if not isinstance(self.reply_to, types.MessageReplyHeader):
                 return None
 
             # Bots cannot access other bots' messages by their ID.
@@ -867,7 +921,7 @@ class Message(ChatGetter, SenderGetter, TLObject):
 
     async def click(self, i=None, j=None,
                     *, text=None, filter=None, data=None, share_phone=None,
-                    share_geo=None, password=None):
+                    share_geo=None, password=None, open_url=None):
         """
         Calls :tl:`SendVote` with the specified poll option
         or `button.click <telethon.tl.custom.messagebutton.MessageButton.click>`
@@ -952,6 +1006,12 @@ class Message(ChatGetter, SenderGetter, TLObject):
                 you need to provide your account's password. Otherwise,
                 `teltehon.errors.PasswordHashInvalidError` is raised.
             
+            open_url (`bool`):
+                When clicking on an inline keyboard URL button :tl:`KeyboardButtonUrl`
+                By default it will return URL of the button, passing ``click(open_url=True)``
+                will lunch the default browser with given URL of the button and 
+                return `True` on success.
+                
             Example:
 
                 .. code-block:: python
@@ -981,7 +1041,7 @@ class Message(ChatGetter, SenderGetter, TLObject):
 
             but = types.KeyboardButtonCallback('', data)
             return await MessageButton(self._client, but, chat, None, self.id).click(
-                share_phone=share_phone, share_geo=share_geo, password=password)
+                share_phone=share_phone, share_geo=share_geo, password=password, open_url=open_url)
 
         if sum(int(x is not None) for x in (i, text, filter)) >= 2:
             raise ValueError('You can only set either of i, text or filter')
@@ -1054,7 +1114,7 @@ class Message(ChatGetter, SenderGetter, TLObject):
         button = find_button()
         if button:
             return await button.click(
-                share_phone=share_phone, share_geo=share_geo, password=password)
+                share_phone=share_phone, share_geo=share_geo, password=password, open_url=open_url)
 
     async def mark_read(self):
         """

telethon/tl/custom/messagebutton.py

@@ -65,7 +65,7 @@ class MessageButton:
         if isinstance(self.button, types.KeyboardButtonUrl):
             return self.button.url
 
-    async def click(self, share_phone=None, share_geo=None, *, password=None):
+    async def click(self, share_phone=None, share_geo=None, *, password=None, open_url=None):
         """
         Emulates the behaviour of clicking this button.
 
@@ -79,7 +79,8 @@ class MessageButton:
         :tl:`StartBotRequest` will be invoked and the resulting updates
         returned.
 
-        If it's a :tl:`KeyboardButtonUrl`, the URL of the button will
+        If it's a :tl:`KeyboardButtonUrl`, the ``URL`` of the button will
+        be returned. If you pass ``open_url=True`` the URL of the button will
         be passed to ``webbrowser.open`` and return `True` on success.
 
         If it's a :tl:`KeyboardButtonRequestPhone`, you must indicate that you
@@ -116,8 +117,10 @@ class MessageButton:
                 bot=self._bot, peer=self._chat, start_param=self.button.query
             ))
         elif isinstance(self.button, types.KeyboardButtonUrl):
+            if open_url:
                 if "webbrowser" in sys.modules:
                     return webbrowser.open(self.button.url)
+            return self.button.url
         elif isinstance(self.button, types.KeyboardButtonGame):
             req = functions.messages.GetBotCallbackAnswerRequest(
                 peer=self._chat, msg_id=self._msg_id, game=True

telethon/utils.py

@@ -14,6 +14,7 @@ import os
 import pathlib
 import re
 import struct
+import warnings
 from collections import namedtuple
 from mimetypes import guess_extension
 from types import GeneratorType
@@ -95,7 +96,8 @@ def get_display_name(entity):
         else:
             return ''
 
-    elif isinstance(entity, (types.Chat, types.ChatForbidden, types.Channel)):
+    elif isinstance(entity, (
+            types.Chat, types.ChatForbidden, types.Channel, types.ChannelForbidden)):
         return entity.title
 
     return ''
@@ -436,15 +438,16 @@ def get_input_media(
         if media.SUBCLASS_OF_ID == 0xfaf846f4:  # crc32(b'InputMedia')
             return media
         elif media.SUBCLASS_OF_ID == 0x846363e0:  # crc32(b'InputPhoto')
-            return types.InputMediaPhoto(media, ttl_seconds=ttl)
+            return types.InputMediaPhoto(media, ttl_seconds=ttl, spoiler=media.spoiler)
         elif media.SUBCLASS_OF_ID == 0xf33fdb68:  # crc32(b'InputDocument')
-            return types.InputMediaDocument(media, ttl_seconds=ttl)
+            return types.InputMediaDocument(media, ttl_seconds=ttl, spoiler=media.spoiler)
     except AttributeError:
         _raise_cast_fail(media, 'InputMedia')
 
     if isinstance(media, types.MessageMediaPhoto):
         return types.InputMediaPhoto(
             id=get_input_photo(media.photo),
+            spoiler=media.spoiler,
             ttl_seconds=ttl or media.ttl_seconds
         )
 
@@ -499,6 +502,14 @@ def get_input_media(
     if isinstance(media, types.MessageMediaGeo):
         return types.InputMediaGeoPoint(geo_point=get_input_geo(media.geo))
 
+    if isinstance(media, types.MessageMediaGeoLive):
+        return types.InputMediaGeoLive(
+            geo_point=get_input_geo(media.geo),
+            period=media.period,
+            heading=media.heading,
+            proximity_notification_radius=media.proximity_notification_radius,
+        )
+
     if isinstance(media, types.MessageMediaVenue):
         return types.InputMediaVenue(
             geo_point=get_input_geo(media.geo),
@@ -600,6 +611,9 @@ def get_message_id(message):
     if isinstance(message, int):
         return message
 
+    if isinstance(message, types.InputMessageID):
+        return message.id
+
     try:
         if message.SUBCLASS_OF_ID == 0x790009e3:
             # hex(crc32(b'Message')) = 0x790009e3
@@ -896,7 +910,7 @@ def is_list_like(obj):
     enough. Things like ``open()`` are also iterable (and probably many
     other things), so just support the commonly known list-like objects.
     """
-    return isinstance(obj, (list, tuple, set, dict, GeneratorType))
+    return isinstance(obj, (list, tuple, set, dict, range, GeneratorType))
 
 
 def parse_phone(phone):
@@ -1544,3 +1558,11 @@ def _photo_size_byte_count(size):
         return max(size.sizes)
     else:
         return None
+
+
+async def maybe_async(coro):
+    result = coro
+    if inspect.isawaitable(result):
+        warnings.warn('Using async sessions support is an experimental feature')
+        result = await result
+    return result

telethon/version.py

@@ -1,3 +1,3 @@
 # Versions should comply with PEP440.
 # This line is parsed in setup.py:
-__version__ = '1.37.0'
+__version__ = '1.40.0'

telethon_generator/data/errors.csv

@@ -525,3 +525,5 @@ WEBPUSH_KEY_INVALID,400,The specified web push elliptic curve Diffie-Hellman pub
 WEBPUSH_TOKEN_INVALID,400,The specified web push token is invalid
 WORKER_BUSY_TOO_LONG_RETRY,500,Telegram workers are too busy to respond immediately
 YOU_BLOCKED_USER,400,You blocked this user
+FROZEN_METHOD_INVALID,420,You tried to use a method that is not available for frozen accounts
+FROZEN_PARTICIPANT_MISSING,400,Your account is frozen and can't access the chat

tests/readthedocs/quick_references/test_client_reference.py

@@ -10,5 +10,5 @@ def test_all_methods_present(docs_dir):
     assert len(present_methods) > 0
     for name in dir(TelegramClient):
         attr = getattr(TelegramClient, name)
-        if callable(attr) and not name.startswith('_'):
+        if callable(attr) and not name.startswith('_') and name != 'sign_up':
             assert name in present_methods

tests/telethon/client/test_messages.py

@@ -1,8 +1,12 @@
 import inspect
+from unittest import mock
+from unittest.mock import MagicMock
 
 import pytest
 
 from telethon import TelegramClient
+from telethon.client import MessageMethods
+from telethon.tl.types import PeerChat, MessageMediaDocument, Message, MessageEntityBold
 
 
 @pytest.mark.asyncio
@@ -38,3 +42,43 @@ async def test_send_message_with_file_forwards_args():
 
     client = MockedClient()
     assert (await client.send_message('a', file='b', **arguments)) == sentinel
+
+
+class TestMessageMethods:
+    @pytest.mark.asyncio
+    @pytest.mark.parametrize(
+        'formatting_entities',
+        ([MessageEntityBold(offset=0, length=0)], None)
+    )
+    async def test_send_msg_and_file(self, formatting_entities):
+        async def async_func(result): # AsyncMock was added only in 3.8
+            return result
+        msg_methods = MessageMethods()
+        expected_result = Message(
+            id=0, peer_id=PeerChat(chat_id=0), message='', date=None,
+        )
+        entity = 'test_entity'
+        message = Message(
+            id=1, peer_id=PeerChat(chat_id=0), message='expected_caption', date=None,
+            entities=[MessageEntityBold(offset=9, length=9)],
+        )
+        media_file = MessageMediaDocument()
+
+        with mock.patch.object(
+            target=MessageMethods, attribute='send_file',
+            new=MagicMock(return_value=async_func(expected_result)), create=True,
+        ) as mock_obj:
+            result = await msg_methods.send_message(
+                entity=entity, message=message, file=media_file,
+                formatting_entities=formatting_entities,
+            )
+            mock_obj.assert_called_once_with(
+                entity, media_file, caption=message.message,
+                formatting_entities=formatting_entities or message.entities,
+                reply_to=None, silent=None, attributes=None, parse_mode=(),
+                force_document=False, thumb=None, buttons=None,
+                clear_draft=False, schedule=None, supports_streaming=False,
+                comment_to=None, background=None, nosound_video=None,
+                send_as=None, message_effect_id=None,
+            )
+            assert result == expected_result

tests/telethon/test_utils.py

@@ -34,9 +34,6 @@ def test_private_get_extension():
 
     assert utils._get_extension('foo.bar.baz') == '.baz'
     assert utils._get_extension(pathlib.Path('foo.bar.baz')) == '.baz'
-    assert utils._get_extension(png_header) == '.png'
-    assert utils._get_extension(png_buffer) == '.png'
-    assert utils._get_extension(png_buffer) == '.png'  # make sure it did seek back
     assert utils._get_extension(CustomFd('foo.bar.baz')) == '.baz'
 
     # Negative cases