Update to layer 211

There no plans for this to ever be non-experimental in v1.

.github/ISSUE_TEMPLATE/bug-report.md

@@ -1,29 +0,0 @@
----
-name: Bug Report
-about: Create a report about a bug inside the library or issues with the documentation
-title: ''
-labels: ''
-assignees: ''
-
----
-
-**Checklist**
-
-<!-- Put x inside the boxes (like [x]) to mark them as complete (but only if you've actually completed them!) -->
-* [ ] The error is in the library's code, and not in my own.
-* [ ] I have searched for this issue before posting it and there isn't a duplicate.
-* [ ] I ran `pip install -U https://github.com/LonamiWebs/Telethon/archive/v1.zip` and triggered the bug in the latest version.
-
-**Code that causes the issue**
-```python
-from telethon.sync import TelegramClient
-...
-
-```
-
-**Traceback**
-```
-Traceback (most recent call last):
-  File "code.py", line 1, in <code>
-
-```

.github/ISSUE_TEMPLATE/bug-report.yml

@@ -0,0 +1,96 @@
+name: Bug Report
+description: Create a report about a bug inside the library.
+body:
+
+  - type: textarea
+    id: reproducing-example
+    attributes:
+      label: Code that causes the issue
+      description: Provide a code example that reproduces the problem. Try to keep it short without other dependencies.
+      placeholder: |
+        ```python
+        from telethon.sync import TelegramClient
+        ...
+
+        ```
+    validations:
+      required: true
+
+  - type: textarea
+    id: expected-behavior
+    attributes:
+      label: Expected behavior
+      description: Explain what you should expect to happen. Include reproduction steps.
+      placeholder: |
+        "I was doing... I was expecting the following to happen..."
+    validations:
+      required: true
+
+  - type: textarea
+    id: actual-behavior
+    attributes:
+      label: Actual behavior
+      description: Explain what actually happens.
+      placeholder: |
+        "This happened instead..."
+    validations:
+      required: true
+
+  - type: textarea
+    id: traceback
+    attributes:
+      label: Traceback
+      description: |
+        The traceback, if the problem is a crash.
+      placeholder: |
+        ```
+        Traceback (most recent call last):
+          File "code.py", line 1, in <code>
+
+        ```
+
+  - type: input
+    id: telethon-version
+    attributes:
+      label: Telethon version
+      description: The output of `python -c "import telethon; print(telethon.__version__)"`.
+      placeholder: "1.x"
+    validations:
+      required: true
+
+  - type: input
+    id: python-version
+    attributes:
+      label: Python version
+      description: The output of `python --version`.
+      placeholder: "3.x"
+    validations:
+      required: true
+
+  - type: input
+    id: os
+    attributes:
+      label: Operating system (including distribution name and version)
+      placeholder: Windows 11, macOS 13.4, Ubuntu 23.04...
+    validations:
+      required: true
+
+  - type: textarea
+    id: other-details
+    attributes:
+      label: Other details
+      placeholder: |
+        Additional details and attachments. Is it a server? Network condition?
+
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: Read this carefully, we will close and ignore your issue if you skimmed through this.
+      options:
+        - label: The error is in the library's code, and not in my own.
+          required: true
+        - label: I have searched for this issue before posting it and there isn't an open duplicate.
+          required: true
+        - label: I ran `pip install -U https://github.com/LonamiWebs/Telethon/archive/v1.zip` and triggered the bug in the latest version.
+          required: true

.github/ISSUE_TEMPLATE/config.yml

@@ -1,3 +1,4 @@
+blank_issues_enabled: false
 contact_links:
   - name: Ask questions in StackOverflow
     url: https://stackoverflow.com/questions/ask?tags=telethon

.github/ISSUE_TEMPLATE/documentation-issue.yml

@@ -0,0 +1,22 @@
+name: Documentation Issue
+description: Report a problem with the documentation.
+labels: [documentation]
+body:
+
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: Describe the problem in detail.
+      placeholder: This part is unclear...
+
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: Read this carefully, we will close and ignore your issue if you skimmed through this.
+      options:
+        - label: This is a documentation problem, not a question or a bug report.
+          required: true
+        - label: I have searched for this issue before posting it and there isn't a duplicate.
+          required: true

.github/ISSUE_TEMPLATE/feature-request.md

@@ -1,10 +0,0 @@
----
-name: Feature Request
-about: Suggest ideas, changes or other enhancements for the library
-title: ''
-labels: enhancement
-assignees: ''
-
----
-
-Please describe your idea. Would you like another friendly method? Renaming them to something more appropriated? Changing the way something works?

.github/ISSUE_TEMPLATE/feature-request.yml

@@ -0,0 +1,22 @@
+name: Feature Request
+description: Suggest ideas, changes or other enhancements for the library.
+labels: [enhancement]
+body:
+
+  - type: textarea
+    id: feature-description
+    attributes:
+      label: Describe your suggested feature
+      description: Please describe your idea. Would you like another friendly method? Renaming them to something more appropriate? Changing the way something works?
+      placeholder: "It should work like this..."
+    validations:
+      required: true
+
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: Read this carefully, we will close and ignore your issue if you skimmed through this.
+      options:
+        - label: I have searched for this issue before posting it and there isn't a duplicate.
+          required: true

.github/pull_request_template.md

@@ -0,0 +1,5 @@
+<!--
+Thanks for the PR! Please keep in mind that v1 is *feature frozen*.
+New features very likely won't be merged, although fixes can be sent.
+All new development should happen in v2. Thanks!
+-->

.readthedocs.yaml

@@ -0,0 +1,18 @@
+# https://docs.readthedocs.io/en/stable/config-file/v2.html
+version: 2
+
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.11"
+
+sphinx:
+  configuration: readthedocs/conf.py
+
+formats:
+  - pdf
+  - epub
+
+python:
+  install:
+    - requirements: readthedocs/requirements.txt

README.rst

@@ -12,6 +12,8 @@ as a user or through a bot account (bot API alternative).
 
     If you have code using Telethon before its 1.0 version, you must
     read `Compatibility and Convenience`_ to learn how to migrate.
+    As with any third-party library for Telegram, be careful not to
+    break `Telegram's ToS`_ or `Telegram can ban the account`_.
 
 What is this?
 -------------
@@ -76,6 +78,8 @@ useful information.
 .. _MTProto: https://core.telegram.org/mtproto
 .. _Telegram: https://telegram.org
 .. _Compatibility and Convenience: https://docs.telethon.dev/en/stable/misc/compatibility-and-convenience.html
+.. _Telegram's ToS: https://core.telegram.org/api/terms
+.. _Telegram can ban the account: https://docs.telethon.dev/en/stable/quick-references/faq.html#my-account-was-deleted-limited-when-using-the-library
 .. _Read The Docs: https://docs.telethon.dev
 
 .. |logo| image:: logo.svg

optional-requirements.txt

@@ -3,3 +3,4 @@ pysocks
 python-socks[asyncio]
 hachoir
 pillow
+isal

readthedocs/basic/updates.rst

@@ -16,7 +16,7 @@ For that, you can use **events**.
     .. code-block:: python
 
         import logging
-        logging.basicConfig(format='[%(levelname) 5s/%(asctime)s] %(name)s: %(message)s',
+        logging.basicConfig(format='[%(levelname) %(asctime)s] %(name)s: %(message)s',
                             level=logging.WARNING)
 
 

readthedocs/concepts/asyncio.rst

@@ -40,22 +40,22 @@ because tasks are smaller than threads, which are smaller than processes.
 What are asyncio basics?
 ========================
 
+The code samples below assume that you have Python 3.7 or greater installed.
+
 .. code-block:: python
 
     # First we need the asyncio library
     import asyncio
 
-    # Then we need a loop to work with
-    loop = asyncio.get_event_loop()
-
     # We also need something to run
     async def main():
         for char in 'Hello, world!\n':
             print(char, end='', flush=True)
             await asyncio.sleep(0.2)
 
-    # Then, we need to run the loop with a task
+    # Then, we can create a new asyncio loop and use it to run our coroutine.
-    loop.run_until_complete(main())
+    # The creation and tear-down of the loop is hidden away from us.
+    asyncio.run(main())
 
 
 What does telethon.sync do?
@@ -101,7 +101,7 @@ Instead of this:
 
     # or, using asyncio's default loop (it's the same)
     import asyncio
-    loop = asyncio.get_event_loop()  # == client.loop
+    loop = asyncio.get_running_loop()  # == client.loop
     me = loop.run_until_complete(client.get_me())
     print(me.username)
 
@@ -158,13 +158,10 @@ loops or use ``async with``:
 
                 print(message.sender.username)
 
-    loop = asyncio.get_event_loop()
+    asyncio.run(main())
-    # ^ this assigns the default event loop from the main thread to a variable
+    # ^ this will create a new asyncio loop behind the scenes and tear it down
-
+    #   once the function returns. It will run the loop untiil main finishes.
-    loop.run_until_complete(main())
+    #   You should only use this function if there is no other loop running.
-    # ^ this runs the *entire* loop until the main() function finishes.
-    #   While the main() function does not finish, the loop will be running.
-    #   While the loop is running, you can't run it again.
 
 
 The ``await`` keyword blocks the *current* task, and the loop can run
@@ -184,14 +181,14 @@ concurrently:
         await asyncio.sleep(delay)  # await tells the loop this task is "busy"
         print('world')  # eventually the loop finishes all tasks
 
-    loop = asyncio.get_event_loop()  # get the default loop for the main thread
+    async def main():
-    loop.create_task(world(2))  # create the world task, passing 2 as delay
+        asyncio.create_task(world(2))  # create the world task, passing 2 as delay
-    loop.create_task(hello(delay=1))  # another task, but with delay 1
+        asyncio.create_task(hello(delay=1))  # another task, but with delay 1
+        await asyncio.sleep(3)  # wait for three seconds before exiting
+
     try:
-        # run the event loop forever; ctrl+c to stop it
+        # create a new temporary asyncio loop and use it to run main
-        # we could also run the loop for three seconds:
+        asyncio.run(main())
-        #     loop.run_until_complete(asyncio.sleep(3))
-        loop.run_forever()
     except KeyboardInterrupt:
         pass
 
@@ -209,10 +206,15 @@ The same example, but without the comment noise:
         await asyncio.sleep(delay)
         print('world')
 
-    loop = asyncio.get_event_loop()
+    async def main():
-    loop.create_task(world(2))
+        asyncio.create_task(world(2))
-    loop.create_task(hello(1))
+        asyncio.create_task(hello(delay=1))
-    loop.run_until_complete(asyncio.sleep(3))
+        await asyncio.sleep(3)
+
+    try:
+        asyncio.run(main())
+    except KeyboardInterrupt:
+        pass
 
 
 Can I use threads?
@@ -250,9 +252,9 @@ You may have seen this error:
 
     RuntimeError: There is no current event loop in thread 'Thread-1'.
 
-It just means you didn't create a loop for that thread, and if you don't
+It just means you didn't create a loop for that thread. Please refer to
-pass a loop when creating the client, it uses ``asyncio.get_event_loop()``,
+the ``asyncio`` documentation to correctly learn how to set the event loop
-which only works in the main thread.
+for non-main threads.
 
 
 client.run_until_disconnected() blocks!

readthedocs/concepts/botapi-vs-mtproto.rst

@@ -28,6 +28,9 @@ their own Telegram bots. Quoting their main page:
 Bot API is simply an HTTP endpoint which translates your requests to it into
 MTProto calls through tdlib_, their bot backend.
 
+Configuration of your bot, such as its available commands and auto-completion,
+is configured through `@BotFather <https://t.me/BotFather>`_.
+
 
 What is MTProto?
 ================
@@ -296,7 +299,7 @@ After rewriting:
 
     class Subbot(TelegramClient):
         def __init__(self, *a, **kw):
-            await super().__init__(*a, **kw)
+            super().__init__(*a, **kw)
             self.add_event_handler(self.on_update, events.NewMessage)
 
         async def connect():

readthedocs/concepts/entities.rst

@@ -268,7 +268,7 @@ That means you can do this:
 .. code-block:: python
 
     message.user_id
-    await message.get_input_user()
+    await message.get_input_sender()
     message.user
     # ...etc
 

readthedocs/concepts/full-api.rst

@@ -10,13 +10,20 @@ The Full API
     methods listed on :ref:`client-ref` unless you have a better reason
     not to, like a method not existing or you wanting more control.
 
+.. contents::
+
+
+Introduction
+============
 
 The :ref:`telethon-client` doesn't offer a method for every single request
 the Telegram API supports. However, it's very simple to *call* or *invoke*
-any request. Whenever you need something, don't forget to `check the documentation`_
+any request defined in Telegram's API.
-and look for the `method you need`_. There you can go through a sorted list
-of everything you can do.
 
+This section will teach you how to use what Telethon calls the `TL reference`_.
+The linked page contains a list and a way to search through *all* types
+generated from the definition of Telegram's API (in ``.tl`` file format,
+hence the name). These types include requests and constructors.
 
 .. note::
 
@@ -25,10 +32,193 @@ of everything you can do.
     as you type, and a "Copy import" button. If you like namespaces, you
     can also do ``from telethon.tl import types, functions``. Both work.
 
+Telegram makes these ``.tl`` files public, which other implementations, such
+as Telethon, can also use to generate code. These files are versioned under
+what's called "layers". ``.tl`` files consist of thousands of definitions,
+and newer layers often add, change, or remove them. Each definition refers
+to either a Remote Procedure Call (RPC) function, or a type (which the
+`TL reference`_ calls "constructors", as they construct particular type
+instances).
 
-You should also refer to the documentation to see what the objects
+As such, the `TL reference`_ is a good place to go to learn about all possible
-(constructors) Telegram returns look like. Every constructor inherits
+requests, types, and what they look like. If you're curious about what's been
-from a common type, and that's the reason for this distinction.
+changed between layers, you can refer to the `TL diff`_ site.
+
+
+Navigating the TL reference
+===========================
+
+Functions
+---------
+
+"Functions" is the term used for the Remote Procedure Calls (RPC) that can be
+sent to Telegram to ask it to perform something (e.g. "send message"). These
+requests have an associated return type. These can be invoked ("called"):
+
+.. code-block:: python
+
+    client = TelegramClient(...)
+    function_instance = SomeRequest(...)
+
+    # Invoke the request
+    returned_type = await client(function_instance)
+
+Whenever you find the type for a function in the `TL reference`_, the page
+will contain the following information:
+
+* What type of account can use the method. This information is regenerated
+  from time to time (by attempting to invoke the function under both account
+  types and finding out where it fails). Some requests can only be used by
+  bot accounts, others by user accounts, and others by both.
+* The TL definition. This helps you get a feel for the what the function
+  looks like. This is not Python code. It just contains the definition in
+  a concise manner.
+* "Copy import" button. Does what it says: it will copy the necessary Python
+  code to import the function to your system's clipboard for easy access.
+* Returns. The returned type. When you invoke the function, this is what the
+  result will be. It also includes which of the constructors can be returned
+  inline, to save you a click.
+* Parameters. The parameters accepted by the function, including their type,
+  whether they expect a list, and whether they're optional.
+* Known RPC errors. A best-effort list of known errors the request may cause.
+  This list is not complete and may be out of date, but should provide an
+  overview of what could go wrong.
+* Example. Autogenerated example, showcasing how you may want to call it.
+  Bear in mind that this is *autogenerated*. It may be spitting out non-sense.
+  The goal of this example is not to show you everything you can do with the
+  request, only to give you a feel for what it looks like to use it.
+
+It is very important to click through the links and navigate to get the full
+picture. A specific page will show you what the specific function returns and
+needs as input parameters. But it may reference other types, so you need to
+navigate to those to learn what those contain or need.
+
+Types
+-----
+
+"Types" as understood by TL are not actually generated in Telethon.
+They would be the "abstract base class" of the constructors, but since Python
+is duck-typed, there is hardly any need to generate mostly unnecessary code.
+The page for a type contains:
+
+* Constructors. Every type will have one or more constructors. These
+  constructors *are* generated and can be immported and used.
+* Requests returning this type. A helpful way to find out "what requests can
+  return this?". This is how you may learn what request you need to use to
+  obtain a particular instance of a type.
+* Requests accepting this type as input. A helpful way to find out "what
+  requests can use this type as one of their input parameters?". This is how
+  you may learn where a type is used.
+* Other types containing this type. A helpful way to find out "where else
+  does this type appear?". This is how you can walk back through nested
+  objects.
+
+Constructors
+------------
+
+Constructors are used to create instances of a particular type, and are also
+returned when invoking requests. You will have to create instances yourself
+when invoking requests that need a particular type as input.
+The page for a constructor contains:
+
+* Belongs to. The parent type. This is a link back to the types page for the
+  specific constructor. It also contains the sibling constructors inline, to
+  save you a click.
+* Members. Both the input parameters *and* fields the constructor contains.
+
+
+Using the TL reference
+======================
+
+After you've found a request you want to send, a good start would be to simply
+copy and paste the autogenerated example into your script. Then you can simply
+tweak it to your needs.
+
+If you want to do it from scratch, first, make sure to import the request into
+your code (either using the "Copy import" button near the top, or by manually
+spelling out the package under ``telethon.tl.functions.*``).
+
+Then, start reading the parameters one by one. If the parameter cannot be
+omitted, you **will** need to specify it, so make sure to spell it out as
+an input parameter when constructing the request instance. Let's look at
+`PingRequest`_ for example. First, we copy the import:
+
+.. code-block:: python
+
+    from telethon.tl.functions import PingRequest
+
+Then, we look at the parameters:
+
+    ping_id - long
+
+A single parameter, and it's a long (a integer number with a large range of
+values). It doesn't say it can be omitted, so we must provide it, like so:
+
+.. code-block:: python
+
+    PingRequest(
+        ping_id=48641868471
+    )
+
+(In this case, the ping ID is a random number. You often have to guess what
+the parameter needs just by looking at the name.)
+
+Now that we have our request, we can invoke it:
+
+.. code-block:: python
+
+    response = await client(PingRequest(
+        ping_id=48641868471
+    ))
+
+To find out what ``response`` looks like, we can do as the autogenerated
+example suggests and "stringify" the result as a pretty-printed string:
+
+.. code-block:: python
+
+    print(result.stringify())
+
+This will print out the following:
+
+.. code-block:: python
+
+    Pong(
+        msg_id=781875678118,
+        ping_id=48641868471
+    )
+
+Which is a very easy way to get a feel for a response. You should nearly
+always print the stringified result, at least once, when trying out requests,
+to get a feel for what the response may look like.
+
+But of course, you don't need to do that. Without writing any code, you could
+have navigated through the "Returns" link to learn ``PingRequest`` returns a
+``Pong``, which only has one constructor, and the constructor has two members,
+``msg_id`` and ``ping_id``.
+
+If you wanted to create your own ``Pong``, you would use both members as input
+parameters:
+
+.. code-block:: python
+
+    my_pong = Pong(
+        msg_id=781875678118,
+        ping_id=48641868471
+    )
+
+(Yes, constructing object instances can use the same code that ``.stringify``
+would return!)
+
+And if you wanted to access the ``msg_id`` member, you would simply access it
+like any other attribute access in Python:
+
+.. code-block:: python
+
+    print(response.msg_id)
+
+
+Example walkthrough
+===================
 
 Say `client.send_message()
 <telethon.client.messages.MessageMethods.send_message>` didn't exist,
@@ -224,6 +414,7 @@ and still access the successful results:
         # The second request failed.
         second = e.exceptions[1]
 
-.. _check the documentation: https://tl.telethon.dev
+.. _TL reference: https://tl.telethon.dev
-.. _method you need: https://tl.telethon.dev/methods/index.html
+.. _TL diff: https://diff.telethon.dev
+.. _PingRequest: https://tl.telethon.dev/methods/ping.html
 .. _use the search: https://tl.telethon.dev/?q=message&redirect=no

readthedocs/concepts/sessions.rst

@@ -143,7 +143,7 @@ output (likely your terminal).
 .. warning::
 
     **Keep this string safe!** Anyone with this string can use it
-    to login into your account and do anything they want to to do.
+    to login into your account and do anything they want to.
 
     This is similar to leaking your ``*.session`` files online,
     but it is easier to leak a string than it is to leak a file.

readthedocs/concepts/updates.rst

@@ -191,8 +191,7 @@ so the code above and the following are equivalent:
     async def main():
         await client.disconnected
 
-    loop = asyncio.get_event_loop()
+    asyncio.run(main())
-    loop.run_until_complete(main())
 
 
 You could also run `client.disconnected
@@ -207,7 +206,7 @@ Notice that unlike `client.disconnected
 <telethon.client.telegrambaseclient.TelegramBaseClient.disconnected>`,
 `client.run_until_disconnected
 <telethon.client.updates.UpdateMethods.run_until_disconnected>` will
-handle ``KeyboardInterrupt`` with you. This method is special and can
+handle ``KeyboardInterrupt`` for you. This method is special and can
 also be ran while the loop is running, so you can do this:
 
 .. code-block:: python

readthedocs/conf.py

@@ -85,7 +85,7 @@ release = version
 #
 # This is also used if you do content translation via gettext catalogs.
 # Usually you set "language" from the command line for these cases.
-language = None
+language = 'en'
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.

readthedocs/examples/working-with-messages.rst

@@ -2,89 +2,12 @@
 Working with messages
 =====================
 
-
 .. note::
 
     These examples assume you have read :ref:`full-api`.
 
-.. contents::
+This section has been `moved to the wiki`_, where it can be easily edited as new
+features arrive and the API changes. Please refer to the linked page to learn how
+to send spoilers, custom emoji, stickers, react to messages, and more things.
 
-
+.. _moved to the wiki: https://github.com/LonamiWebs/Telethon/wiki/Sending-more-than-just-messages
-Sending stickers
-================
-
-Stickers are nothing else than ``files``, and when you successfully retrieve
-the stickers for a certain sticker set, all you will have are ``handles`` to
-these files. Remember, the files Telegram holds on their servers can be
-referenced through this pair of ID/hash (unique per user), and you need to
-use this handle when sending a "document" message. This working example will
-send yourself the very first sticker you have:
-
-.. code-block:: python
-
-    # Get all the sticker sets this user has
-    from telethon.tl.functions.messages import GetAllStickersRequest
-    sticker_sets = await client(GetAllStickersRequest(0))
-
-    # Choose a sticker set
-    from telethon.tl.functions.messages import GetStickerSetRequest
-    from telethon.tl.types import InputStickerSetID
-    sticker_set = sticker_sets.sets[0]
-
-    # Get the stickers for this sticker set
-    stickers = await client(GetStickerSetRequest(
-        stickerset=InputStickerSetID(
-            id=sticker_set.id, access_hash=sticker_set.access_hash
-        )
-    ))
-
-    # Stickers are nothing more than files, so send that
-    await client.send_file('me', stickers.documents[0])
-
-
-Sending reactions
-=================
-
-It works very similar to replying to a message. You need to specify the chat,
-message ID you wish to react to, and reaction, using :tl:`SendReaction`:
-
-.. code-block:: python
-
-    from telethon.tl.functions.messages import SendReactionRequest
-
-    await client(SendReactionRequest(
-        peer=chat,
-        msg_id=42,
-        reaction='❤️'
-    ))
-
-Note that you cannot use strings like ``:heart:`` for the reaction. You must
-use the desired emoji directly. You can most easily achieve this by
-copy-pasting the emoji from an official application such as Telegram Desktop.
-
-If for some reason you cannot embed emoji directly into the code, you can also
-use its unicode escape (which you can find using websites like
-`unicode-table.com`_), or install a different package, like `emoji`_:
-
-.. code-block:: python
-
-    # All of these work exactly the same (you only need one):
-    import emoji
-    reaction = emoji.emojize(':red_heart:')
-    reaction = '❤️'
-    reaction = '\u2764'
-
-    from telethon.tl.functions.messages import SendReactionRequest
-    await client(SendReactionRequest(
-        peer=chat,
-        msg_id=42,
-        reaction=reaction
-    ))
-
-Please make sure to check the help pages of the respective websites you use
-if you need a more in-depth explanation on how they work. Telethon only needs
-you to provide the emoji in some form. Some packages or websites can make this
-easier.
-
-.. _unicode-table.com: https://unicode-table.com/en/emoji/
-.. _emoji: https://pypi.org/project/emoji/

readthedocs/index.rst

@@ -103,7 +103,6 @@ You can also use the menu on the left to quickly skip over sections.
     :caption: Miscellaneous
 
     misc/changelog
-    misc/wall-of-shame.rst
     misc/compatibility-and-convenience
 
 .. toctree::

readthedocs/misc/changelog.rst

@@ -13,6 +13,430 @@ it can take advantage of new goodies!
 
 .. contents:: List of All Versions
 
+New layer (v1.40)
+=================
+
++------------------------+
+| Scheme layer used: 201 |
++------------------------+
+
+`View new and changed raw API methods <https://diff.telethon.dev/?from=199&to=201>`__.
+
+Additions
+~~~~~~~~~
+
+* ``send_as`` and ``effect`` added to ``send_message`` and related methods.
+* :tl:`MessageMediaGeoLive` is now recognized for auto-input conversion.
+
+Enhancements
+~~~~~~~~~~~~
+
+* Improved wording when using a likely unintended session file.
+* Improved behaviour for matching Markdown links.
+* A truly clean update-state is now fetched upon login. This was most notably important for bots.
+* Time offset is now updated more reliably after connecting. This should fix legitimate "message too old/new" issues.
+
+Bug fixes
+~~~~~~~~~
+
+* :tl:`ChannelParticipantLeft` is now skipped in ``iter_participants``.
+* ``spoiler`` flag was lost on :tl:`MessageMediaPhoto` auto-input conversion.
+* :tl:`KeyboardButtonCopy` is now recognized as an inline button.
+* Downloading web-documents should now work again. Note that this still fetches the file from the original server.
+
+
+New layer (v1.39)
+=================
+
++------------------------+
+| Scheme layer used: 199 |
++------------------------+
+
+`View new and changed raw API methods <https://diff.telethon.dev/?from=193&to=199>`__.
+
+Additions
+~~~~~~~~~
+
+* ``drop_media_captions`` added to ``forward_messages``, and documented together with ``drop_author``.
+* :tl:`InputMediaDocumentExternal` is now recognized when sending albums.
+
+Enhancements
+~~~~~~~~~~~~
+
+* ``receive_updates=False`` now covers more cases, however, Telegram is still free to ignore it.
+* Better type-hints in several methods.
+* Markdown parsing of inline links should cover more cases.
+* ``range`` is now considered "list-like" and can be used on e.g. ``ids`` parameters.
+
+Bug fixes
+~~~~~~~~~
+
+* Session is now saved after setting the DC.
+* Fixed rare crash in entity cache handling when iterating through dialogs.
+* Fixed IOError that could occur during automatic resizing of some photos.
+
+
+New layer (v1.38)
+=================
+
++------------------------+
+| Scheme layer used: 193 |
++------------------------+
+
+`View new and changed raw API methods <https://diff.telethon.dev/?from=188&to=193>`__.
+
+Bug fixes
+~~~~~~~~~
+
+* Formatting entities misbehaved with albums.
+* Sending a Message object with a file did not use the new file.
+
+
+New layer (v1.37)
+=================
+
++------------------------+
+| Scheme layer used: 188 |
++------------------------+
+
+`View new and changed raw API methods <https://diff.telethon.dev/?from=181&to=188>`__.
+
+Additions
+~~~~~~~~~
+
+* Support for CDN downloads should be back. Telethon still prefers no CDN by default.
+
+Enhancements
+~~~~~~~~~~~~
+
+* ``FloodWaitPremium`` should now be handled like any other floodwaits.
+
+Bug fixes
+~~~~~~~~~
+
+* Fixed edge-case when using ``get_messages(..., reverse=True)``.
+* ``ConnectionError`` when using proxies should be raised properly.
+
+
+New layer (v1.36)
+=================
+
++------------------------+
+| Scheme layer used: 181 |
++------------------------+
+
+`View new and changed raw API methods <https://diff.telethon.dev/?from=178&to=181>`__.
+
+Bug fixes
+~~~~~~~~~
+
+* Certain updates, such as :tl:`UpdateBotStopped`, should now be processed reliably.
+
+
+New layer (v1.35)
+=================
+
++------------------------+
+| Scheme layer used: 178 |
++------------------------+
+
+`View new and changed raw API methods <https://diff.telethon.dev/?from=173&to=178>`__.
+
+Additions
+~~~~~~~~~
+
+* ``drop_author`` parameter now exposed in ``forward_messages``.
+
+Enhancements
+~~~~~~~~~~~~
+
+* "Custom secret support" should work with ``TcpMTProxy``.
+* Some type hints should now be more accurate.
+
+Bug fixes
+~~~~~~~~~
+
+* Session path couldn't be a ``pathlib.Path`` or ``None``.
+* Python versions older than 3.9 should now be supported again.
+* Readthedocs should hopefully build the v1 documentation again.
+
+
+New layer (v1.34)
+=================
+
++------------------------+
+| Scheme layer used: 173 |
++------------------------+
+
+`View new and changed raw API methods <https://diff.telethon.dev/?from=167&to=173>`__.
+
+Additions
+~~~~~~~~~
+
+* ``reply_to_chat`` and ``reply_to_sender`` are now in ``Message``.
+  This is useful when you lack access to the chat, but Telegram still included some basic information.
+
+Bug fixes
+~~~~~~~~~
+
+* ``parse_mode`` with a custom instance containing both ``parse`` and ``unparse`` should now work.
+* Parsing and unparsing message entities should now behave better in certain corner-cases.
+
+
+New layer (v1.33)
+=================
+
++------------------------+
+| Scheme layer used: 167 |
++------------------------+
+
+`View new and changed raw API methods <https://diff.telethon.dev/?from=166&to=167>`__.
+
+Enhancements
+~~~~~~~~~~~~
+
+* ``webbrowser`` is now imported conditionally, to support niche environments.
+* Library should now retry on the suddenly-common ``TimedOutError``.
+
+Bug fixes
+~~~~~~~~~
+
+* Sending photos which were automatically resized should work again (included in the v1.32 series).
+
+
+New layer (v1.32)
+=================
+
++------------------------+
+| Scheme layer used: 166 |
++------------------------+
+
+`View new and changed raw API methods <https://diff.telethon.dev/?from=165&to=166>`__.
+
+This enables you to use custom languages in preformatted blocks using HTML:
+
+.. code-block:: html
+
+  <pre>
+    <code class='language-python'>from telethon import TelegramClient</code>
+  </pre>
+
+Note that Telethon v1's markdown is a custom format and won't support language tags.
+If you want to set a custom language, you have to use HTML or a custom formatter.
+
+
+Dropped imghdr support (v1.31)
+==============================
+
++------------------------+
+| Scheme layer used: 165 |
++------------------------+
+
+This release contains a breaking change in preparation for Python 3.12.
+If you were sending photos from in-memory ``bytes`` or ``BytesIO`` containing images,
+you should now use ``BytesIO`` and set the ``.name`` property to a dummy name.
+This will allow Telethon to detect the correct extension (and file type).
+
+.. code-block:: python
+
+    # before
+    image_data = b'...'
+    client.send_file(chat, image_data)
+
+    # after
+    from io import BytesIO
+    image_data = BytesIO(b'...')
+    image_data.name = 'a.jpg'  # any name, only the extension matters
+    client.send_file(chat, image_data)
+
+
+Bug fixes
+~~~~~~~~~
+
+* Code generation wasn't working under PyPy.
+* Obtaining markdown or HTML from message text could produce unexpected results sometimes.
+* Other fixes for bugs from the previous version, which were already fixed in patch versions.
+
+Breaking Changes
+~~~~~~~~~~~~~~~~
+
+* ``imghdr`` is deprecated in newer Python versions, so Telethon no longer uses it.
+  This means there might be some cases where Telethon fails to infer the file extension for buffers containing images.
+  If you were relying on this, add ``.name = 'a.jpg'`` (or other extension) to the ``BytesIO`` buffers you upload.
+
+Layer bump and small changes (v1.30)
+====================================
+
++------------------------+
+| Scheme layer used: 162 |
++------------------------+
+
+Some of the bug fixes were already present in patch versions of ``v1.29``, but
+the new layer necessitated a minor bump.
+
+Enhancements
+~~~~~~~~~~~~
+
+* Removed client-side checks for editing messages.
+  This only affects ``Message.edit``, as ``client.edit_message`` already had
+  no checks.
+* Library should not understand more server-side errors during update handling
+  which should reduce crashes.
+* Client-side image compression should behave better now.
+
+Bug fixes
+~~~~~~~~~
+
+* Some updates such as ``UpdateChatParticipant`` were being missed due to the
+  order in which Telegram sent them. The library now more carefully checks for
+  the sequence and pts contained in them to avoid dropping them.
+* Fixed ``is_inline`` check for :tl:`KeyboardButtonWebView`.
+* Fixed some issues getting entity from cache by ID.
+* ``reply_to`` should now work when sending albums.
+
+
+More bug fixing (v1.29)
+=======================
+
++------------------------+
+| Scheme layer used: 160 |
++------------------------+
+
+This layer introduces the necessary raw API methods to work with stories.
+
+The library is aiming to be "feature-frozen" for as long as v1 is active,
+so friendly client methods are not implemented, but example code to use
+stories can be found in the GitHub wiki of the project.
+
+Enhancements
+~~~~~~~~~~~~
+
+* Removed client-side checks for methods dealing with chat permissions.
+  In particular, this means you can now ban channels.
+* Improved some error messages and added new classes for more RPC errors.
+* The client-side check for valid usernames has been loosened, so that
+  very short premium usernames are no longer considered invalid.
+
+Bug fixes
+~~~~~~~~~
+
+* Attempting to download a thumbnail from documnets without one would fail,
+  rather than do nothing (since nothing can be downloaded if there is no thumb).
+* More errors are caught in the update handling loop.
+* HTML ``.text`` should now "unparse" any message contents correctly.
+* Fixed some problems related to logging.
+* ``comment_to`` should now work as expected with albums.
+* ``asyncio.CancelledError`` should now correctly propagate from the update loop.
+* Removed some absolute imports in favour of relative imports.
+* ``UserUpdate.last_seen`` should now behave correctly.
+* Fixed a rare ``ValueError`` during ``connect`` if the session cache was bad.
+
+
+New Layer and housekeeping (v1.28)
+==================================
+
++------------------------+
+| Scheme layer used: 155 |
++------------------------+
+
+Plenty of stale issues closed, as well as improvements for some others.
+
+Additions
+~~~~~~~~~
+
+* New ``entity_cache_limit`` parameter in the ``TelegramClient`` constructor.
+  This should help a bit in keeping memory usage in check.
+
+Enhancements
+~~~~~~~~~~~~
+
+* ``progress_callback`` is now called when dealing with albums. See the
+  documentation on `client.send_file() <telethon.client.uploads.UploadMethods.send_file>`
+  for details.
+* Update state and entities are now periodically saved, so that the information
+  isn't lost in the case of crash or unexpected script terminations. You should
+  still be calling ``disconnect`` or using the context-manager, though.
+* The client should no longer unnecessarily call ``get_me`` every time it's started.
+
+Bug fixes
+~~~~~~~~~
+
+* Messages obtained via raw API could not be used in ``forward_messages``.
+* ``force_sms`` and ``sign_up`` have been deprecated. See `issue 4050`_ for details.
+  It is no longer possible for third-party applications, such as those made with
+  Telethon, to use those features.
+* ``events.ChatAction`` should now work in more cases in groups with hidden members.
+* Errors that occur at the connection level should now be properly propagated, so that
+  you can actually have a chance to handle them.
+* Update handling should be more resilient.
+* ``PhoneCodeExpiredError`` will correctly clear the stored hash if it occurs in ``sign_in``.
+* In patch ``v1.28.2``, :tl:`InputBotInlineMessageID64` can now be used
+  to edit inline messages.
+
+
+.. _issue 4050: https://github.com/LonamiWebs/Telethon/issues/4050
+
+
+New Layer and some Bug fixes (v1.27)
+====================================
+
++------------------------+
+| Scheme layer used: 152 |
++------------------------+
+
+Bug fixes
+~~~~~~~~~
+
+* When the account is logged-out, the library should now correctly propagate
+  an error through ``run_until_disconnected`` to let you handle it.
+* The library no longer uses ``asyncio.get_event_loop()`` in newer Python
+  versions, which should get rid of some deprecation warnings.
+* It could happen that bots would receive messages sent by themselves,
+  very often right after they deleted a message. This should happen far
+  less often now (but might still happen with unlucky timings).
+* Maximum photo size for automatic image resizing is now larger.
+* The initial request is now correctly wrapped in ``invokeWithoutUpdates``
+  when updates are disabled after constructing the client instance.
+* Using a ``pathlib.Path`` to download contacts and web documents should
+  now work correctly.
+
+New Layer and some Bug fixes (v1.26)
+====================================
+
++------------------------+
+| Scheme layer used: 149 |
++------------------------+
+
+This new layer includes things such as emoji status, more admin log events,
+forum topics and message reactions, among other things. You can access these
+using raw API. It also contains a few bug fixes.
+
+These were fixed in the v1.25 series:
+
+* ``client.edit_admin`` did not work on small group chats.
+* ``client.get_messages`` could stop early in some channels.
+* ``client.download_profile_photo`` now should work even if ``User.min``.
+* ``client.disconnect`` should no longer hang when being called from within
+  an event handlers.
+* ``client.get_dialogs`` now initializes the update state for channels.
+* The message sender should not need to be fetched in more cases.
+* Lowered the severity of some log messages to be less spammy.
+
+These are new to v1.26.0:
+
+* Layer update.
+* New documented RPC errors.
+* Sometimes the first message update to a channel could be missed if said
+  message was read immediately.
+* ``client.get_dialogs`` would fail when the total count evenly divided
+  the chunk size of 100.
+* ``client.get_messages`` could get stuck during a global search.
+* Potentially fixed some issues when sending certain videos.
+* Update handling should be more resilient.
+* The client should handle having its auth key destroyed more gracefully.
+* Fixed some issues when logging certain messages.
+
+
 Bug fixes (v1.25.1)
 ===================
 
@@ -2055,7 +2479,7 @@ the scenes! This means you're now able to do both of the following:
     async def main():
       await client.send_message('me', 'Hello!')
 
-    asyncio.get_event_loop().run_until_complete(main())
+    asyncio.run(main())
 
     # ...can be rewritten as:
 

readthedocs/misc/compatibility-and-convenience.rst

@@ -161,19 +161,17 @@ just get rid of ``telethon.sync`` and work inside an ``async def``:
 
             await client.run_until_disconnected()
 
-    loop = asyncio.get_event_loop()
+    asyncio.run(main())
-    loop.run_until_complete(main())
 
 
-The ``telethon.sync`` magic module simply wraps every method behind:
+The ``telethon.sync`` magic module essentially wraps every method behind:
 
 .. code-block:: python
 
-    loop = asyncio.get_event_loop()
+    asyncio.run(main())
-    loop.run_until_complete(main())
 
-So that you don't have to write it yourself every time. That's the
+With some other tricks, so that you don't have to write it yourself every time.
-overhead you pay if you import it, and what you save if you don't.
+That's the overhead you pay if you import it, and what you save if you don't.
 
 Learning
 ========

readthedocs/misc/wall-of-shame.rst

@@ -1,65 +0,0 @@
-=============
-Wall of Shame
-=============
-
-
-This project has an
-`issues <https://github.com/LonamiWebs/Telethon/issues>`__ section for
-you to file **issues** whenever you encounter any when working with the
-library. Said section is **not** for issues on *your* program but rather
-issues with Telethon itself.
-
-If you have not made the effort to 1. read through the docs and 2.
-`look for the method you need <https://tl.telethon.dev/>`__,
-you will end up on the `Wall of
-Shame <https://github.com/LonamiWebs/Telethon/issues?q=is%3Aissue+label%3ARTFM+is%3Aclosed>`__,
-i.e. all issues labeled
-`"RTFM" <http://www.urbandictionary.com/define.php?term=RTFM>`__:
-
-       **rtfm**
-       Literally "Read The F--king Manual"; a term showing the
-       frustration of being bothered with questions so trivial that the asker
-       could have quickly figured out the answer on their own with minimal
-       effort, usually by reading readily-available documents. People who
-       say"RTFM!" might be considered rude, but the true rude ones are the
-       annoying people who take absolutely no self-responibility and expect to
-       have all the answers handed to them personally.
-
-       *"Damn, that's the twelveth time that somebody posted this question
-       to the messageboard today! RTFM, already!"*
-
-       *by Bill M. July 27, 2004*
-
-If you have indeed read the docs, and have tried looking for the method,
-and yet you didn't find what you need, **that's fine**. Telegram's API
-can have some obscure names at times, and for this reason, there is a
-`"question"
-label <https://github.com/LonamiWebs/Telethon/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20label%3Aquestion%20>`__
-with questions that are okay to ask. Just state what you've tried so
-that we know you've made an effort, or you'll go to the Wall of Shame.
-
-Of course, if the issue you're going to open is not even a question but
-a real issue with the library (thankfully, most of the issues have been
-that!), you won't end up here. Don't worry.
-
-Current winner
---------------
-
-The current winner is `issue
-213 <https://github.com/LonamiWebs/Telethon/issues/213>`__:
-
-**Issue:**
-
-.. figure:: https://user-images.githubusercontent.com/6297805/29822978-9a9a6ef0-8ccd-11e7-9ec5-934ea0f57681.jpg
-
-:alt: Winner issue
-
-Winner issue
-
-**Answer:**
-
-.. figure:: https://user-images.githubusercontent.com/6297805/29822983-9d523402-8ccd-11e7-9fb1-5783740ee366.jpg
-
-:alt: Winner issue answer
-
-Winner issue answer

readthedocs/quick-references/client-reference.rst

@@ -32,7 +32,6 @@ Auth
     send_code_request
     sign_in
     qr_login
-    sign_up
     log_out
     edit_2fa
 

readthedocs/quick-references/faq.rst

@@ -20,7 +20,7 @@ To enable logging, at the following code to the top of your main file:
 .. code-block:: python
 
     import logging
-    logging.basicConfig(format='[%(levelname) 5s/%(asctime)s] %(name)s: %(message)s',
+    logging.basicConfig(format='[%(levelname) %(asctime)s] %(name)s: %(message)s',
                         level=logging.WARNING)
 
 You can change the logging level to be something different, from less to more information:
@@ -60,6 +60,16 @@ And except them as such:
 My account was deleted/limited when using the library
 =====================================================
 
+First and foremost, **this is not a problem exclusive to Telethon.
+Any third-party library is prone to cause the accounts to appear banned.**
+Even official applications can make Telegram ban an account under certain
+circumstances. Third-party libraries such as Telethon are a lot easier to
+use, and as such, they are misused to spam, which causes Telegram to learn
+certain patterns and ban suspicious activity.
+
+There is no point in Telethon trying to circumvent this. Even if it succeeded,
+spammers would then abuse the library again, and the cycle would repeat.
+
 The library will only do things that you tell it to do. If you use
 the library with bad intentions, Telegram will hopefully ban you.
 
@@ -67,8 +77,7 @@ However, you may also be part of a limited country, such as Iran or Russia.
 In that case, we have bad news for you. Telegram is much more likely to ban
 these numbers, as they are often used to spam other accounts, likely through
 the use of libraries like this one. The best advice we can give you is to not
-abuse the API, like calling many requests really quickly, and to sign up with
+abuse the API, like calling many requests really quickly.
-these phones through an official application.
 
 We have also had reports from Kazakhstan and China, where connecting
 would fail. To solve these connection problems, you should use a proxy.
@@ -76,6 +85,16 @@ would fail. To solve these connection problems, you should use a proxy.
 Telegram may also ban virtual (VoIP) phone numbers,
 as again, they're likely to be used for spam.
 
+More recently (year 2023 onwards), Telegram has started putting a lot more
+measures to prevent spam (with even additions such as anonymous participants
+in groups or the inability to fetch group members at all). This means some
+of the anti-spam measures have gotten more aggressive.
+
+The recommendation has usually been to use the library only on well-established
+accounts (and not an account you just created), and to not perform actions that
+could be seen as abuse. Telegram decides what those actions are, and they're
+free to change how they operate at any time.
+
 If you want to check if your account has been limited,
 simply send a private message to `@SpamBot`_ through Telegram itself.
 You should notice this by getting errors like ``PeerFloodError``,
@@ -179,6 +198,137 @@ won't do unnecessary work unless you need to:
         sender = await event.get_sender()
 
 
+File download is slow or sending files takes too long
+=====================================================
+
+The communication with Telegram is encrypted. Encryption requires a lot of
+math, and doing it in pure Python is very slow. ``cryptg`` is a library which
+containns the encryption functions used by Telethon. If it is installed (via
+``pip install cryptg``), it will automatically be used and should provide
+a considerable speed boost. You can know whether it's used by configuring
+``logging`` (at ``INFO`` level or lower) *before* importing ``telethon``.
+
+Note that the library does *not* download or upload files in parallel, which
+can also help with the speed of downloading or uploading a single file. There
+are snippets online implementing that. The reason why this is not built-in
+is because the limiting factor in the long run are ``FloodWaitError``, and
+using parallel download or uploads only makes them occur sooner.
+
+
+What does "Server sent a very new message with ID" mean?
+========================================================
+
+You may also see this error as "Server sent a very old message with ID".
+
+This is a security feature from Telethon that cannot be disabled and is
+meant to protect you against replay attacks.
+
+When this message is incorrectly reported as a "bug",
+the most common patterns seem to be:
+
+* Your system time is incorrect.
+* The proxy you're using may be interfering somehow.
+* The Telethon session is being used or has been used from somewhere else.
+  Make sure that you created the session from Telethon, and are not using the
+  same session anywhere else. If you need to use the same account from
+  multiple places, login and use a different session for each place you need.
+
+
+What does "Server replied with a wrong session ID" mean?
+========================================================
+
+This is a security feature from Telethon that cannot be disabled and is
+meant to protect you against unwanted session reuse.
+
+When this message is reported as a "bug", the most common patterns seem to be:
+
+* The proxy you're using may be interfering somehow.
+* The Telethon session is being used or has been used from somewhere else.
+  Make sure that you created the session from Telethon, and are not using the
+  same session anywhere else. If you need to use the same account from
+  multiple places, login and use a different session for each place you need.
+* You may be using multiple connections to the Telegram server, which seems
+  to confuse Telegram.
+
+Most of the time it should be safe to ignore this warning. If the library
+still doesn't behave correctly, make sure to check if any of the above bullet
+points applies in your case and try to work around it.
+
+If the issue persists and there is a way to reliably reproduce this error,
+please add a comment with any additional details you can provide to
+`issue 3759`_, and perhaps some additional investigation can be done
+(but it's unlikely, as Telegram *is* sending unexpected data).
+
+
+What does "Could not find a matching Constructor ID for the TLObject" mean?
+===========================================================================
+
+Telegram uses "layers", which you can think of as "versions" of the API they
+offer. When Telethon reads responses that the Telegram servers send, these
+need to be deserialized (into what Telethon calls "TLObjects").
+
+Every Telethon version understands a single Telegram layer. When Telethon
+connects to Telegram, both agree on the layer to use. If the layers don't
+match, Telegram may send certain objects which Telethon no longer understands.
+
+When this message is reported as a "bug", the most common patterns seem to be
+that the Telethon session is being used or has been used from somewhere else.
+Make sure that you created the session from Telethon, and are not using the
+same session anywhere else. If you need to use the same account from
+multiple places, login and use a different session for each place you need.
+
+
+What does "Task was destroyed but it is pending" mean?
+======================================================
+
+Your script likely finished abruptly, the ``asyncio`` event loop got
+destroyed, and the library did not get a chance to properly close the
+connection and close the session.
+
+Make sure you're either using the context manager for the client or always
+call ``await client.disconnect()`` (by e.g. using a ``try/finally``).
+
+
+What does "The asyncio event loop must not change after connection" mean?
+=========================================================================
+
+Telethon uses ``asyncio``, and makes use of things like tasks and queues
+internally to manage the connection to the server and match responses to the
+requests you make. Most of them are initialized after the client is connected.
+
+For example, if the library expects a result to a request made in loop A, but
+you attempt to get that result in loop B, you will very likely find a deadlock.
+To avoid a deadlock, the library checks to make sure the loop in use is the
+same as the one used to initialize everything, and if not, it throws an error.
+
+The most common cause is ``asyncio.run``, since it creates a new event loop.
+If you ``asyncio.run`` a function to create the client and set it up, and then
+you ``asyncio.run`` another function to do work, things won't work, so the
+library throws an error early to let you know something is wrong.
+
+Instead, it's often a good idea to have a single ``async def main`` and simply
+``asyncio.run()`` it and do all the work there. From it, you're also able to
+call other ``async def`` without having to touch ``asyncio.run`` again:
+
+.. code-block:: python
+
+    # It's fine to create the client outside as long as you don't connect
+    client = TelegramClient(...)
+
+    async def main():
+        # Now the client will connect, so the loop must not change from now on.
+        # But as long as you do all the work inside main, including calling
+        # other async functions, things will work.
+        async with client:
+            ....
+
+    if __name__ == '__main__':
+        asyncio.run(main())
+
+Be sure to read the ``asyncio`` documentation if you want a better
+understanding of event loop, tasks, and what functions you can use.
+
+
 What does "bases ChatGetter" mean?
 ==================================
 
@@ -204,6 +354,36 @@ Telegram has a lot to offer, and inheritance helps the library reduce
 boilerplate, so it's important to know this concept. For newcomers,
 this may be a problem, so we explain what it means here in the FAQ.
 
+Can I send files by ID?
+=======================
+
+When people talk about IDs, they often refer to one of two things:
+the integer ID inside media, and a random-looking long string.
+
+You cannot use the integer ID to send media. Generally speaking, sending media
+requires a combination of ID, ``access_hash`` and ``file_reference``.
+The first two are integers, while the last one is a random ``bytes`` sequence.
+
+* The integer ``id`` will always be the same for every account, so every user
+  or bot looking at a particular media file, will see a consistent ID.
+* The ``access_hash`` will always be the same for a given account, but
+  different accounts will each see their own, different ``access_hash``.
+  This makes it impossible to get media object from one account and use it in
+  another. The other account must fetch the media object itself.
+* The ``file_reference`` is random for everyone and will only work for a few
+  hours before it expires. It must be refetched before the media can be used
+  (to either resend the media or download it).
+
+The second type of "`file ID <https://core.telegram.org/bots/api#inputfile>`_"
+people refer to is a concept from the HTTP Bot API. It's a custom format which
+encodes enough information to use the media.
+
+Telethon provides an old version of these HTTP Bot API-style file IDs via
+``message.file.id``, however, this feature is no longer maintained, so it may
+not work. It will be removed in future versions. Nonetheless, it is possible
+to find a different Python package (or write your own) to parse these file IDs
+and construct the necessary input file objects to send or download the media.
+
 
 Can I use Flask with the library?
 =================================
@@ -239,4 +419,5 @@ file and run that, or use the normal ``python`` interpreter.
 .. _logging: https://docs.python.org/3/library/logging.html
 .. _@SpamBot: https://t.me/SpamBot
 .. _issue 297: https://github.com/LonamiWebs/Telethon/issues/297
+.. _issue 3759: https://github.com/LonamiWebs/Telethon/issues/3759
 .. _quart_login.py: https://github.com/LonamiWebs/Telethon/tree/v1/telethon_examples#quart_loginpy

readthedocs/requirements.txt

@@ -1 +1,2 @@
-telethon
+./
+sphinx-rtd-theme~=1.3.0

setup.py

@@ -16,6 +16,7 @@ import os
 import re
 import shutil
 import sys
+import urllib.request
 from pathlib import Path
 from subprocess import run
 
@@ -43,6 +44,8 @@ class TempWorkDir:
         os.chdir(self.original)
 
 
+API_REF_URL = 'https://tl.telethon.dev/'
+
 GENERATOR_DIR = Path('telethon_generator')
 LIBRARY_DIR = Path('telethon')
 
@@ -155,14 +158,31 @@ def main(argv):
         generate(argv[2:], argv[1])
 
     elif len(argv) >= 2 and argv[1] == 'pypi':
+        # Make sure tl.telethon.dev is up-to-date first
+        with urllib.request.urlopen(API_REF_URL) as resp:
+            html = resp.read()
+            m = re.search(br'layer\s+(\d+)', html)
+            if not m:
+                print('Failed to check that the API reference is up to date:', API_REF_URL)
+                return
+
+            from telethon_generator.parsers import find_layer
+            layer = next(filter(None, map(find_layer, TLOBJECT_IN_TLS)))
+            published_layer = int(m[1])
+            if published_layer != layer:
+                print('Published layer', published_layer, 'does not match current layer', layer, '.')
+                print('Make sure to update the API reference site first:', API_REF_URL)
+                return
+
         # (Re)generate the code to make sure we don't push without it
         generate(['tl', 'errors'])
 
         # Try importing the telethon module to assert it has no errors
         try:
             import telethon
-        except:
+        except Exception as e:
             print('Packaging for PyPi aborted, importing the module failed.')
+            print(e)
             return
 
         remove_dirs = ['__pycache__', 'build', 'dist', 'Telethon.egg-info']

telethon/_updates/entitycache.py

@@ -9,15 +9,17 @@ class EntityCache:
         self,
         hash_map: dict = _sentinel,
         self_id: int = None,
-        self_bot: bool = False
+        self_bot: bool = None
     ):
         self.hash_map = {} if hash_map is _sentinel else hash_map
         self.self_id = self_id
         self.self_bot = self_bot
 
-    def set_self_user(self, id, bot):
+    def set_self_user(self, id, bot, hash):
         self.self_id = id
         self.self_bot = bot
+        if hash:
+            self.hash_map[id] = (hash, EntityType.BOT if bot else EntityType.USER)
 
     def get(self, id):
         try:
@@ -47,8 +49,11 @@ class EntityCache:
             if getattr(c, 'access_hash', None) and not getattr(c, 'min', None)
         )
 
-    def get_all_entities(self):
-        return [Entity(ty, id, hash) for id, (hash, ty) in self.hash_map.items()]
-
     def put(self, entity):
         self.hash_map[entity.id] = (entity.hash, entity.ty)
+
+    def retain(self, filter):
+        self.hash_map = {k: v for k, v in self.hash_map.items() if filter(k)}
+
+    def __len__(self):
+        return len(self.hash_map)

telethon/_updates/messagebox.py

@@ -19,9 +19,11 @@ to get the difference.
 import asyncio
 import datetime
 import time
+import logging
 from enum import Enum
 from .session import SessionState, ChannelState
 from ..tl import types as tl, functions as fn
+from ..helpers import get_running_loop
 
 
 # Telegram sends `seq` equal to `0` when "it doesn't matter", so we use that value too.
@@ -43,18 +45,34 @@ POSSIBLE_GAP_TIMEOUT = 0.5
 # Documentation recommends 15 minutes without updates (https://core.telegram.org/api/updates).
 NO_UPDATES_TIMEOUT = 15 * 60
 
+# object() but with a tag to make it easier to debug
+class Sentinel:
+    __slots__ = ('tag',)
+
+    def __init__(self, tag=None):
+        self.tag = tag or '_'
+
+    def __repr__(self):
+        return self.tag
+
 # Entry "enum".
 # Account-wide `pts` includes private conversations (one-to-one) and small group chats.
-ENTRY_ACCOUNT = object()
+ENTRY_ACCOUNT = Sentinel('ACCOUNT')
 # Account-wide `qts` includes only "secret" one-to-one chats.
-ENTRY_SECRET = object()
+ENTRY_SECRET = Sentinel('SECRET')
 # Integers will be Channel-specific `pts`, and includes "megagroup", "broadcast" and "supergroup" channels.
 
-_sentinel = object()
+# Python's logging doesn't define a TRACE level. Pick halfway between DEBUG and NOTSET.
+# We don't define a name for this as libraries shouldn't do that though.
+LOG_LEVEL_TRACE = (logging.DEBUG - logging.NOTSET) // 2
+
+_sentinel = Sentinel()
 
 def next_updates_deadline():
-    return asyncio.get_running_loop().time() + NO_UPDATES_TIMEOUT
+    return get_running_loop().time() + NO_UPDATES_TIMEOUT
 
+def epoch():
+    return datetime.datetime(*time.gmtime(0)[:6]).replace(tzinfo=datetime.timezone.utc)
 
 class GapError(ValueError):
     def __repr__(self):
@@ -93,19 +111,12 @@ class PtsInfo:
 
         qts = getattr(update, 'qts', None)
         if qts:
-            pts_count = 1 if isinstance(update, tl.UpdateNewEncryptedMessage) else 0
+            return cls(pts=qts, pts_count=1, entry=ENTRY_SECRET)
-            return cls(pts=qts, pts_count=pts_count, entry=ENTRY_SECRET)
 
         return None
 
     def __repr__(self):
-        if self.entry is ENTRY_ACCOUNT:
+        return f'PtsInfo(pts={self.pts}, pts_count={self.pts_count}, entry={self.entry})'
-            entry = 'ENTRY_ACCOUNT'
-        elif self.entry is ENTRY_SECRET:
-            entry = 'ENTRY_SECRET'
-        else:
-            entry = self.entry
-        return f'PtsInfo(pts={self.pts}, pts_count={self.pts_count}, entry={entry})'
 
 
 # The state of a particular entry in the message box.
@@ -154,15 +165,16 @@ class PossibleGap:
 #
 # See https://core.telegram.org/api/updates#message-related-event-sequences.
 class MessageBox:
-    __slots__ = ('map', 'date', 'seq', 'next_deadline', 'possible_gaps', 'getting_diff_for', 'reset_deadlines_for')
+    __slots__ = ('_log', 'map', 'date', 'seq', 'next_deadline', 'possible_gaps', 'getting_diff_for')
 
     def __init__(
         self,
+        log,
         # Map each entry to their current state.
         map: dict = _sentinel,  # entry -> state
 
         # Additional fields beyond PTS needed by `ENTRY_ACCOUNT`.
-        date: datetime.datetime = datetime.datetime(*time.gmtime(0)[:6]).replace(tzinfo=datetime.timezone.utc),
+        date: datetime.datetime = epoch() + datetime.timedelta(seconds=1),
         seq: int = NO_SEQ,
 
         # Holds the entry with the closest deadline (optimization to avoid recalculating the minimum deadline).
@@ -179,18 +191,26 @@ class MessageBox:
 
         # For which entries are we currently getting difference.
         getting_diff_for: set = _sentinel,  # entry
-
-        # Temporarily stores which entries should have their update deadline reset.
-        # Stored in the message box in order to reuse the allocation.
-        reset_deadlines_for: set = _sentinel  # entry
     ):
+        self._log = log
         self.map = {} if map is _sentinel else map
         self.date = date
         self.seq = seq
         self.next_deadline = next_deadline
         self.possible_gaps = {} if possible_gaps is _sentinel else possible_gaps
         self.getting_diff_for = set() if getting_diff_for is _sentinel else getting_diff_for
-        self.reset_deadlines_for = set() if reset_deadlines_for is _sentinel else reset_deadlines_for
+
+        if __debug__:
+            self._trace('MessageBox initialized')
+
+    def _trace(self, msg, *args, **kwargs):
+        # Calls to trace can't really be removed beforehand without some dark magic.
+        # So every call to trace is prefixed with `if __debug__`` instead, to remove
+        # it when using `python -O`. Probably unnecessary, but it's nice to avoid
+        # paying the cost for something that is not used.
+        self._log.log(LOG_LEVEL_TRACE, 'Current MessageBox state: seq = %r, date = %s, map = %r',
+                      self.seq, self.date.isoformat(), self.map)
+        self._log.log(LOG_LEVEL_TRACE, msg, *args, **kwargs)
 
     # region Creation, querying, and setting base state.
 
@@ -198,6 +218,9 @@ class MessageBox:
         """
         Create a [`MessageBox`] from a previously known update state.
         """
+        if __debug__:
+            self._trace('Loading MessageBox with session_state = %r, channel_states = %r', session_state, channel_states)
+
         deadline = next_updates_deadline()
 
         self.map.clear()
@@ -207,7 +230,7 @@ class MessageBox:
             self.map[ENTRY_SECRET] = State(pts=session_state.qts, deadline=deadline)
         self.map.update((s.channel_id, State(pts=s.pts, deadline=deadline)) for s in channel_states)
 
-        self.date = datetime.datetime.fromtimestamp(session_state.date).replace(tzinfo=datetime.timezone.utc)
+        self.date = datetime.datetime.fromtimestamp(session_state.date, tz=datetime.timezone.utc)
         self.seq = session_state.seq
         self.next_deadline = ENTRY_ACCOUNT
 
@@ -237,7 +260,7 @@ class MessageBox:
         If a deadline expired, the corresponding entries will be marked as needing to get its difference.
         While there are entries pending of getting their difference, this method returns the current instant.
         """
-        now = asyncio.get_running_loop().time()
+        now = get_running_loop().time()
 
         if self.getting_diff_for:
             return now
@@ -255,8 +278,11 @@ class MessageBox:
         # timeout for updates several times (it also makes sense to get difference if now is the deadline).
         if now >= deadline:
             # Check all expired entries and add them to the list that needs getting difference.
-            self.getting_diff_for.update(entry for entry, gap in self.possible_gaps.items() if now > gap.deadline)
+            self.getting_diff_for.update(entry for entry, gap in self.possible_gaps.items() if now >= gap.deadline)
-            self.getting_diff_for.update(entry for entry, state in self.map.items() if now > state.deadline)
+            self.getting_diff_for.update(entry for entry, state in self.map.items() if now >= state.deadline)
+
+            if __debug__:
+                self._trace('Deadlines met, now getting diff for %r', self.getting_diff_for)
 
             # When extending `getting_diff_for`, it's important to have the moral equivalent of
             # `begin_get_diff` (that is, clear possible gaps if we're now getting difference).
@@ -265,44 +291,38 @@ class MessageBox:
 
         return deadline
 
-    # Reset the deadline for the periods without updates for a given entry.
+    # Reset the deadline for the periods without updates for the given entries.
     #
     # It also updates the next deadline time to reflect the new closest deadline.
-    def reset_deadline(self, entry, deadline):
+    def reset_deadlines(self, entries, deadline):
+        if not entries:
+            return
+        for entry in entries:
             if entry not in self.map:
                 raise RuntimeError('Called reset_deadline on an entry for which we do not have state')
             self.map[entry].deadline = deadline
 
-        if self.next_deadline == entry:
+        if self.next_deadline in entries:
             # If the updated deadline was the closest one, recalculate the new minimum.
             self.next_deadline = min(self.map.items(), key=lambda entry_state: entry_state[1].deadline)[0]
         elif self.next_deadline in self.map and deadline < self.map[self.next_deadline].deadline:
             # If the updated deadline is smaller than the next deadline, change the next deadline to be the new one.
+            # Any entry will do, so the one from the last iteration is fine.
             self.next_deadline = entry
         # else an unrelated deadline was updated, so the closest one remains unchanged.
 
     # Convenience to reset a channel's deadline, with optional timeout.
     def reset_channel_deadline(self, channel_id, timeout):
-        self.reset_deadline(channel_id, asyncio.get_running_loop().time() + (timeout or NO_UPDATES_TIMEOUT))
+        self.reset_deadlines({channel_id}, get_running_loop().time() + (timeout or NO_UPDATES_TIMEOUT))
-
-    # Reset all the deadlines in `reset_deadlines_for` and then empty the set.
-    def apply_deadlines_reset(self):
-        next_deadline = next_updates_deadline()
-
-        reset_deadlines_for = self.reset_deadlines_for
-        self.reset_deadlines_for = set()  # "move" the set to avoid self.reset_deadline() from touching it during iter
-
-        for entry in reset_deadlines_for:
-            self.reset_deadline(entry, next_deadline)
-
-        reset_deadlines_for.clear()  # reuse allocation, the other empty set was a temporary dummy value
-        self.reset_deadlines_for = reset_deadlines_for
 
     # Sets the update state.
     #
     # Should be called right after login if [`MessageBox::new`] was used, otherwise undesirable
     # updates will be fetched.
     def set_state(self, state, reset=True):
+        if __debug__:
+            self._trace('Setting state %s', state)
+
         deadline = next_updates_deadline()
 
         if state.pts != NO_SEQ or not reset:
@@ -329,6 +349,9 @@ class MessageBox:
     #
     # The update state will only be updated if no entry was known previously.
     def try_set_channel_state(self, id, pts):
+        if __debug__:
+            self._trace('Trying to set channel state for %r: %r', id, pts)
+
         if id not in self.map:
             self.map[id] = State(pts=pts, deadline=next_updates_deadline())
 
@@ -336,15 +359,18 @@ class MessageBox:
     # Fails if the entry does not have a previously-known state that can be used to get its difference.
     #
     # Clears any previous gaps.
-    def try_begin_get_diff(self, entry):
+    def try_begin_get_diff(self, entry, reason):
         if entry not in self.map:
             # Won't actually be able to get difference for this entry if we don't have a pts to start off from.
             if entry in self.possible_gaps:
                 raise RuntimeError('Should not have a possible_gap for an entry not in the state map')
 
-            # TODO it would be useful to log when this happens
+            if __debug__:
+                self._trace('Should get difference for %r because %s but cannot due to missing hash', entry, reason)
             return
 
+        if __debug__:
+            self._trace('Marking %r as needing difference because %s', entry, reason)
         self.getting_diff_for.add(entry)
         self.possible_gaps.pop(entry, None)
 
@@ -357,7 +383,7 @@ class MessageBox:
         except KeyError:
             raise RuntimeError('Called end_get_diff on an entry which was not getting diff for')
 
-        self.reset_deadline(entry, next_updates_deadline())
+        self.reset_deadlines({entry}, next_updates_deadline())
         assert entry not in self.possible_gaps, "gaps shouldn't be created while getting difference"
 
     # endregion Creation, querying, and setting base state.
@@ -382,11 +408,6 @@ class MessageBox:
         chat_hashes,
         result,  # out list of updates; returns list of user, chat, or raise if gap
     ):
-        date = getattr(updates, 'date', None)
-        if date is None:
-            # updatesTooLong is the only one with no date (we treat it as a gap)
-            self.try_begin_get_diff(ENTRY_ACCOUNT)
-            raise GapError
 
         # v1 has never sent updates produced by the client itself to the handlers.
         # However proper update handling requires those to be processed.
@@ -395,11 +416,25 @@ class MessageBox:
         real_result = result
         result = []
 
-        seq = getattr(updates, 'seq', None) or NO_SEQ
+        date = getattr(updates, 'date', None)
-        seq_start = getattr(updates, 'seq_start', None) or seq
+        seq = getattr(updates, 'seq', None)
+        seq_start = getattr(updates, 'seq_start', None)
         users = getattr(updates, 'users', None) or []
         chats = getattr(updates, 'chats', None) or []
 
+        if __debug__:
+            self._trace('Processing updates with seq = %r, seq_start = %r, date = %s: %s',
+                        seq, seq_start, date.isoformat() if date else None, updates)
+
+        if date is None:
+            # updatesTooLong is the only one with no date (we treat it as a gap)
+            self.try_begin_get_diff(ENTRY_ACCOUNT, 'received updatesTooLong')
+            raise GapError
+        if seq is None:
+            seq = NO_SEQ
+        if seq_start is None:
+            seq_start = seq
+
         # updateShort is the only update which cannot be dispatched directly but doesn't have 'updates' field
         updates = getattr(updates, 'updates', None) or [updates.update if isinstance(updates, tl.UpdateShort) else updates]
 
@@ -411,26 +446,34 @@ class MessageBox:
         if seq_start != NO_SEQ:
             if self.seq + 1 > seq_start:
                 # Skipping updates that were already handled
+                if __debug__:
+                    self._trace('Skipping updates as they should have already been handled')
                 return (users, chats)
             elif self.seq + 1 < seq_start:
                 # Gap detected
-                self.try_begin_get_diff(ENTRY_ACCOUNT)
+                self.try_begin_get_diff(ENTRY_ACCOUNT, 'detected gap')
                 raise GapError
             # else apply
 
-            self.date = date
-            if seq != NO_SEQ:
-                self.seq = seq
-
-        result.extend(filter(None, (self.apply_pts_info(u, reset_deadline=True) for u in updates)))
-
-        self.apply_deadlines_reset()
-
         def _sort_gaps(update):
             pts = PtsInfo.from_update(update)
             return pts.pts - pts.pts_count if pts else 0
 
+        reset_deadlines = set()  # temporary buffer
+
+        result.extend(filter(None, (
+            self.apply_pts_info(u, reset_deadlines=reset_deadlines)
+            # Telegram can send updates out of order (e.g. ReadChannelInbox first
+            # and then NewChannelMessage, both with the same pts, but the count is
+            # 0 and 1 respectively), so we sort them first.
+            for u in sorted(updates, key=_sort_gaps))))
+
+        self.reset_deadlines(reset_deadlines, next_updates_deadline())
+
         if self.possible_gaps:
+            if __debug__:
+                self._trace('Trying to re-apply %r possible gaps', len(self.possible_gaps))
+
             # For each update in possible gaps, see if the gap has been resolved already.
             for key in list(self.possible_gaps.keys()):
                 self.possible_gaps[key].updates.sort(key=_sort_gaps)
@@ -440,15 +483,27 @@ class MessageBox:
 
                     # If this fails to apply, it will get re-inserted at the end.
                     # All should fail, so the order will be preserved (it would've cycled once).
-                    update = self.apply_pts_info(update, reset_deadline=False)
+                    update = self.apply_pts_info(update, reset_deadlines=None)
                     if update:
                         result.append(update)
+                        if __debug__:
+                            self._trace('Resolved gap with %r: %s', PtsInfo.from_update(update), update)
 
             # Clear now-empty gaps.
             self.possible_gaps = {entry: gap for entry, gap in self.possible_gaps.items() if gap.updates}
 
         real_result.extend(u for u in result if not u._self_outgoing)
 
+        if result and not self.possible_gaps:
+            # > If the updates were applied, local *Updates* state must be updated
+            # > with `seq` (unless it's 0) and `date` from the constructor.
+            if __debug__:
+                self._trace('Updating seq as all updates were applied')
+            if date != epoch():
+                self.date = date
+            if seq != NO_SEQ:
+                self.seq = seq
+
         return (users, chats)
 
     # Tries to apply the input update if its `PtsInfo` follows the correct order.
@@ -460,16 +515,18 @@ class MessageBox:
         self,
         update,
         *,
-        reset_deadline,
+        reset_deadlines,
     ):
         # This update means we need to call getChannelDifference to get the updates from the channel
         if isinstance(update, tl.UpdateChannelTooLong):
-            self.try_begin_get_diff(update.channel_id)
+            self.try_begin_get_diff(update.channel_id, 'received updateChannelTooLong')
             return None
 
         pts = PtsInfo.from_update(update)
         if not pts:
             # No pts means that the update can be applied in any order.
+            if __debug__:
+                self._trace('No pts in update, so it can be applied in any order: %s', update)
             return update
 
         # As soon as we receive an update of any form related to messages (has `PtsInfo`),
@@ -478,25 +535,31 @@ class MessageBox:
         # Build the `HashSet` to avoid calling `reset_deadline` more than once for the same entry.
         #
         # By the time this method returns, self.map will have an entry for which we can reset its deadline.
-        if reset_deadline:
+        if reset_deadlines:
-            self.reset_deadlines_for.add(pts.entry)
+            reset_deadlines.add(pts.entry)
 
         if pts.entry in self.getting_diff_for:
             # Note: early returning here also prevents gap from being inserted (which they should
             # not be while getting difference).
+            if __debug__:
+                self._trace('Skipping update with %r as its difference is being fetched', pts)
             return None
 
         if pts.entry in self.map:
             local_pts = self.map[pts.entry].pts
             if local_pts + pts.pts_count > pts.pts:
                 # Ignore
+                if __debug__:
+                    self._trace('Skipping update since local pts %r > %r: %s', local_pts, pts, update)
                 return None
             elif local_pts + pts.pts_count < pts.pts:
                 # Possible gap
                 # TODO store chats too?
+                if __debug__:
+                    self._trace('Possible gap since local pts %r < %r: %s', local_pts, pts, update)
                 if pts.entry not in self.possible_gaps:
                     self.possible_gaps[pts.entry] = PossibleGap(
-                        deadline=asyncio.get_running_loop().time() + POSSIBLE_GAP_TIMEOUT,
+                        deadline=get_running_loop().time() + POSSIBLE_GAP_TIMEOUT,
                         updates=[]
                     )
 
@@ -504,24 +567,36 @@ class MessageBox:
                 return None
             else:
                 # Apply
-                pass
+                if __debug__:
-        else:
+                    self._trace('Applying update pts since local pts %r = %r: %s', local_pts, pts, update)
-            # No previous `pts` known, and because this update has to be "right" (it's the first one) our
-            # `local_pts` must be the one before the server pts.
-            local_pts = pts.pts - pts.pts_count
 
         # In a channel, we may immediately receive:
-        # * ReadChannelInbox (pts = X)
+        # * ReadChannelInbox (pts = X, pts_count = 0)
         # * NewChannelMessage (pts = X, pts_count = 1)
         #
-        # Notice how both `pts` are the same. The first one however would've triggered a gap
+        # Notice how both `pts` are the same. If they were to be applied out of order, the first
-        # because `local_pts` + `pts_count` of 0 would be less than `remote_pts`. So there is
+        # one however would've triggered a gap because `local_pts` + `pts_count` of 0 would be
-        # no risk by setting the `local_pts` to match the `remote_pts` here of missing the new
+        # less than `remote_pts`. So there is no risk by setting the `local_pts` to match the
-        # message.
+        # `remote_pts` here of missing the new message.
+        #
+        # The message would however be lost if we initialized the pts with the first one, since
+        # the second one would appear "already handled". To prevent this we set the pts to be
+        # one less when the count is 0 (which might be wrong and trigger a gap later on, but is
+        # unlikely). This will prevent us from losing updates in the unlikely scenario where these
+        # two updates arrive in different packets (and therefore couldn't be sorted beforehand).
         if pts.entry in self.map:
             self.map[pts.entry].pts = pts.pts
         else:
-            self.map[pts.entry] = State(pts=pts.pts, deadline=next_updates_deadline())
+            # When a chat is migrated to a megagroup, the first update can be a `ReadChannelInbox`
+            # with `pts = 1, pts_count = 0` followed by a `NewChannelMessage` with `pts = 2, pts_count=1`.
+            # Note how the `pts` for the message is 2 and not 1 unlike the case described before!
+            # This is likely because the `pts` cannot be 0 (or it would fail with PERSISTENT_TIMESTAMP_EMPTY),
+            # which forces the first update to be 1. But if we got difference with 1 and the second update
+            # also used 1, we would miss it, so Telegram probably uses 2 to work around that.
+            self.map[pts.entry] = State(
+                pts=(pts.pts - (0 if pts.pts_count else 1)) or 1,
+                deadline=next_updates_deadline()
+            )
 
         return update
 
@@ -536,12 +611,15 @@ class MessageBox:
                 if entry not in self.map:
                     raise RuntimeError('Should not try to get difference for an entry without known state')
 
-                return fn.updates.GetDifferenceRequest(
+                gd = fn.updates.GetDifferenceRequest(
                     pts=self.map[ENTRY_ACCOUNT].pts,
                     pts_total_limit=None,
                     date=self.date,
                     qts=self.map[ENTRY_SECRET].pts if ENTRY_SECRET in self.map else NO_SEQ,
                 )
+                if __debug__:
+                    self._trace('Requesting account difference %s', gd)
+                return gd
 
         return None
 
@@ -551,6 +629,9 @@ class MessageBox:
         diff,
         chat_hashes,
     ):
+        if __debug__:
+            self._trace('Applying account difference %s', diff)
+
         finish = None
         result = None
 
@@ -577,7 +658,7 @@ class MessageBox:
             secret = ENTRY_SECRET in self.getting_diff_for
 
             if not account and not secret:
-                raise RuntimeWarning('Should not be applying the difference when neither account or secret was diff was active')
+                raise RuntimeError('Should not be applying the difference when neither account or secret was diff was active')
 
             # Both may be active if both expired at the same time.
             if account:
@@ -602,7 +683,7 @@ class MessageBox:
             updates=diff.other_updates,
             users=diff.users,
             chats=diff.chats,
-            date=1,  # anything not-None
+            date=epoch(),
             seq=NO_SEQ,  # this way date is not used
         ), chat_hashes, updates)
 
@@ -619,11 +700,14 @@ class MessageBox:
         return updates, diff.users, diff.chats
 
     def end_difference(self):
+        if __debug__:
+            self._trace('Ending account difference')
+
         account = ENTRY_ACCOUNT in self.getting_diff_for
         secret = ENTRY_SECRET in self.getting_diff_for
 
         if not account and not secret:
-            raise RuntimeWarning('Should not be ending get difference when neither account or secret was diff was active')
+            raise RuntimeError('Should not be ending get difference when neither account or secret was diff was active')
 
         # Both may be active if both expired at the same time.
         if account:
@@ -658,13 +742,16 @@ class MessageBox:
         if not state:
             raise RuntimeError('Should not try to get difference for an entry without known state')
 
-        return fn.updates.GetChannelDifferenceRequest(
+        gd = fn.updates.GetChannelDifferenceRequest(
             force=False,
             channel=tl.InputChannel(packed.id, packed.hash),
             filter=tl.ChannelMessagesFilterEmpty(),
             pts=state.pts,
             limit=BOT_CHANNEL_DIFF_LIMIT if chat_hashes.self_bot else USER_CHANNEL_DIFF_LIMIT
         )
+        if __debug__:
+            self._trace('Requesting channel difference %s', gd)
+        return gd
 
     # Similar to [`MessageBox::process_updates`], but using the result from getting difference.
     def apply_channel_difference(
@@ -674,6 +761,9 @@ class MessageBox:
         chat_hashes,
     ):
         entry = request.channel.channel_id
+        if __debug__:
+            self._trace('Applying channel difference for %r: %s', entry, diff)
+
         self.possible_gaps.pop(entry, None)
 
         if isinstance(diff, tl.updates.ChannelDifferenceEmpty):
@@ -702,7 +792,7 @@ class MessageBox:
                 updates=diff.other_updates,
                 users=diff.users,
                 chats=diff.chats,
-                date=1,  # anything not-None
+                date=epoch(),
                 seq=NO_SEQ,  # this way date is not used
             ), chat_hashes, updates)
 
@@ -717,6 +807,8 @@ class MessageBox:
 
     def end_channel_difference(self, request, reason: PrematureEndReason, chat_hashes):
         entry = request.channel.channel_id
+        if __debug__:
+            self._trace('Ending channel difference for %r because %s', entry, reason)
 
         if reason == PrematureEndReason.TEMPORARY_SERVER_ISSUES:
             # Temporary issues. End getting difference without updating the pts so we can retry later.

telethon/_updates/session.py

@@ -1,7 +1,7 @@
 from typing import Optional, Tuple
 from enum import IntEnum
 from ..tl.types import InputPeerUser, InputPeerChat, InputPeerChannel
-
+import struct
 
 class SessionState:
     """
@@ -173,7 +173,7 @@ class Entity:
         try:
             ty, id, hash = struct.unpack('<Bqq', blob)
         except struct.error:
-            raise ValueError(f'malformed entity data, got {string!r}') from None
+            raise ValueError(f'malformed entity data, got {blob!r}') from None
 
         return cls(EntityType(ty), id, hash)
 

telethon/client/auth.py

@@ -19,8 +19,8 @@ class AuthMethods:
 
     def start(
             self: 'TelegramClient',
-            phone: typing.Callable[[], str] = lambda: input('Please enter your phone (or bot token): '),
+            phone: typing.Union[typing.Callable[[], str], str] = lambda: input('Please enter your phone (or bot token): '),
-            password: typing.Callable[[], str] = lambda: getpass.getpass('Please enter your password: '),
+            password: typing.Union[typing.Callable[[], str], str] = lambda: getpass.getpass('Please enter your password: '),
             *,
             bot_token: str = None,
             force_sms: bool = False,
@@ -34,12 +34,6 @@ class AuthMethods:
         By default, this method will be interactive (asking for
         user input if needed), and will handle 2FA if enabled too.
 
-        If the phone doesn't belong to an existing account (and will hence
-        `sign_up` for a new one),  **you are agreeing to Telegram's
-        Terms of Service. This is required and your account
-        will be banned otherwise.** See https://telegram.org/tos
-        and https://core.telegram.org/api/terms.
-
         If the event loop is already running, this method returns a
         coroutine that you should await on your own code; otherwise
         the loop is ran until said coroutine completes.
@@ -153,14 +147,16 @@ class AuthMethods:
                 if bot_token[:bot_token.find(':')] != str(me.id):
                     warnings.warn(
                         'the session already had an authorized user so it did '
-                        'not login to the bot account using the provided '
+                        'not login to the bot account using the provided bot_token; '
-                        'bot_token (it may not be using the user you expect)'
+                        'if you were expecting a different user, check whether '
+                        'you are accidentally reusing an existing session'
                     )
             elif phone and not callable(phone) and utils.parse_phone(phone) != me.phone:
                 warnings.warn(
                     'the session already had an authorized user so it did '
-                    'not login to the user account using the provided '
+                    'not login to the user account using the provided phone; '
-                    'phone (it may not be using the user you expect)'
+                    'if you were expecting a different user, check whether '
+                    'you are accidentally reusing an existing session'
                 )
 
             return self
@@ -188,7 +184,6 @@ class AuthMethods:
         two_step_detected = False
 
         await self.send_code_request(phone, force_sms=force_sms)
-        sign_up = False  # assume login
         while attempts < max_attempts:
             try:
                 value = code_callback()
@@ -201,19 +196,12 @@ class AuthMethods:
                 if not value:
                     raise errors.PhoneCodeEmptyError(request=None)
 
-                if sign_up:
-                    me = await self.sign_up(value, first_name, last_name)
-                else:
                 # Raises SessionPasswordNeededError if 2FA enabled
                 me = await self.sign_in(phone, code=value)
                 break
             except errors.SessionPasswordNeededError:
                 two_step_detected = True
                 break
-            except errors.PhoneNumberOccupiedError:
-                sign_up = False
-            except errors.PhoneNumberUnoccupiedError:
-                sign_up = True
             except (errors.PhoneCodeEmptyError,
                     errors.PhoneCodeExpiredError,
                     errors.PhoneCodeHashEmptyError,
@@ -253,12 +241,13 @@ class AuthMethods:
 
         # We won't reach here if any step failed (exit by exception)
         signed, name = 'Signed in successfully as ', utils.get_display_name(me)
+        tos = '; remember to not break the ToS or you will risk an account ban!'
         try:
-            print(signed, name)
+            print(signed, name, tos, sep='')
         except UnicodeEncodeError:
             # Some terminals don't support certain characters
             print(signed, name.encode('utf-8', errors='ignore')
-                              .decode('ascii', errors='ignore'))
+                              .decode('ascii', errors='ignore'), tos, sep='')
 
         return self
 
@@ -366,7 +355,12 @@ class AuthMethods:
                 'and a password only if an RPCError was raised before.'
             )
 
+        try:
             result = await self(request)
+        except errors.PhoneCodeExpiredError:
+            self._phone_code_hash.pop(phone, None)
+            raise
+
         if isinstance(result, types.auth.AuthorizationSignUpRequired):
             # Emulate pre-layer 104 behaviour
             self._tos = result.terms_of_service
@@ -383,91 +377,10 @@ class AuthMethods:
             phone: str = None,
             phone_code_hash: str = None) -> 'types.User':
         """
-        Signs up to Telegram as a new user account.
+        This method can no longer be used, and will immediately raise a ``ValueError``.
-
+        See `issue #4050 <https://github.com/LonamiWebs/Telethon/issues/4050>`_ for context.
-        Use this if you don't have an account yet.
-
-        You must call `send_code_request` first.
-
-        **By using this method you're agreeing to Telegram's
-        Terms of Service. This is required and your account
-        will be banned otherwise.** See https://telegram.org/tos
-        and https://core.telegram.org/api/terms.
-
-        Arguments
-            code (`str` | `int`):
-                The code sent by Telegram
-
-            first_name (`str`):
-                The first name to be used by the new account.
-
-            last_name (`str`, optional)
-                Optional last name.
-
-            phone (`str` | `int`, optional):
-                The phone to sign up. This will be the last phone used by
-                default (you normally don't need to set this).
-
-            phone_code_hash (`str`, optional):
-                The hash returned by `send_code_request`. This can be left as
-                `None` to use the last hash known for the phone to be used.
-
-        Returns
-            The new created :tl:`User`.
-
-        Example
-            .. code-block:: python
-
-                phone = '+34 123 123 123'
-                await client.send_code_request(phone)
-
-                code = input('enter code: ')
-                await client.sign_up(code, first_name='Anna', last_name='Banana')
         """
-        me = await self.get_me()
+        raise ValueError('Third-party applications cannot sign up for Telegram. See https://github.com/LonamiWebs/Telethon/issues/4050 for details')
-        if me:
-            return me
-
-        # To prevent abuse, one has to try to sign in before signing up. This
-        # is the current way in which Telegram validates the code to sign up.
-        #
-        # `sign_in` will set `_tos`, so if it's set we don't need to call it
-        # because the user already tried to sign in.
-        #
-        # We're emulating pre-layer 104 behaviour so except the right error:
-        if not self._tos:
-            try:
-                return await self.sign_in(
-                    phone=phone,
-                    code=code,
-                    phone_code_hash=phone_code_hash,
-                )
-            except errors.PhoneNumberUnoccupiedError:
-                pass  # code is correct and was used, now need to sign in
-
-        if self._tos and self._tos.text:
-            if self.parse_mode:
-                t = self.parse_mode.unparse(self._tos.text, self._tos.entities)
-            else:
-                t = self._tos.text
-            sys.stderr.write("{}\n".format(t))
-            sys.stderr.flush()
-
-        phone, phone_code_hash = \
-            self._parse_phone_and_hash(phone, phone_code_hash)
-
-        result = await self(functions.auth.SignUpRequest(
-            phone_number=phone,
-            phone_code_hash=phone_code_hash,
-            first_name=first_name,
-            last_name=last_name
-        ))
-
-        if self._tos:
-            await self(
-                functions.help.AcceptTermsOfServiceRequest(self._tos.id))
-
-        return await self._on_login(result.user)
 
     async def _on_login(self, user):
         """
@@ -475,11 +388,20 @@ class AuthMethods:
 
         Returns the input user parameter.
         """
-        self._bot = bool(user.bot)
+        self._mb_entity_cache.set_self_user(user.id, user.bot, user.access_hash)
-        self._self_input_peer = utils.get_input_peer(user, allow_self=False)
         self._authorized = True
 
         state = await self(functions.updates.GetStateRequest())
+        # the server may send an old qts in getState
+        difference = await self(functions.updates.GetDifferenceRequest(pts=state.pts, date=state.date, qts=state.qts))
+
+        if isinstance(difference, types.updates.Difference):
+            state = difference.state
+        elif isinstance(difference, types.updates.DifferenceSlice):
+            state = difference.intermediate_state
+        elif isinstance(difference, types.updates.DifferenceTooLong):
+            state.pts = difference.pts
+
         self._message_box.load(SessionState(0, 0, 0, state.pts, state.qts, int(state.date.timestamp()), state.seq, 0), [])
 
         return user
@@ -498,7 +420,8 @@ class AuthMethods:
                 The phone to which the code will be sent.
 
             force_sms (`bool`, optional):
-                Whether to force sending as SMS.
+                Whether to force sending as SMS. This has been deprecated.
+                See `issue #4050 <https://github.com/LonamiWebs/Telethon/issues/4050>`_ for context.
 
         Returns
             An instance of :tl:`SentCode`.
@@ -510,6 +433,10 @@ class AuthMethods:
                 sent = await client.send_code_request(phone)
                 print(sent)
         """
+        if force_sms:
+            warnings.warn('force_sms has been deprecated and no longer works')
+            force_sms = False
+
         result = None
         phone = utils.parse_phone(phone) or self._phone
         phone_hash = self._phone_code_hash.get(phone)
@@ -524,6 +451,10 @@ class AuthMethods:
                 return await self.send_code_request(
                     phone, force_sms=force_sms, _retry_count=_retry_count+1)
 
+            # TODO figure out when/if/how this can happen
+            if isinstance(result, types.auth.SentCodeSuccess):
+                raise RuntimeError('logged in right after sending the code')
+
             # If we already sent a SMS, do not resend the code (hash may be empty)
             if isinstance(result.type, types.auth.SentCodeTypeSms):
                 force_sms = False
@@ -550,6 +481,9 @@ class AuthMethods:
                 return await self.send_code_request(
                     phone, force_sms=False, _retry_count=_retry_count+1)
 
+            if isinstance(result, types.auth.SentCodeSuccess):
+                raise RuntimeError('logged in right after resending the code')
+
             self._phone_code_hash[phone] = result.phone_code_hash
 
         return result
@@ -614,12 +548,12 @@ class AuthMethods:
         except errors.RPCError:
             return False
 
-        self._bot = None
+        self._mb_entity_cache.set_self_user(None, None, None)
-        self._self_input_peer = None
         self._authorized = False
 
         await self.disconnect()
-        self.session.delete()
+        await utils.maybe_async(self.session.delete())
+        self.session = None
         return True
 
     async def edit_2fa(

telethon/client/buttons.py

@@ -7,8 +7,8 @@ from ..tl import types, custom
 class ButtonMethods:
     @staticmethod
     def build_reply_markup(
-            buttons: 'typing.Optional[hints.MarkupLike]',
+            buttons: 'typing.Optional[hints.MarkupLike]'
-            inline_only: bool = False) -> 'typing.Optional[types.TypeReplyMarkup]':
+    ) -> 'typing.Optional[types.TypeReplyMarkup]':
         """
         Builds a :tl:`ReplyInlineMarkup` or :tl:`ReplyKeyboardMarkup` for
         the given buttons.
@@ -26,9 +26,6 @@ class ButtonMethods:
                 The button, list of buttons, array of buttons or markup
                 to convert into a markup.
 
-            inline_only (`bool`, optional):
-                Whether the buttons **must** be inline buttons only or not.
-
         Example
             .. code-block:: python
 
@@ -42,8 +39,8 @@ class ButtonMethods:
             return None
 
         try:
-            if buttons.SUBCLASS_OF_ID == 0xe2e10ef2:
+            if buttons.SUBCLASS_OF_ID == 0xe2e10ef2:  # crc32(b'ReplyMarkup'):
-                return buttons  # crc32(b'ReplyMarkup'):
+                return buttons
         except AttributeError:
             pass
 
@@ -57,6 +54,8 @@ class ButtonMethods:
         resize = None
         single_use = None
         selective = None
+        persistent = None
+        placeholder = None
 
         rows = []
         for row in buttons:
@@ -69,6 +68,10 @@ class ButtonMethods:
                         single_use = button.single_use
                     if button.selective is not None:
                         selective = button.selective
+                    if button.persistent is not None:
+                        persistent = button.persistent
+                    if button.placeholder is not None:
+                        placeholder = button.placeholder
 
                     button = button.button
                 elif isinstance(button, custom.MessageButton):
@@ -78,19 +81,21 @@ class ButtonMethods:
                 is_inline |= inline
                 is_normal |= not inline
 
-                if button.SUBCLASS_OF_ID == 0xbad74a3:
+                if button.SUBCLASS_OF_ID == 0xbad74a3:  # crc32(b'KeyboardButton')
-                    # 0xbad74a3 == crc32(b'KeyboardButton')
                     current.append(button)
 
             if current:
                 rows.append(types.KeyboardButtonRow(current))
 
-        if inline_only and is_normal:
+        if is_inline and is_normal:
-            raise ValueError('You cannot use non-inline buttons here')
-        elif is_inline == is_normal and is_normal:
             raise ValueError('You cannot mix inline with normal buttons')
         elif is_inline:
             return types.ReplyInlineMarkup(rows)
-        # elif is_normal:
         return types.ReplyKeyboardMarkup(
-            rows, resize=resize, single_use=single_use, selective=selective)
+            rows=rows,
+            resize=resize,
+            single_use=single_use,
+            selective=selective,
+            persistent=persistent,
+            placeholder=placeholder
+        )

telethon/client/chats.py

@@ -218,8 +218,10 @@ class _ParticipantsIter(RequestIter):
         self.requests.offset += len(participants.participants)
         users = {user.id: user for user in participants.users}
         for participant in participants.participants:
-
+            if isinstance(participant, types.ChannelParticipantLeft):
-            if isinstance(participant, types.ChannelParticipantBanned):
+                # See issue #3231 to learn why this is ignored.
+                continue
+            elif isinstance(participant, types.ChannelParticipantBanned):
                 if not isinstance(participant.peer, types.PeerUser):
                     # May have the entire channel banned. See #3105.
                     continue
@@ -926,9 +928,6 @@ class ChatMethods:
         """
         entity = await self.get_input_entity(entity)
         user = await self.get_input_entity(user)
-        ty = helpers._entity_type(user)
-        if ty != helpers._EntityType.USER:
-            raise ValueError('You must pass a user entity')
 
         perm_names = (
             'change_info', 'post_messages', 'edit_messages', 'delete_messages',
@@ -966,7 +965,7 @@ class ChatMethods:
                 is_admin = any(locals()[x] for x in perm_names)
 
             return await self(functions.messages.EditChatAdminRequest(
-                entity, user, is_admin=is_admin))
+                entity.chat_id, user, is_admin=is_admin))
 
         else:
             raise ValueError(
@@ -1115,12 +1114,6 @@ class ChatMethods:
             ))
 
         user = await self.get_input_entity(user)
-        ty = helpers._entity_type(user)
-        if ty != helpers._EntityType.USER:
-            raise ValueError('You must pass a user entity')
-
-        if isinstance(user, types.InputPeerSelf):
-            raise ValueError('You cannot restrict yourself')
 
         return await self(functions.channels.EditBannedRequest(
             channel=entity,
@@ -1167,8 +1160,6 @@ class ChatMethods:
         """
         entity = await self.get_input_entity(entity)
         user = await self.get_input_entity(user)
-        if helpers._entity_type(user) != helpers._EntityType.USER:
-            raise ValueError('You must pass a user entity')
 
         ty = helpers._entity_type(entity)
         if ty == helpers._EntityType.CHAT:
@@ -1245,8 +1236,6 @@ class ChatMethods:
 
         entity = await self.get_input_entity(entity)
         user = await self.get_input_entity(user)
-        if helpers._entity_type(user) != helpers._EntityType.USER:
-            raise ValueError('You must pass a user entity')
         if helpers._entity_type(entity) == helpers._EntityType.CHANNEL:
             participant = await self(functions.channels.GetParticipantRequest(
                 entity,

telethon/client/dialogs.py

@@ -58,6 +58,8 @@ class _DialogsIter(RequestIter):
                     for x in itertools.chain(r.users, r.chats)
                     if not isinstance(x, (types.UserEmpty, types.ChatEmpty))}
 
+        self.client._mb_entity_cache.extend(r.users, r.chats)
+
         messages = {}
         for m in r.messages:
             m._finish_init(self.client, entities, None)
@@ -82,14 +84,16 @@ class _DialogsIter(RequestIter):
 
                 cd = custom.Dialog(self.client, d, entities, message)
                 if cd.dialog.pts:
-                    self.client._channel_pts[cd.id] = cd.dialog.pts
+                    self.client._message_box.try_set_channel_state(
+                        utils.get_peer_id(d.peer, add_mark=False), cd.dialog.pts)
 
                 if not self.ignore_migrated or getattr(
                         cd.entity, 'migrated_to', None) is None:
                     self.buffer.append(cd)
 
-        if len(r.dialogs) < self.request.limit\
+        if not self.buffer or len(r.dialogs) < self.request.limit\
                 or not isinstance(r, types.messages.DialogsSlice):
+            # Buffer being empty means all returned dialogs were skipped (due to offsets).
             # Less than we requested means we reached the end, or
             # we didn't get a DialogsSlice which means we got all.
             return True

telethon/client/downloads.py

@@ -27,12 +27,22 @@ MAX_CHUNK_SIZE = 512 * 1024
 # 2021-01-15, users reported that `errors.TimeoutError` can occur while downloading files.
 TIMED_OUT_SLEEP = 1
 
+
+class _CdnRedirect(Exception):
+    def __init__(self, cdn_redirect=None):
+        self.cdn_redirect = cdn_redirect
+      
+  
 class _DirectDownloadIter(RequestIter):
     async def _init(
-            self, file, dc_id, offset, stride, chunk_size, request_size, file_size, msg_data
+            self, file, dc_id, offset, stride, chunk_size, request_size, file_size, msg_data, cdn_redirect=None):
-    ):
         self.request = functions.upload.GetFileRequest(
             file, offset=offset, limit=request_size) 
+        self._client = self.client
+        self._cdn_redirect = cdn_redirect
+        if cdn_redirect is not None:
+          self.request = functions.upload.GetCdnFileRequest(cdn_redirect.file_token, offset=offset, limit=request_size)
+          self._client = await self.client._get_cdn_client(cdn_redirect)
         
         self.total = file_size
         self._stride = stride
@@ -41,7 +51,7 @@ class _DirectDownloadIter(RequestIter):
         self._msg_data = msg_data
         self._timed_out = False
         
-        self._exported = dc_id and self.client.session.dc_id != dc_id
+        self._exported = dc_id and self._client.session.dc_id != dc_id
         if not self._exported:
             # The used sender will also change if ``FileMigrateError`` occurs
             self._sender = self.client._sender
@@ -53,9 +63,12 @@ class _DirectDownloadIter(RequestIter):
                 config = await self.client(functions.help.GetConfigRequest())
                 for option in config.dc_options:
                     if option.ip_address == self.client.session.server_address:
+                        await utils.maybe_async(
                             self.client.session.set_dc(
-                            option.id, option.ip_address, option.port)
+                                option.id, option.ip_address, option.port
-                        self.client.session.save()
+                            )
+                        )
+                        await utils.maybe_async(self.client.session.save())
                         break
 
                 # TODO Figure out why the session may have the wrong DC ID
@@ -73,14 +86,20 @@ class _DirectDownloadIter(RequestIter):
 
     async def _request(self):
         try:
-            result = await self.client._call(self._sender, self.request)
+            result = await self._client._call(self._sender, self.request)
             self._timed_out = False
             if isinstance(result, types.upload.FileCdnRedirect):
-                raise NotImplementedError  # TODO Implement
+                if self.client._mb_entity_cache.self_bot:
+                    raise ValueError('FileCdnRedirect but the GetCdnFileRequest API access for bot users is restricted. Try to change api_id to avoid FileCdnRedirect')
+                raise _CdnRedirect(result)
+            if isinstance(result, types.upload.CdnFileReuploadNeeded):
+                await self.client._call(self.client._sender, functions.upload.ReuploadCdnFileRequest(file_token=self._cdn_redirect.file_token, request_token=result.request_token))
+                result = await self._client._call(self._sender, self.request)
+                return result.bytes
             else:
                 return result.bytes
 
-        except errors.TimeoutError as e:
+        except errors.TimedOutError as e:
             if self._timed_out:
                 self.client._log[__name__].warning('Got two timeouts in a row while downloading file')
                 raise
@@ -96,7 +115,7 @@ class _DirectDownloadIter(RequestIter):
             self._exported = True
             return await self._request()
 
-        except errors.FilerefUpgradeNeededError as e:
+        except (errors.FilerefUpgradeNeededError, errors.FileReferenceExpiredError) as e:
             # Only implemented for documents which are the ones that may take that long to download
             if not self._msg_data \
                     or not isinstance(self.request.location, types.InputDocumentFileLocation) \
@@ -223,7 +242,8 @@ class DownloadMethods:
                 The output file path, directory, or stream-like object.
                 If the path exists and is a file, it will be overwritten.
                 If file is the type `bytes`, it will be downloaded in-memory
-                as a bytestring (e.g. ``file=bytes``).
+                and returned as a bytestring (i.e. ``file=bytes``, without
+                parentheses or quotes).
 
             download_big (`bool`, optional):
                 Whether to use the big version of the available photos.
@@ -272,7 +292,9 @@ class DownloadMethods:
         if isinstance(photo, (types.UserProfilePhoto, types.ChatPhoto)):
             dc_id = photo.dc_id
             loc = types.InputPeerPhotoFileLocation(
-                peer=await self.get_input_entity(entity),
+                # min users can be used to download profile photos
+                # self.get_input_entity would otherwise not accept those
+                peer=utils.get_input_peer(entity, check_hash=False),
                 photo_id=photo.photo_id,
                 big=download_big
             )
@@ -331,7 +353,8 @@ class DownloadMethods:
                 The output file path, directory, or stream-like object.
                 If the path exists and is a file, it will be overwritten.
                 If file is the type `bytes`, it will be downloaded in-memory
-                as a bytestring (e.g. ``file=bytes``).
+                and returned as a bytestring (i.e. ``file=bytes``, without
+                parentheses or quotes).
 
             progress_callback (`callable`, optional):
                 A callback function accepting two parameters:
@@ -374,6 +397,9 @@ class DownloadMethods:
                 path = await message.download_media()
                 await message.download_media(filename)
 
+                # Downloading to memory
+                blob = await client.download_media(message, bytes)
+
                 # Printing download progress
                 def callback(current, total):
                     print('Downloaded', current, 'out of', total,
@@ -509,7 +535,9 @@ class DownloadMethods:
             dc_id: int = None,
             key: bytes = None,
             iv: bytes = None,
-            msg_data: tuple = None) -> typing.Optional[bytes]:
+            msg_data: tuple = None,
+            cdn_redirect: types.upload.FileCdnRedirect = None
+    ) -> typing.Optional[bytes]:
         if not part_size_kb:
             if not file_size:
                 part_size_kb = 64  # Reasonable default
@@ -536,7 +564,7 @@ class DownloadMethods:
 
         try:
             async for chunk in self._iter_download(
-                    input_location, request_size=part_size, dc_id=dc_id, msg_data=msg_data):
+                    input_location, request_size=part_size, dc_id=dc_id, msg_data=msg_data, cdn_redirect=cdn_redirect):
                 if iv and key:
                     chunk = AES.decrypt_ige(chunk, key, iv)
                 r = f.write(chunk)
@@ -554,6 +582,20 @@ class DownloadMethods:
 
             if in_memory:
                 return f.getvalue()
+        except _CdnRedirect as e:
+          self._log[__name__].info('FileCdnRedirect to CDN data center %s', e.cdn_redirect.dc_id)
+          return await self._download_file(
+              input_location=input_location,
+              file=file,
+              part_size_kb=part_size_kb,
+              file_size=file_size,
+              progress_callback=progress_callback,
+              dc_id=e.cdn_redirect.dc_id,
+              key=e.cdn_redirect.encryption_key,
+              iv=e.cdn_redirect.encryption_iv,
+              msg_data=msg_data,
+              cdn_redirect=e.cdn_redirect
+          )
         finally:
             if isinstance(file, str) or in_memory:
                 f.close()
@@ -675,7 +717,8 @@ class DownloadMethods:
             request_size: int = MAX_CHUNK_SIZE,
             file_size: int = None,
             dc_id: int = None,
-            msg_data: tuple = None
+            msg_data: tuple = None,
+            cdn_redirect: types.upload.FileCdnRedirect = None
     ):
         info = utils._get_file_info(file)
         if info.dc_id is not None:
@@ -726,6 +769,7 @@ class DownloadMethods:
             request_size=request_size,
             file_size=file_size,
             msg_data=msg_data,
+            cdn_redirect=cdn_redirect
         )
 
     # endregion
@@ -734,6 +778,9 @@ class DownloadMethods:
 
     @staticmethod
     def _get_thumb(thumbs, thumb):
+        if not thumbs:
+            return None
+
         # Seems Telegram has changed the order and put `PhotoStrippedSize`
         # last while this is the smallest (layer 116). Ensure we have the
         # sizes sorted correctly with a custom function.
@@ -876,6 +923,9 @@ class DownloadMethods:
         else:
             file = self._get_proper_filename(file, 'photo', '.jpg', date=date)
             size = self._get_thumb(document.thumbs, thumb)
+            if not size or isinstance(size, types.PhotoSizeEmpty):
+                return
+
             if isinstance(size, (types.PhotoCachedSize, types.PhotoStrippedSize)):
                 return self._download_cached_photo_size(size, file)
 
@@ -916,22 +966,19 @@ class DownloadMethods:
             'END:VCARD\n'
         ).format(f=first_name, l=last_name, p=phone_number).encode('utf-8')
 
-        if file is bytes:
-            return result
-        elif isinstance(file, str):
         file = cls._get_proper_filename(
             file, 'contact', '.vcard',
             possible_names=[first_name, phone_number, last_name]
         )
-            f = open(file, 'wb')
+        if file is bytes:
-        else:
+            return result
-            f = file
+        f = file if hasattr(file, 'write') else open(file, 'wb')
 
         try:
             f.write(result)
         finally:
             # Only close the stream if we opened it
-            if isinstance(file, str):
+            if f != file:
                 f.close()
 
         return file
@@ -948,18 +995,17 @@ class DownloadMethods:
             )
 
         # TODO Better way to get opened handles of files and auto-close
-        in_memory = file is bytes
-        if in_memory:
-            f = io.BytesIO()
-        elif isinstance(file, str):
         kind, possible_names = cls._get_kind_and_names(web.attributes)
         file = cls._get_proper_filename(
             file, kind, utils.get_extension(web),
             possible_names=possible_names
         )
-            f = open(file, 'wb')
+        if file is bytes:
-        else:
+            f = io.BytesIO()
+        elif hasattr(file, 'write'):
             f = file
+        else:
+            f = open(file, 'wb')
 
         try:
             async with aiohttp.ClientSession() as session:
@@ -972,10 +1018,10 @@ class DownloadMethods:
                             break
                         f.write(chunk)
         finally:
-            if isinstance(file, str) or file is bytes:
+            if f != file:
                 f.close()
 
-        return f.getvalue() if in_memory else file
+        return f.getvalue() if file is bytes else file
 
     @staticmethod
     def _get_proper_filename(file, kind, extension,

telethon/client/messages.py

@@ -204,7 +204,24 @@ class _MessagesIter(RequestIter):
             message._finish_init(self.client, entities, self.entity)
             self.buffer.append(message)
 
-        if len(r.messages) < self.request.limit:
+        # Not a slice (using offset would return the same, with e.g. SearchGlobal).
+        if isinstance(r, types.messages.Messages):
+            return True
+
+        # Some channels are "buggy" and may return less messages than
+        # requested (apparently, the messages excluded are, for example,
+        # "not displayable due to local laws").
+        #
+        # This means it's not safe to rely on `len(r.messages) < req.limit` as
+        # the stop condition. Unfortunately more requests must be made.
+        #
+        # However we can still check if the highest ID is equal to or lower
+        # than the limit, in which case there won't be any more messages
+        # because the lowest message ID is 1.
+        #
+        # We also assume the API will always return, at least, one message if
+        # there is more to fetch.
+        if not r.messages or (not self.reverse and r.messages[0].id <= self.request.limit):
             return True
 
         # Get the last message that's not empty (in some rare cases
@@ -536,7 +553,9 @@ class MessageMethods:
             scheduled=scheduled
         )
 
-    async def get_messages(self: 'TelegramClient', *args, **kwargs) -> 'hints.TotalList':
+    async def get_messages(
+            self: 'TelegramClient', *args, **kwargs
+    ) -> typing.Union['hints.TotalList', typing.Optional['types.Message']]:
         """
         Same as `iter_messages()`, but returns a
         `TotalList <telethon.helpers.TotalList>` instead.
@@ -600,7 +619,7 @@ class MessageMethods:
             peer=entity,
             msg_id=utils.get_message_id(message)
         ))
-        m = r.messages[0]
+        m = min(r.messages, key=lambda msg: msg.id)
         chat = next(c for c in r.chats if c.id == m.peer_id.channel_id)
         return utils.get_input_peer(chat), m.id
 
@@ -623,7 +642,10 @@ class MessageMethods:
             background: bool = None,
             supports_streaming: bool = False,
             schedule: 'hints.DateLike' = None,
-            comment_to: 'typing.Union[int, types.Message]' = None
+            comment_to: 'typing.Union[int, types.Message]' = None,
+            nosound_video: bool = None,
+            send_as: typing.Optional['hints.EntityLike'] = None,
+            message_effect_id: typing.Optional[int] = None
     ) -> 'types.Message':
         """
         Sends a message to the specified user, chat or channel.
@@ -737,6 +759,25 @@ class MessageMethods:
                 This parameter takes precedence over ``reply_to``. If there is
                 no linked chat, `telethon.errors.sgIdInvalidError` is raised.
 
+            nosound_video (`bool`, optional):
+                Only applicable when sending a video file without an audio
+                track. If set to ``True``, the video will be displayed in
+                Telegram as a video. If set to ``False``, Telegram will attempt
+                to display the video as an animated gif. (It may still display
+                as a video due to other factors.) The value is ignored if set
+                on non-video files. This is set to ``True`` for albums, as gifs
+                cannot be sent in albums.
+
+            send_as (`entity`):
+                Unique identifier (int) or username (str) of the chat or channel to send the message as.
+                You can use this to send the message on behalf of a chat or channel where you have appropriate permissions.
+                Use the GetSendAs to return the list of message sender identifiers, which can be used to send messages in the chat,
+                This setting applies to the current message and will remain effective for future messages unless explicitly changed.
+                To set this behavior permanently for all messages, use SaveDefaultSendAs.
+
+            message_effect_id (`int`, optional):
+                Unique identifier of the message effect to be added to the message; for private chats only
+
         Returns
             The sent `custom.Message <telethon.tl.custom.message.Message>`.
 
@@ -797,6 +838,9 @@ class MessageMethods:
                 await client.send_message(chat, 'Hi, future!', schedule=timedelta(minutes=5))
         """
         if file is not None:
+            if isinstance(message, types.Message):
+                formatting_entities = formatting_entities or message.entities
+                message = message.message
             return await self.send_file(
                 entity, file, caption=message, reply_to=reply_to,
                 attributes=attributes, parse_mode=parse_mode,
@@ -804,12 +848,16 @@ class MessageMethods:
                 buttons=buttons, clear_draft=clear_draft, silent=silent,
                 schedule=schedule, supports_streaming=supports_streaming,
                 formatting_entities=formatting_entities,
-                comment_to=comment_to, background=background
+                comment_to=comment_to, background=background,
+                nosound_video=nosound_video,
+                send_as=send_as, message_effect_id=message_effect_id
             )
 
         entity = await self.get_input_entity(entity)
         if comment_to is not None:
             entity, reply_to = await self._get_comment_data(entity, comment_to)
+        else:
+            reply_to = utils.get_message_id(reply_to)
 
         if isinstance(message, types.Message):
             if buttons is None:
@@ -831,7 +879,9 @@ class MessageMethods:
                     reply_to=reply_to,
                     buttons=markup,
                     formatting_entities=message.entities,
-                    schedule=schedule
+                    parse_mode=None,  # explicitly disable parse_mode to force using even empty formatting_entities
+                    schedule=schedule,
+                    send_as=send_as, message_effect_id=message_effect_id
                 )
 
             request = functions.messages.SendMessageRequest(
@@ -839,13 +889,15 @@ class MessageMethods:
                 message=message.message or '',
                 silent=silent,
                 background=background,
-                reply_to_msg_id=utils.get_message_id(reply_to),
+                reply_to=None if reply_to is None else types.InputReplyToMessage(reply_to),
                 reply_markup=markup,
                 entities=message.entities,
                 clear_draft=clear_draft,
                 no_webpage=not isinstance(
                     message.media, types.MessageMediaWebPage),
-                schedule_date=schedule
+                schedule_date=schedule,
+                send_as=await self.get_input_entity(send_as) if send_as else None,
+                effect=message_effect_id
             )
             message = message.message
         else:
@@ -861,12 +913,14 @@ class MessageMethods:
                 message=message,
                 entities=formatting_entities,
                 no_webpage=not link_preview,
-                reply_to_msg_id=utils.get_message_id(reply_to),
+                reply_to=None if reply_to is None else types.InputReplyToMessage(reply_to),
                 clear_draft=clear_draft,
                 silent=silent,
                 background=background,
                 reply_markup=self.build_reply_markup(buttons),
-                schedule_date=schedule
+                schedule_date=schedule,
+                send_as=await self.get_input_entity(send_as) if send_as else None,
+                effect=message_effect_id
             )
 
         result = await self(request)
@@ -881,7 +935,7 @@ class MessageMethods:
                 entities=result.entities,
                 reply_markup=request.reply_markup,
                 ttl_period=result.ttl_period,
-                reply_to=types.MessageReplyHeader(request.reply_to_msg_id)
+                reply_to=request.reply_to
             )
             message._finish_init(self, {}, entity)
             return message
@@ -898,7 +952,9 @@ class MessageMethods:
             with_my_score: bool = None,
             silent: bool = None,
             as_album: bool = None,
-            schedule: 'hints.DateLike' = None
+            schedule: 'hints.DateLike' = None,
+            drop_author: bool = None,
+            drop_media_captions: bool = None,
     ) -> 'typing.Sequence[types.Message]':
         """
         Forwards the given messages to the specified entity.
@@ -942,6 +998,12 @@ class MessageMethods:
                 instead they will be scheduled to be automatically sent
                 at a later time.
 
+            drop_author (`bool`, optional):
+                Whether to forward messages without quoting the original author.
+
+            drop_media_captions (`bool`, optional):
+                Whether to strip captions from media. Setting this to `True` requires that `drop_author` also be set to `True`.
+
         Returns
             The list of forwarded `Message <telethon.tl.custom.message.Message>`,
             or a single one if a list wasn't provided as input.
@@ -1000,7 +1062,7 @@ class MessageMethods:
             if isinstance(chunk[0], int):
                 chat = from_peer
             else:
-                chat = await chunk[0].get_input_chat()
+                chat = from_peer or await self.get_input_entity(chunk[0].peer_id)
                 chunk = [m.id for m in chunk]
 
             req = functions.messages.ForwardMessagesRequest(
@@ -1010,7 +1072,9 @@ class MessageMethods:
                 silent=silent,
                 background=background,
                 with_my_score=with_my_score,
-                schedule_date=schedule
+                schedule_date=schedule,
+                drop_author=drop_author,
+                drop_media_captions=drop_media_captions
             )
             result = await self(req)
             sent.extend(self._get_response_message(req, result, entity))
@@ -1020,7 +1084,7 @@ class MessageMethods:
     async def edit_message(
             self: 'TelegramClient',
             entity: 'typing.Union[hints.EntityLike, types.Message]',
-            message: 'hints.MessageLike' = None,
+            message: 'typing.Union[int, types.Message, types.InputMessageID, str]' = None,
             text: str = None,
             *,
             parse_mode: str = (),
@@ -1046,11 +1110,11 @@ class MessageMethods:
                 from it, so the next parameter will be assumed to be the
                 message text.
 
-                You may also pass a :tl:`InputBotInlineMessageID`,
+                You may also pass a :tl:`InputBotInlineMessageID` or :tl:`InputBotInlineMessageID64`,
                 which is the only way to edit messages that were sent
                 after the user selects an inline query result.
 
-            message (`int` | `Message <telethon.tl.custom.message.Message>` | `str`):
+            message (`int` | `Message <telethon.tl.custom.message.Message>` | :tl:`InputMessageID` | `str`):
                 The ID of the message (or `Message
                 <telethon.tl.custom.message.Message>` itself) to be edited.
                 If the `entity` was a `Message
@@ -1118,7 +1182,7 @@ class MessageMethods:
 
         Returns
             The edited `Message <telethon.tl.custom.message.Message>`,
-            unless `entity` was a :tl:`InputBotInlineMessageID` in which
+            unless `entity` was a :tl:`InputBotInlineMessageID` or :tl:`InputBotInlineMessageID64` in which
             case this method returns a boolean.
 
         Raises
@@ -1144,7 +1208,7 @@ class MessageMethods:
                 # or
                 await client.edit_message(message, 'hello!!!')
         """
-        if isinstance(entity, types.InputBotInlineMessageID):
+        if isinstance(entity, (types.InputBotInlineMessageID, types.InputBotInlineMessageID64)):
             text = text or message
             message = entity
         elif isinstance(entity, types.Message):
@@ -1160,7 +1224,7 @@ class MessageMethods:
                 attributes=attributes,
                 force_document=force_document)
 
-        if isinstance(entity, types.InputBotInlineMessageID):
+        if isinstance(entity, (types.InputBotInlineMessageID, types.InputBotInlineMessageID64)):
             request = functions.messages.EditInlineBotMessageRequest(
                 id=entity,
                 message=text,

telethon/client/telegrambaseclient.py

@@ -1,4 +1,5 @@
 import abc
+import inspect
 import re
 import asyncio
 import collections
@@ -7,10 +8,10 @@ import platform
 import time
 import typing
 import datetime
+import pathlib
 
-from .. import version, helpers, __name__ as __base_name__
+from .. import utils, version, helpers, __name__ as __base_name__
 from ..crypto import rsa
-from ..entitycache import EntityCache
 from ..extensions import markdown
 from ..network import MTProtoSender, Connection, ConnectionTcpFull, TcpMTProxy
 from ..sessions import Session, SQLiteSession, MemorySession
@@ -192,7 +193,7 @@ class TelegramBaseClient(abc.ABC):
             Defaults to `lang_code`.
 
         loop (`asyncio.AbstractEventLoop`, optional):
-            Asyncio event loop to use. Defaults to `asyncio.get_event_loop()`.
+            Asyncio event loop to use. Defaults to `asyncio.get_running_loop()`.
             This argument is ignored.
 
         base_logger (`str` | `logging.Logger`, optional):
@@ -209,6 +210,20 @@ class TelegramBaseClient(abc.ABC):
             so event handlers, conversations, and QR login will not work.
             However, certain scripts don't need updates, so this will reduce
             the amount of bandwidth used.
+
+        entity_cache_limit (`int`, optional):
+            How many users, chats and channels to keep in the in-memory cache
+            at most. This limit is checked against when processing updates.
+
+            When this limit is reached or exceeded, all entities that are not
+            required for update handling will be flushed to the session file.
+
+            Note that this implies that there is a lower bound to the amount
+            of entities that must be kept in memory.
+
+            Setting this limit too low will cause the library to attempt to
+            flush entities to the session file even if no entities can be
+            removed from the in-memory cache, which will degrade performance.
     """
 
     # Current TelegramClient version
@@ -222,7 +237,7 @@ class TelegramBaseClient(abc.ABC):
 
     def __init__(
             self: 'TelegramClient',
-            session: 'typing.Union[str, Session]',
+            session: 'typing.Union[str, pathlib.Path, Session]',
             api_id: int,
             api_hash: str,
             *,
@@ -246,7 +261,8 @@ class TelegramBaseClient(abc.ABC):
             loop: asyncio.AbstractEventLoop = None,
             base_logger: typing.Union[str, logging.Logger] = None,
             receive_updates: bool = True,
-            catch_up: bool = False
+            catch_up: bool = False,
+            entity_cache_limit: int = 5000
     ):
         if not api_id or not api_hash:
             raise ValueError(
@@ -270,9 +286,9 @@ class TelegramBaseClient(abc.ABC):
         self._log = _Loggers()
 
         # Determine what session object we have
-        if isinstance(session, str) or session is None:
+        if isinstance(session, (str, pathlib.Path)):
             try:
-                session = SQLiteSession(session)
+                session = SQLiteSession(str(session))
             except ImportError:
                 import warnings
                 warnings.warn(
@@ -283,24 +299,17 @@ class TelegramBaseClient(abc.ABC):
                     'you use another session storage'
                 )
                 session = MemorySession()
+        elif session is None:
+            session = MemorySession()
         elif not isinstance(session, Session):
             raise TypeError(
                 'The given session must be a str or a Session instance.'
             )
 
-        # ':' in session.server_address is True if it's an IPv6 address
-        if (not session.server_address or
-                (':' in session.server_address) != use_ipv6):
-            session.set_dc(
-                DEFAULT_DC_ID,
-                DEFAULT_IPV6_IP if self._use_ipv6 else DEFAULT_IPV4_IP,
-                DEFAULT_PORT
-            )
-
         self.flood_sleep_threshold = flood_sleep_threshold
 
         # TODO Use AsyncClassWrapper(session)
-        # ChatGetter and SenderGetter can use the in-memory _entity_cache
+        # ChatGetter and SenderGetter can use the in-memory _mb_entity_cache
         # to avoid network access and the need for await in session files.
         #
         # The session files only wants the entities to persist
@@ -308,7 +317,6 @@ class TelegramBaseClient(abc.ABC):
         # TODO Session should probably return all cached
         #      info of entities, not just the input versions
         self.session = session
-        self._entity_cache = EntityCache()
         self.api_id = int(api_id)
         self.api_hash = api_hash
 
@@ -385,12 +393,13 @@ class TelegramBaseClient(abc.ABC):
         # Cache ``{dc_id: (_ExportState, MTProtoSender)}`` for all borrowed senders
         self._borrowed_senders = {}
         self._borrow_sender_lock = asyncio.Lock()
+        self._exported_sessions = {}
 
+        self._loop = None  # only used as a sanity check
         self._updates_error = None
         self._updates_handle = None
         self._keepalive_handle = None
         self._last_request = time.time()
-        self._channel_pts = {}
         self._no_updates = not receive_updates
 
         # Used for non-sequential updates, in order to terminate all pending tasks on disconnect.
@@ -423,19 +432,15 @@ class TelegramBaseClient(abc.ABC):
         self._phone = None
         self._tos = None
 
-        # Sometimes we need to know who we are, cache the self peer
-        self._self_input_peer = None
-        self._bot = None
-
         # A place to store if channels are a megagroup or not (see `edit_admin`)
         self._megagroup_cache = {}
 
         # This is backported from v2 in a very ad-hoc way just to get proper update handling
         self._catch_up = catch_up
         self._updates_queue = asyncio.Queue()
-        self._message_box = MessageBox()
+        self._message_box = MessageBox(self._log['messagebox'])
-        # This entity cache is tailored for the messagebox and is not used for absolutely everything like _entity_cache
         self._mb_entity_cache = MbEntityCache()  # required for proper update handling (to know when to getDifference)
+        self._entity_cache_limit = entity_cache_limit
 
         self._sender = MTProtoSender(
             self.session.auth_key,
@@ -471,7 +476,7 @@ class TelegramBaseClient(abc.ABC):
                 # Join the task (wait for it to complete)
                 await task
         """
-        return asyncio.get_event_loop()
+        return helpers.get_running_loop()
 
     @property
     def disconnected(self: 'TelegramClient') -> asyncio.Future:
@@ -524,6 +529,26 @@ class TelegramBaseClient(abc.ABC):
                 except OSError:
                     print('Failed to connect')
         """
+        if self.session is None:
+            raise ValueError('TelegramClient instance cannot be reused after logging out')
+
+        if self._loop is None:
+            self._loop = helpers.get_running_loop()
+        elif self._loop != helpers.get_running_loop():
+            raise RuntimeError('The asyncio event loop must not change after connection (see the FAQ for details)')
+
+        # ':' in session.server_address is True if it's an IPv6 address
+        if (not self.session.server_address or
+                (':' in self.session.server_address) != self._use_ipv6):
+            await utils.maybe_async(
+                self.session.set_dc(
+                    DEFAULT_DC_ID,
+                    DEFAULT_IPV6_IP if self._use_ipv6 else DEFAULT_IPV4_IP,
+                    DEFAULT_PORT
+                )
+            )
+            await utils.maybe_async(self.session.save())
+
         if not await self._sender.connect(self._connection(
             self.session.server_address,
             self.session.port,
@@ -536,13 +561,23 @@ class TelegramBaseClient(abc.ABC):
             return
 
         self.session.auth_key = self._sender.auth_key
-        self.session.save()
+        await utils.maybe_async(self.session.save())
+
+        try:
+            # See comment when saving entities to understand this hack
+            self_entity = await utils.maybe_async(self.session.get_input_entity(0))
+            self_id = self_entity.access_hash
+            self_user = await utils.maybe_async(self.session.get_input_entity(self_id))
+            self._mb_entity_cache.set_self_user(self_id, None, self_user.access_hash)
+        except ValueError:
+            pass
 
         if self._catch_up:
             ss = SessionState(0, 0, False, 0, 0, 0, 0, None)
             cs = []
 
-            for entity_id, state in self.session.get_update_states():
+            update_states = await utils.maybe_async(self.session.get_update_states())
+            for entity_id, state in update_states:
                 if entity_id == 0:
                     # TODO current session doesn't store self-user info but adding that is breaking on downstream session impls
                     ss = SessionState(0, 0, False, state.pts, state.qts, int(state.date.timestamp()), state.seq, None)
@@ -551,15 +586,21 @@ class TelegramBaseClient(abc.ABC):
 
             self._message_box.load(ss, cs)
             for state in cs:
-                entity = self.session.get_input_entity(state.channel_id)
+                try:
-                if entity:
+                    entity = await utils.maybe_async(self.session.get_input_entity(state.channel_id))
+                except ValueError:
+                    self._log[__name__].warning(
+                        'No access_hash in cache for channel %s, will not catch up', state.channel_id)
+                else:
                     self._mb_entity_cache.put(Entity(EntityType.CHANNEL, entity.channel_id, entity.access_hash))
 
         self._init_request.query = functions.help.GetConfigRequest()
 
-        await self._sender.send(functions.InvokeWithLayerRequest(
+        req = self._init_request
-            LAYER, self._init_request
+        if self._no_updates:
-        ))
+            req = functions.InvokeWithoutUpdatesRequest(req)
+
+        await self._sender.send(functions.InvokeWithLayerRequest(LAYER, req))
 
         if self._message_box.is_empty():
             me = await self.get_me()
@@ -592,6 +633,12 @@ class TelegramBaseClient(abc.ABC):
         coroutine that you should await on your own code; otherwise
         the loop is ran until said coroutine completes.
 
+        Event handlers which are currently running will be cancelled before
+        this function returns (in order to properly clean-up their tasks).
+        In particular, this means that using ``disconnect`` in a handler
+        will cause code after the ``disconnect`` to never run. If this is
+        needed, consider spawning a separate task to do the remaining work.
+
         Example
             .. code-block:: python
 
@@ -599,7 +646,11 @@ class TelegramBaseClient(abc.ABC):
                 await client.disconnect()
         """
         if self.loop.is_running():
-            return self._disconnect_coro()
+            # Disconnect may be called from an event handler, which would
+            # cancel itself during itself and never actually complete the
+            # disconnection. Shield the task to prevent disconnect itself
+            # from being cancelled. See issue #3942 for more details.
+            return asyncio.shield(self.loop.create_task(self._disconnect_coro()))
         else:
             try:
                 self.loop.run_until_complete(self._disconnect_coro())
@@ -640,7 +691,32 @@ class TelegramBaseClient(abc.ABC):
             else:
                 connection._proxy = proxy
 
+    async def _save_states_and_entities(self: 'TelegramClient'):
+        # As a hack to not need to change the session files, save ourselves with ``id=0`` and ``access_hash`` of our ``id``.
+        # This way it is possible to determine our own ID by querying for 0. However, whether we're a bot is not saved.
+        # Piggy-back on an arbitrary TL type with users and chats so the session can understand to read the entities.
+        # It doesn't matter if we put users in the list of chats.
+        if self._mb_entity_cache.self_id:
+            await utils.maybe_async(
+                self.session.process_entities(
+                    types.contacts.ResolvedPeer(None, [types.InputPeerUser(0, self._mb_entity_cache.self_id)], [])
+                )
+            )
+
+        ss, cs = self._message_box.session_state()
+        await utils.maybe_async(self.session.set_update_state(0, types.updates.State(**ss, unread_count=0)))
+        now = datetime.datetime.now()  # any datetime works; channels don't need it
+        for channel_id, pts in cs.items():
+            await utils.maybe_async(
+                self.session.set_update_state(
+                    channel_id, types.updates.State(pts, 0, now, 0, unread_count=0)
+                )
+            )
+
     async def _disconnect_coro(self: 'TelegramClient'):
+        if self.session is None:
+            return  # already logged out and disconnected
+
         await self._disconnect()
 
         # Also clean-up all exported senders because we're done with them
@@ -667,19 +743,9 @@ class TelegramBaseClient(abc.ABC):
             await asyncio.wait(self._event_handler_tasks)
             self._event_handler_tasks.clear()
 
-        entities = self._mb_entity_cache.get_all_entities()
+        await self._save_states_and_entities()
 
-        # Piggy-back on an arbitrary TL type with users and chats so the session can understand to read the entities.
+        await utils.maybe_async(self.session.close())
-        # It doesn't matter if we put users in the list of chats.
-        self.session.process_entities(types.contacts.ResolvedPeer(None, [e._as_input_peer() for e in entities], []))
-
-        ss, cs = self._message_box.session_state()
-        self.session.set_update_state(0, types.updates.State(**ss, unread_count=0))
-        now = datetime.datetime.now()  # any datetime works; channels don't need it
-        for channel_id, pts in cs.items():
-            self.session.set_update_state(channel_id, types.updates.State(pts, 0, now, 0, unread_count=0))
-
-        self.session.close()
 
     async def _disconnect(self: 'TelegramClient'):
         """
@@ -700,22 +766,22 @@ class TelegramBaseClient(abc.ABC):
         self._log[__name__].info('Reconnecting to new data center %s', new_dc)
         dc = await self._get_dc(new_dc)
 
-        self.session.set_dc(dc.id, dc.ip_address, dc.port)
+        await utils.maybe_async(self.session.set_dc(dc.id, dc.ip_address, dc.port))
         # auth_key's are associated with a server, which has now changed
         # so it's not valid anymore. Set to None to force recreating it.
         self._sender.auth_key.key = None
         self.session.auth_key = None
-        self.session.save()
+        await utils.maybe_async(self.session.save())
         await self._disconnect()
         return await self.connect()
 
-    def _auth_key_callback(self: 'TelegramClient', auth_key):
+    async def _auth_key_callback(self: 'TelegramClient', auth_key):
         """
         Callback from the sender whenever it needed to generate a
         new authorization key. This means we are not authorized.
         """
         self.session.auth_key = auth_key
-        self.session.save()
+        await utils.maybe_async(self.session.save())
 
     # endregion
 
@@ -730,7 +796,8 @@ class TelegramBaseClient(abc.ABC):
         if cdn and not self._cdn_config:
             cls._cdn_config = await self(functions.help.GetCdnConfigRequest())
             for pk in cls._cdn_config.public_keys:
-                rsa.add_key(pk.public_key)
+                if pk.dc_id == dc_id:
+                    rsa.add_key(pk.public_key, old=False)
 
         try:
             return next(
@@ -743,10 +810,13 @@ class TelegramBaseClient(abc.ABC):
                 'Failed to get DC %s (cdn = %s) with use_ipv6 = %s; retrying ignoring IPv6 check',
                 dc_id, cdn, self._use_ipv6
             )
+            try:
                 return next(
                     dc for dc in cls._config.dc_options
                     if dc.id == dc_id and bool(dc.cdn) == cdn
                 )
+            except StopIteration:
+                raise ValueError(f'Failed to get DC {dc_id} (cdn = {cdn})')
 
     async def _create_exported_sender(self: 'TelegramClient', dc_id):
         """
@@ -834,28 +904,30 @@ class TelegramBaseClient(abc.ABC):
 
     async def _get_cdn_client(self: 'TelegramClient', cdn_redirect):
         """Similar to ._borrow_exported_client, but for CDNs"""
-        # TODO Implement
-        raise NotImplementedError
         session = self._exported_sessions.get(cdn_redirect.dc_id)
         if not session:
             dc = await self._get_dc(cdn_redirect.dc_id, cdn=True)
-            session = self.session.clone()
+            session = await utils.maybe_async(self.session.clone())
-            session.set_dc(dc.id, dc.ip_address, dc.port)
+            await utils.maybe_async(session.set_dc(dc.id, dc.ip_address, dc.port))
             self._exported_sessions[cdn_redirect.dc_id] = session
 
         self._log[__name__].info('Creating new CDN client')
-        client = TelegramBaseClient(
+        client = self.__class__(
             session, self.api_id, self.api_hash,
-            proxy=self._sender.connection.conn.proxy,
+            proxy=self._proxy,
-            timeout=self._sender.connection.get_timeout()
+            timeout=self._timeout,
+            loop=self.loop
         )
 
-        # This will make use of the new RSA keys for this specific CDN.
+        session.auth_key = self._sender.auth_key
-        #
+        await client._sender.connect(self._connection(
-        # We won't be calling GetConfigRequest because it's only called
+            session.server_address,
-        # when needed by ._get_dc, and also it's static so it's likely
+            session.port,
-        # set already. Avoid invoking non-CDN methods by not syncing updates.
+            session.dc_id,
-        client.connect(_sync_updates=False)
+            loggers=self._log,
+            proxy=self._proxy,
+            local_addr=self._local_addr
+        ))
         return client
 
     # endregion

telethon/client/updates.py

@@ -7,12 +7,17 @@ import time
 import traceback
 import typing
 import logging
+import warnings
 from collections import deque
+import sqlite3
 
 from .. import events, utils, errors
 from ..events.common import EventBuilder, EventCommon
 from ..tl import types, functions
 from .._updates import GapError, PrematureEndReason
+from ..helpers import get_running_loop
+from ..version import __version__
+
 
 if typing.TYPE_CHECKING:
     from .telegramclient import TelegramClient
@@ -251,6 +256,11 @@ class UpdateMethods:
     # region Private methods
 
     async def _update_loop(self: 'TelegramClient'):
+        # If the MessageBox is not empty, the account had to be logged-in to fill in its state.
+        # This flag is used to propagate the "you got logged-out" error up (but getting logged-out
+        # can only happen if it was once logged-in).
+        was_once_logged_in = self._authorized is True or not self._message_box.is_empty()
+
         self._updates_error = None
         try:
             if self._catch_up:
@@ -269,16 +279,39 @@ class UpdateMethods:
                             # TODO if _dispatch_update fails for whatever reason, it's not logged! this should be fixed
                             task = self.loop.create_task(self._dispatch_update(updates_to_dispatch.popleft()))
                             self._event_handler_tasks.add(task)
-                            task.add_done_callback(lambda _: self._event_handler_tasks.discard(task))
+                            task.add_done_callback(self._event_handler_tasks.discard)
 
                     continue
 
+                if len(self._mb_entity_cache) >= self._entity_cache_limit:
+                    self._log[__name__].info(
+                        'In-memory entity cache limit reached (%s/%s), flushing to session',
+                        len(self._mb_entity_cache),
+                        self._entity_cache_limit
+                    )
+                    await self._save_states_and_entities()
+                    self._mb_entity_cache.retain(lambda id: id == self._mb_entity_cache.self_id or id in self._message_box.map)
+                    if len(self._mb_entity_cache) >= self._entity_cache_limit:
+                        warnings.warn('in-memory entities exceed entity_cache_limit after flushing; consider setting a larger limit')
+
+                    self._log[__name__].info(
+                        'In-memory entity cache at %s/%s after flushing to session',
+                        len(self._mb_entity_cache),
+                        self._entity_cache_limit
+                    )
+
+
                 get_diff = self._message_box.get_difference()
                 if get_diff:
-                    self._log[__name__].info('Getting difference for account updates')
+                    self._log[__name__].debug('Getting difference for account updates')
                     try:
                         diff = await self(get_diff)
-                    except (errors.ServerError, ValueError) as e:
+                    except (
+                        errors.ServerError,
+                        errors.TimedOutError,
+                        errors.FloodWaitError,
+                        ValueError
+                    ) as e:
                         # Telegram is having issues
                         self._log[__name__].info('Cannot get difference since Telegram is having issues: %s', type(e).__name__)
                         self._message_box.end_difference()
@@ -287,22 +320,73 @@ class UpdateMethods:
                         # Not logged in or broken authorization key, can't get difference
                         self._log[__name__].info('Cannot get difference since the account is not logged in: %s', type(e).__name__)
                         self._message_box.end_difference()
+                        if was_once_logged_in:
+                            self._updates_error = e
+                            await self.disconnect()
+                            break
+                        continue
+                    except (errors.TypeNotFoundError, sqlite3.OperationalError) as e:
+                        # User is likely doing weird things with their account or session and Telegram gets confused as to what layer they use
+                        self._log[__name__].warning('Cannot get difference since the account is likely misusing the session: %s', e)
+                        self._message_box.end_difference()
+                        self._updates_error = e
+                        await self.disconnect()
+                        break
+                    except OSError as e:
+                        # Network is likely down, but it's unclear for how long.
+                        # If disconnect is called this task will be cancelled along with the sleep.
+                        # If disconnect is not called, getting difference should be retried after a few seconds.
+                        self._log[__name__].info('Cannot get difference since the network is down: %s: %s', type(e).__name__, e)
+                        await asyncio.sleep(5)
                         continue
                     updates, users, chats = self._message_box.apply_difference(diff, self._mb_entity_cache)
-                    updates_to_dispatch.extend(self._preprocess_updates(updates, users, chats))
+                    if updates:
+                        self._log[__name__].info('Got difference for account updates')
+
+                    _preprocess_updates = await utils.maybe_async(self._preprocess_updates(updates, users, chats))
+                    updates_to_dispatch.extend(_preprocess_updates)
                     continue
 
                 get_diff = self._message_box.get_channel_difference(self._mb_entity_cache)
                 if get_diff:
-                    self._log[__name__].info('Getting difference for channel updates')
+                    self._log[__name__].debug('Getting difference for channel %s updates', get_diff.channel.channel_id)
                     try:
                         diff = await self(get_diff)
+                    except (errors.UnauthorizedError, errors.AuthKeyError) as e:
+                        # Not logged in or broken authorization key, can't get difference
+                        self._log[__name__].warning(
+                            'Cannot get difference for channel %s since the account is not logged in: %s',
+                            get_diff.channel.channel_id, type(e).__name__
+                        )
+                        self._message_box.end_channel_difference(
+                            get_diff,
+                            PrematureEndReason.TEMPORARY_SERVER_ISSUES,
+                            self._mb_entity_cache
+                        )
+                        if was_once_logged_in:
+                            self._updates_error = e
+                            await self.disconnect()
+                            break
+                        continue
+                    except (errors.TypeNotFoundError, sqlite3.OperationalError) as e:
+                        self._log[__name__].warning(
+                            'Cannot get difference for channel %s since the account is likely misusing the session: %s',
+                            get_diff.channel.channel_id, e
+                        )
+                        self._message_box.end_channel_difference(
+                            get_diff,
+                            PrematureEndReason.TEMPORARY_SERVER_ISSUES,
+                            self._mb_entity_cache
+                        )
+                        self._updates_error = e
+                        await self.disconnect()
+                        break
                     except (
                         errors.PersistentTimestampOutdatedError,
                         errors.PersistentTimestampInvalidError,
                         errors.ServerError,
-                        errors.UnauthorizedError,
+                        errors.TimedOutError,
-                        errors.AuthKeyError,
+                        errors.FloodWaitError,
                         ValueError
                     ) as e:
                         # According to Telegram's docs:
@@ -322,9 +406,9 @@ class UpdateMethods:
                         # We treat this as PersistentTimestampOutdatedError for now.
                         # TODO investigate why/when this happens and if this is the proper solution
                         self._log[__name__].warning(
-                            'Getting difference for channel updates caused %s;'
+                            'Getting difference for channel updates %s caused %s;'
                             ' ending getting difference prematurely until server issues are resolved',
-                            type(e).__name__
+                            get_diff.channel.channel_id, type(e).__name__
                         )
                         self._message_box.end_channel_difference(
                             get_diff,
@@ -346,19 +430,30 @@ class UpdateMethods:
                             self._mb_entity_cache
                         )
                         continue
+                    except OSError as e:
+                        self._log[__name__].info(
+                            'Cannot get difference for channel %d since the network is down: %s: %s',
+                            get_diff.channel.channel_id, type(e).__name__, e
+                        )
+                        await asyncio.sleep(5)
+                        continue
 
                     updates, users, chats = self._message_box.apply_channel_difference(get_diff, diff, self._mb_entity_cache)
-                    updates_to_dispatch.extend(self._preprocess_updates(updates, users, chats))
+                    if updates:
+                        self._log[__name__].info('Got difference for channel %d updates', get_diff.channel.channel_id)
+
+                    _preprocess_updates = await utils.maybe_async(self._preprocess_updates(updates, users, chats))
+                    updates_to_dispatch.extend(_preprocess_updates)
                     continue
 
                 deadline = self._message_box.check_deadlines()
-                deadline_delay = deadline - asyncio.get_running_loop().time()
+                deadline_delay = deadline - get_running_loop().time()
                 if deadline_delay > 0:
                     # Don't bother sleeping and timing out if the delay is already 0 (pollutes the logs).
                     try:
                         updates = await asyncio.wait_for(self._updates_queue.get(), deadline_delay)
                     except asyncio.TimeoutError:
-                        self._log[__name__].info('Timeout waiting for updates expired')
+                        self._log[__name__].debug('Timeout waiting for updates expired')
                         continue
                 else:
                     continue
@@ -369,16 +464,18 @@ class UpdateMethods:
                 except GapError:
                     continue  # get(_channel)_difference will start returning requests
 
-                updates_to_dispatch.extend(self._preprocess_updates(processed, users, chats))
+                _preprocess_updates = await utils.maybe_async(self._preprocess_updates(processed, users, chats))
+                updates_to_dispatch.extend(_preprocess_updates)
         except asyncio.CancelledError:
             pass
         except Exception as e:
-            self._log[__name__].exception('Fatal error handling updates (this is a bug in Telethon, please report it)')
+            self._log[__name__].exception(f'Fatal error handling updates (this is a bug in Telethon v{__version__}, please report it)')
             self._updates_error = e
             await self.disconnect()
 
-    def _preprocess_updates(self, updates, users, chats):
+    async def _preprocess_updates(self, updates, users, chats):
         self._mb_entity_cache.extend(users, chats)
+        await utils.maybe_async(self.session.process_entities(types.contacts.ResolvedPeer(None, users, chats)))
         entities = {utils.get_peer_id(x): x
                     for x in itertools.chain(users, chats)}
         for u in updates:
@@ -421,16 +518,18 @@ class UpdateMethods:
             # inserted because this is a rather expensive operation
             # (default's sqlite3 takes ~0.1s to commit changes). Do
             # it every minute instead. No-op if there's nothing new.
-            self.session.save()
+            await self._save_states_and_entities()
+
+            await utils.maybe_async(self.session.save())
 
     async def _dispatch_update(self: 'TelegramClient', update):
         # TODO only used for AlbumHack, and MessageBox is not really designed for this
         others = None
 
-        if not self._self_input_peer:
+        if not self._mb_entity_cache.self_id:
             # Some updates require our own ID, so we must make sure
             # that the event builder has offline access to it. Calling
-            # `get_me()` will cache it under `self._self_input_peer`.
+            # `get_me()` will cache it under `self._mb_entity_cache`.
             #
             # It will return `None` if we haven't logged in yet which is
             # fine, we will just retry next time anyway.

telethon/client/uploads.py

@@ -18,7 +18,6 @@ try:
 except ImportError:
     PIL = None
 
-
 if typing.TYPE_CHECKING:
     from .telegramclient import TelegramClient
 
@@ -36,7 +35,7 @@ class _CacheType:
 
 
 def _resize_photo_if_needed(
-        file, is_image, width=1280, height=1280, background=(255, 255, 255)):
+        file, is_image, width=2560, height=2560, background=(255, 255, 255)):
 
     # https://github.com/telegramdesktop/tdesktop/blob/12905f0dcb9d513378e7db11989455a1b764ef75/Telegram/SourceFiles/boxes/photo_crop_box.cpp#L254
     if (not is_image
@@ -47,7 +46,17 @@ def _resize_photo_if_needed(
     if isinstance(file, bytes):
         file = io.BytesIO(file)
 
-    before = file.tell() if isinstance(file, io.IOBase) else None
+    if isinstance(file, io.IOBase):
+        # Pillow seeks to 0 unconditionally later anyway
+        old_pos = file.tell()
+        file.seek(0, io.SEEK_END)
+        before = file.tell()
+    elif isinstance(file, str) and os.path.exists(file):
+        # Check if file exists as a path and if so, get its size on disk
+        before = os.path.getsize(file)
+    else:
+        # Would be weird...
+        before = None
 
     try:
         # Don't use a `with` block for `image`, or `file` would be closed.
@@ -58,34 +67,43 @@ def _resize_photo_if_needed(
         except KeyError:
             kwargs = {}
 
-        if image.width <= width and image.height <= height:
+        if image.mode == 'RGB':
+            # Check if image is within acceptable bounds, if so, check if the image is at or below 10 MB, or assume it isn't if size is None or 0
+            if image.width <= width and image.height <= height and (before <= 10000000 if before else False):
                 return file
 
-        image.thumbnail((width, height), PIL.Image.ANTIALIAS)
+            # If the image is already RGB, don't convert it
-
+            # certain modes such as 'P' have no alpha index but can't be saved as JPEG directly
-        alpha_index = image.mode.find('A')
+            image.thumbnail((width, height), PIL.Image.LANCZOS)
-        if alpha_index == -1:
-            # If the image mode doesn't have alpha
-            # channel then don't bother masking it away.
             result = image
         else:
             # We could save the resized image with the original format, but
             # JPEG often compresses better -> smaller size -> faster upload
             # We need to mask away the alpha channel ([3]), since otherwise
             # IOError is raised when trying to save alpha channels in JPEG.
+            image.thumbnail((width, height), PIL.Image.LANCZOS)
             result = PIL.Image.new('RGB', image.size, background)
-            result.paste(image, mask=image.split()[alpha_index])
+            mask = None
+
+            if image.has_transparency_data:
+                if image.mode == 'RGBA':
+                    mask = image.getchannel('A')
+                else:
+                    mask = image.convert('RGBA').getchannel('A')
+
+            result.paste(image, mask=mask)
 
         buffer = io.BytesIO()
-        result.save(buffer, 'JPEG', **kwargs)
+        result.save(buffer, 'JPEG', progressive=True, **kwargs)
         buffer.seek(0)
+        buffer.name = 'a.jpg'
         return buffer
-
     except IOError:
         return file
     finally:
-        if before is not None:
+        # The original position might matter
-            file.seek(before, io.SEEK_SET)
+        if isinstance(file, io.IOBase):
+            file.seek(old_pos)
 
 
 class UploadMethods:
@@ -99,6 +117,7 @@ class UploadMethods:
             *,
             caption: typing.Union[str, typing.Sequence[str]] = None,
             force_document: bool = False,
+            mime_type: str = None,
             file_size: int = None,
             clear_draft: bool = False,
             progress_callback: 'hints.ProgressCallback' = None,
@@ -107,7 +126,11 @@ class UploadMethods:
             thumb: 'hints.FileLike' = None,
             allow_cache: bool = True,
             parse_mode: str = (),
-            formatting_entities: typing.Optional[typing.List[types.TypeMessageEntity]] = None,
+            formatting_entities: typing.Optional[
+                typing.Union[
+                    typing.List[types.TypeMessageEntity], typing.List[typing.List[types.TypeMessageEntity]]
+                ]
+            ] = None,
             voice_note: bool = False,
             video_note: bool = False,
             buttons: typing.Optional['hints.MarkupLike'] = None,
@@ -117,7 +140,10 @@ class UploadMethods:
             schedule: 'hints.DateLike' = None,
             comment_to: 'typing.Union[int, types.Message]' = None,
             ttl: int = None,
-            **kwargs) -> 'types.Message':
+            nosound_video: bool = None,
+            send_as: typing.Optional['hints.EntityLike'] = None,
+            message_effect_id: typing.Optional[int] = None,
+            **kwargs) -> typing.Union[typing.List[typing.Any], typing.Any]:
         """
         Sends message with the given file to the specified entity.
 
@@ -184,6 +210,13 @@ class UploadMethods:
                 the extension of an image file or a video file, it will be
                 sent as such. Otherwise always as a document.
 
+            mime_type (`str`, optional):
+                Custom mime type to use for the file to be sent (for example,
+                ``audio/mpeg``, ``audio/x-vorbis+ogg``, etc.).
+                It can change the type of files displayed.
+                If not set to any value, the mime type will be determined
+                automatically based on the file's extension.
+
             file_size (`int`, optional):
                 The size of the file to be uploaded if it needs to be uploaded,
                 which will be determined automatically if not specified.
@@ -230,7 +263,11 @@ class UploadMethods:
                 default.
 
             formatting_entities (`list`, optional):
-                A list of message formatting entities. When provided, the ``parse_mode`` is ignored.
+                Optional formatting entities for the sent media message. When sending an album,
+                `formatting_entities` can be a list of lists, where each inner list contains
+                `types.TypeMessageEntity`. Each inner list will be assigned to the corresponding
+                file in a pairwise manner with the caption. If provided, the ``parse_mode``
+                parameter will be ignored.
 
             voice_note (`bool`, optional):
                 If `True` the audio will be sent as a voice note.
@@ -286,6 +323,25 @@ class UploadMethods:
                 Not all types of media can be used with this parameter, such
                 as text documents, which will fail with ``TtlMediaInvalidError``.
 
+            nosound_video (`bool`, optional):
+                Only applicable when sending a video file without an audio
+                track. If set to ``True``, the video will be displayed in
+                Telegram as a video. If set to ``False``, Telegram will attempt
+                to display the video as an animated gif. (It may still display
+                as a video due to other factors.) The value is ignored if set
+                on non-video files. This is set to ``True`` for albums, as gifs
+                cannot be sent in albums.
+
+            send_as (`entity`):
+                Unique identifier (int) or username (str) of the chat or channel to send the message as.
+                You can use this to send the message on behalf of a chat or channel where you have appropriate permissions.
+                Use the GetSendAs to return the list of message sender identifiers, which can be used to send messages in the chat,
+                This setting applies to the current message and will remain effective for future messages unless explicitly changed.
+                To set this behavior permanently for all messages, use SaveDefaultSendAs.
+
+            message_effect_id (`int`, optional):
+                Unique identifier of the message effect to be added to the message; for private chats only
+
         Returns
             The `Message <telethon.tl.custom.message.Message>` (or messages)
             containing the sent file, or messages if a list of them was passed.
@@ -343,6 +399,9 @@ class UploadMethods:
         if not caption:
             caption = ''
 
+        if not formatting_entities:
+            formatting_entities = []
+
         entity = await self.get_input_entity(entity)
         if comment_to is not None:
             entity, reply_to = await self._get_comment_data(entity, comment_to)
@@ -352,38 +411,46 @@ class UploadMethods:
         # First check if the user passed an iterable, in which case
         # we may want to send grouped.
         if utils.is_list_like(file):
+            sent_count = 0
+            used_callback = None if not progress_callback else (
+                lambda s, t: progress_callback(sent_count + s, len(file))
+            )
+
             if utils.is_list_like(caption):
                 captions = caption
             else:
                 captions = [caption]
 
+            # Check that formatting_entities list is valid
+            if all(utils.is_list_like(obj) for obj in formatting_entities):
+                formatting_entities = formatting_entities
+            elif utils.is_list_like(formatting_entities):
+                formatting_entities = [formatting_entities]
+            else:
+                raise TypeError('The formatting_entities argument must be a list or a sequence of lists')
+
+            # Check that all entities in all lists are of the correct type
+            if not all(isinstance(ent, types.TypeMessageEntity) for sublist in formatting_entities for ent in sublist):
+                raise TypeError('All entities must be instances of <types.TypeMessageEntity>')
+
             result = []
             while file:
                 result += await self._send_album(
-                    entity, file[:10], caption=captions[:10],
+                    entity, file[:10], caption=captions[:10], formatting_entities=formatting_entities[:10],
-                    progress_callback=progress_callback, reply_to=reply_to,
+                    progress_callback=used_callback, reply_to=reply_to,
                     parse_mode=parse_mode, silent=silent, schedule=schedule,
                     supports_streaming=supports_streaming, clear_draft=clear_draft,
                     force_document=force_document, background=background,
+                    send_as=send_as, message_effect_id=message_effect_id
                 )
                 file = file[10:]
                 captions = captions[10:]
-
+                formatting_entities = formatting_entities[10:]
-            for doc, cap in zip(file, captions):
+                sent_count += 10
-                result.append(await self.send_file(
-                    entity, doc, allow_cache=allow_cache,
-                    caption=cap, force_document=force_document,
-                    progress_callback=progress_callback, reply_to=reply_to,
-                    attributes=attributes, thumb=thumb, voice_note=voice_note,
-                    video_note=video_note, buttons=buttons, silent=silent,
-                    supports_streaming=supports_streaming, schedule=schedule,
-                    clear_draft=clear_draft, background=background,
-                    **kwargs
-                ))
 
             return result
 
-        if formatting_entities is not None:
+        if formatting_entities:
             msg_entities = formatting_entities
         else:
             caption, msg_entities =\
@@ -391,11 +458,13 @@ class UploadMethods:
 
         file_handle, media, image = await self._file_to_media(
             file, force_document=force_document,
+            mime_type=mime_type,
             file_size=file_size,
             progress_callback=progress_callback,
             attributes=attributes, allow_cache=allow_cache, thumb=thumb,
             voice_note=voice_note, video_note=video_note,
-            supports_streaming=supports_streaming, ttl=ttl
+            supports_streaming=supports_streaming, ttl=ttl,
+            nosound_video=nosound_video,
         )
 
         # e.g. invalid cast from :tl:`MessageMediaWebPage`
@@ -403,19 +472,25 @@ class UploadMethods:
             raise TypeError('Cannot use {!r} as file'.format(file))
 
         markup = self.build_reply_markup(buttons)
+        reply_to = None if reply_to is None else types.InputReplyToMessage(reply_to)
         request = functions.messages.SendMediaRequest(
-            entity, media, reply_to_msg_id=reply_to, message=caption,
+            entity, media, reply_to=reply_to, message=caption,
             entities=msg_entities, reply_markup=markup, silent=silent,
             schedule_date=schedule, clear_draft=clear_draft,
-            background=background
+            background=background,
+            send_as=await self.get_input_entity(send_as) if send_as else None,
+            effect=message_effect_id
         )
         return self._get_response_message(request, await self(request), entity)
 
     async def _send_album(self: 'TelegramClient', entity, files, caption='',
+                          formatting_entities=None,
                           progress_callback=None, reply_to=None,
                           parse_mode=(), silent=None, schedule=None,
                           supports_streaming=None, clear_draft=None,
-                          force_document=False, background=None, ttl=None):
+                          force_document=False, background=None, ttl=None,
+                          send_as: typing.Optional['hints.EntityLike'] = None,
+                          message_effect_id: typing.Optional[int] = None):
         """Specialized version of .send_file for albums"""
         # We don't care if the user wants to avoid cache, we will use it
         # anyway. Why? The cached version will be exactly the same thing
@@ -423,36 +498,51 @@ class UploadMethods:
         # cache only makes a difference for documents where the user may
         # want the attributes used on them to change.
         #
-        # In theory documents can be sent inside the albums but they appear
+        # In theory documents can be sent inside the albums, but they appear
         # as different messages (not inside the album), and the logic to set
         # the attributes/avoid cache is already written in .send_file().
         entity = await self.get_input_entity(entity)
         if not utils.is_list_like(caption):
             caption = (caption,)
+        if not all(isinstance(obj, list) for obj in formatting_entities):
+            formatting_entities = (formatting_entities,)
 
         captions = []
+        # If the formatting_entities argument is provided, we don't use parse_mode
+        if formatting_entities:
+            # Pop from the end (so reverse)
+            capt_with_ent = itertools.zip_longest(reversed(caption), reversed(formatting_entities), fillvalue=None)
+            for msg_caption, msg_entities in capt_with_ent:
+                captions.append((msg_caption, msg_entities))
+        else:
             for c in reversed(caption):  # Pop from the end (so reverse)
                 captions.append(await self._parse_message_text(c or '', parse_mode))
 
         reply_to = utils.get_message_id(reply_to)
 
+        used_callback = None if not progress_callback else (
+            # use an integer when sent matches total, to easily determine a file has been fully sent
+            lambda s, t: progress_callback(sent_count + 1 if s == t else sent_count + s / t, len(files))
+        )
+
         # Need to upload the media first, but only if they're not cached yet
         media = []
-        for file in files:
+        for sent_count, file in enumerate(files):
             # Albums want :tl:`InputMedia` which, in theory, includes
-            # :tl:`InputMediaUploadedPhoto`. However using that will
+            # :tl:`InputMediaUploadedPhoto`. However, using that will
             # make it `raise MediaInvalidError`, so we need to upload
             # it as media and then convert that to :tl:`InputMediaPhoto`.
             fh, fm, _ = await self._file_to_media(
                 file, supports_streaming=supports_streaming,
-                force_document=force_document, ttl=ttl)
+                force_document=force_document, ttl=ttl,
+                progress_callback=used_callback, nosound_video=True)
             if isinstance(fm, (types.InputMediaUploadedPhoto, types.InputMediaPhotoExternal)):
                 r = await self(functions.messages.UploadMediaRequest(
                     entity, media=fm
                 ))
 
                 fm = utils.get_input_media(r.photo)
-            elif isinstance(fm, types.InputMediaUploadedDocument):
+            elif isinstance(fm, (types.InputMediaUploadedDocument, types.InputMediaDocumentExternal)):
                 r = await self(functions.messages.UploadMediaRequest(
                     entity, media=fm
                 ))
@@ -473,9 +563,11 @@ class UploadMethods:
 
         # Now we can construct the multi-media request
         request = functions.messages.SendMultiMediaRequest(
-            entity, reply_to_msg_id=reply_to, multi_media=media,
+            entity, reply_to=None if reply_to is None else types.InputReplyToMessage(reply_to), multi_media=media,
             silent=silent, schedule_date=schedule, clear_draft=clear_draft,
-            background=background
+            background=background,
+            send_as=await self.get_input_entity(send_as) if send_as else None,
+            effect=message_effect_id
         )
         result = await self(request)
 
@@ -546,6 +638,13 @@ class UploadMethods:
                 A callback function accepting two parameters:
                 ``(sent bytes, total)``.
 
+                When sending an album, the callback will receive a number
+                between 0 and the amount of files as the "sent" parameter,
+                and the amount of files as the "total". Note that the first
+                parameter will be a floating point number to indicate progress
+                within a file (e.g. ``2.5`` means it has sent 50% of the third
+                file, because it's between 2 and 3).
+
         Returns
             :tl:`InputFileBig` if the file size is larger than 10MB,
             `InputSizedFile <telethon.tl.custom.inputsizedfile.InputSizedFile>`
@@ -668,7 +767,7 @@ class UploadMethods:
             progress_callback=None, attributes=None, thumb=None,
             allow_cache=True, voice_note=False, video_note=False,
             supports_streaming=False, mime_type=None, as_image=None,
-            ttl=None):
+            ttl=None, nosound_video=None):
         if not file:
             return None, None, None
 
@@ -753,13 +852,18 @@ class UploadMethods:
                     thumb = str(thumb.absolute())
                 thumb = await self.upload_file(thumb, file_size=file_size)
 
+            # setting `nosound_video` to `True` doesn't affect videos with sound
+            # instead it prevents sending silent videos as GIFs
+            nosound_video = nosound_video if mime_type.split("/")[0] == 'video' else None
+
             media = types.InputMediaUploadedDocument(
                 file=file_handle,
                 mime_type=mime_type,
                 attributes=attributes,
                 thumb=thumb,
                 force_file=force_document and not is_image,
-                ttl_seconds=ttl
+                ttl_seconds=ttl,
+                nosound_video=nosound_video
             )
         return file_handle, media, as_image
 

telethon/client/users.py

@@ -30,10 +30,15 @@ class UserMethods:
         return await self._call(self._sender, request, ordered=ordered)
 
     async def _call(self: 'TelegramClient', sender, request, ordered=False, flood_sleep_threshold=None):
+        if self._loop is not None and self._loop != helpers.get_running_loop():
+            raise RuntimeError('The asyncio event loop must not change after connection (see the FAQ for details)')
+        # if the loop is None it will fail with a connection error later on
+
         if flood_sleep_threshold is None:
             flood_sleep_threshold = self.flood_sleep_threshold
-        requests = (request if utils.is_list_like(request) else (request,))
+        requests = list(request) if utils.is_list_like(request) else [request]
-        for r in requests:
+        request = list(request) if utils.is_list_like(request) else request
+        for i, r in enumerate(requests):
             if not isinstance(r, TLRequest):
                 raise _NOT_A_REQUEST()
             await r.resolve(self, utils)
@@ -52,7 +57,11 @@ class UserMethods:
                     raise errors.FloodWaitError(request=r, capture=diff)
 
             if self._no_updates:
-                r = functions.InvokeWithoutUpdatesRequest(r)
+                if utils.is_list_like(request):
+                    request[i] = functions.InvokeWithoutUpdatesRequest(r)
+                else:
+                    # This should only run once as requests should be a list of 1 item
+                    request = functions.InvokeWithoutUpdatesRequest(r)
 
         request_index = 0
         last_error = None
@@ -71,8 +80,7 @@ class UserMethods:
                             exceptions.append(e)
                             results.append(None)
                             continue
-                        self.session.process_entities(result)
+                        await utils.maybe_async(self.session.process_entities(result))
-                        self._entity_cache.add(result)
                         exceptions.append(None)
                         results.append(result)
                         request_index += 1
@@ -82,11 +90,11 @@ class UserMethods:
                         return results
                 else:
                     result = await future
-                    self.session.process_entities(result)
+                    await utils.maybe_async(self.session.process_entities(result))
-                    self._entity_cache.add(result)
                     return result
             except (errors.ServerError, errors.RpcCallFailError,
                     errors.RpcMcgetFailError, errors.InterdcCallErrorError,
+                    errors.TimedOutError,
                     errors.InterdcCallRichErrorError) as e:
                 last_error = e
                 self._log[__name__].warning(
@@ -94,7 +102,8 @@ class UserMethods:
                     e.__class__.__name__, e)
 
                 await asyncio.sleep(2)
-            except (errors.FloodWaitError, errors.SlowModeWaitError, errors.FloodTestPhoneWaitError) as e:
+            except (errors.FloodWaitError, errors.FloodPremiumWaitError,
+                    errors.SlowModeWaitError, errors.FloodTestPhoneWaitError) as e:
                 last_error = e
                 if utils.is_list_like(request):
                     request = request[request_index]
@@ -154,20 +163,17 @@ class UserMethods:
                 me = await client.get_me()
                 print(me.username)
         """
-        if input_peer and self._self_input_peer:
+        if input_peer and self._mb_entity_cache.self_id:
-            return self._self_input_peer
+            return self._mb_entity_cache.get(self._mb_entity_cache.self_id)._as_input_peer()
 
         try:
             me = (await self(
                 functions.users.GetUsersRequest([types.InputUserSelf()])))[0]
 
-            self._bot = me.bot
+            if not self._mb_entity_cache.self_id:
-            if not self._self_input_peer:
+                self._mb_entity_cache.set_self_user(me.id, me.bot, me.access_hash)
-                self._self_input_peer = utils.get_input_peer(
-                    me, allow_self=False
-                )
 
-            return self._self_input_peer if input_peer else me
+            return utils.get_input_peer(me, allow_self=False) if input_peer else me
         except errors.UnauthorizedError:
             return None
 
@@ -179,7 +185,7 @@ class UserMethods:
         This property is used in every update, and some like `updateLoginToken`
         occur prior to login, so it gracefully handles when no ID is known yet.
         """
-        return self._self_input_peer.user_id if self._self_input_peer else None
+        return self._mb_entity_cache.self_id
 
     async def is_bot(self: 'TelegramClient') -> bool:
         """
@@ -193,10 +199,10 @@ class UserMethods:
                 else:
                     print('Hello')
         """
-        if self._bot is None:
+        if self._mb_entity_cache.self_bot is None:
-            self._bot = (await self.get_me()).bot
+            await self.get_me(input_peer=True)
 
-        return self._bot
+        return self._mb_entity_cache.self_bot
 
     async def is_user_authorized(self: 'TelegramClient') -> bool:
         """
@@ -222,7 +228,7 @@ class UserMethods:
 
     async def get_entity(
             self: 'TelegramClient',
-            entity: 'hints.EntitiesLike') -> 'hints.Entity':
+            entity: 'hints.EntitiesLike') -> typing.Union['hints.Entity', typing.List['hints.Entity']]:
         """
         Turns the given entity into a valid Telegram :tl:`User`, :tl:`Chat`
         or :tl:`Channel`. You can also pass a list or iterable of entities,
@@ -321,7 +327,9 @@ class UserMethods:
 
         # Merge users, chats and channels into a single dictionary
         id_entity = {
-            utils.get_peer_id(x): x
+            # `get_input_entity` might've guessed the type from a non-marked ID,
+            # so the only way to match that with the input is by not using marks here.
+            utils.get_peer_id(x, add_mark=False): x
             for x in itertools.chain(users, chats, channels)
         }
 
@@ -334,7 +342,7 @@ class UserMethods:
             if isinstance(x, str):
                 result.append(await self._get_entity_from_string(x))
             elif not isinstance(x, types.InputPeerSelf):
-                result.append(id_entity[utils.get_peer_id(x)])
+                result.append(id_entity[utils.get_peer_id(x, add_mark=False)])
             else:
                 result.append(next(
                     u for u in id_entity.values()
@@ -417,8 +425,8 @@ class UserMethods:
         try:
             # 0x2d45687 == crc32(b'Peer')
             if isinstance(peer, int) or peer.SUBCLASS_OF_ID == 0x2d45687:
-                return self._entity_cache[peer]
+                return self._mb_entity_cache.get(utils.get_peer_id(peer, add_mark=False))._as_input_peer()
-        except (AttributeError, KeyError):
+        except AttributeError:
             pass
 
         # Then come known strings that take precedence
@@ -427,7 +435,8 @@ class UserMethods:
 
         # No InputPeer, cached peer, or known string. Fetch from disk cache
         try:
-            return self.session.get_input_entity(peer)
+            input_entity = await utils.maybe_async(self.session.get_input_entity(peer))
+            return input_entity
         except ValueError:
             pass
 
@@ -566,8 +575,8 @@ class UserMethods:
                     pass
             try:
                 # Nobody with this username, maybe it's an exact name/title
-                return await self.get_entity(
+                input_entity = await utils.maybe_async(self.session.get_input_entity(string))
-                    self.session.get_input_entity(string))
+                return await self.get_entity(input_entity)
             except ValueError:
                 pass
 

telethon/entitycache.py

@@ -1,147 +0,0 @@
-import inspect
-import itertools
-
-from . import utils
-from .tl import types
-
-# Which updates have the following fields?
-_has_field = {
-    ('user_id', int): [],
-    ('chat_id', int): [],
-    ('channel_id', int): [],
-    ('peer', 'TypePeer'): [],
-    ('peer', 'TypeDialogPeer'): [],
-    ('message', 'TypeMessage'): [],
-}
-
-# Note: We don't bother checking for some rare:
-# * `UpdateChatParticipantAdd.inviter_id` integer.
-# * `UpdateNotifySettings.peer` dialog peer.
-# * `UpdatePinnedDialogs.order` list of dialog peers.
-# * `UpdateReadMessagesContents.messages` list of messages.
-# * `UpdateChatParticipants.participants` list of participants.
-#
-# There are also some uninteresting `update.message` of type string.
-
-
-def _fill():
-    for name in dir(types):
-        update = getattr(types, name)
-        if getattr(update, 'SUBCLASS_OF_ID', None) == 0x9f89304e:
-            cid = update.CONSTRUCTOR_ID
-            sig = inspect.signature(update.__init__)
-            for param in sig.parameters.values():
-                vec = _has_field.get((param.name, param.annotation))
-                if vec is not None:
-                    vec.append(cid)
-
-    # Future-proof check: if the documentation format ever changes
-    # then we won't be able to pick the update types we are interested
-    # in, so we must make sure we have at least an update for each field
-    # which likely means we are doing it right.
-    if not all(_has_field.values()):
-        raise RuntimeError('FIXME: Did the init signature or updates change?')
-
-
-# We use a function to avoid cluttering the globals (with name/update/cid/doc)
-_fill()
-
-
-class EntityCache:
-    """
-    In-memory input entity cache, defaultdict-like behaviour.
-    """
-    def add(self, entities):
-        """
-        Adds the given entities to the cache, if they weren't saved before.
-        """
-        if not utils.is_list_like(entities):
-            # Invariant: all "chats" and "users" are always iterables,
-            # and "user" never is (so we wrap it inside a list).
-            entities = itertools.chain(
-                getattr(entities, 'chats', []),
-                getattr(entities, 'users', []),
-                (hasattr(entities, 'user') and [entities.user]) or []
-            )
-
-        for entity in entities:
-            try:
-                pid = utils.get_peer_id(entity)
-                if pid not in self.__dict__:
-                    # Note: `get_input_peer` already checks for `access_hash`
-                    self.__dict__[pid] = utils.get_input_peer(entity)
-            except TypeError:
-                pass
-
-    def __getitem__(self, item):
-        """
-        Gets the corresponding :tl:`InputPeer` for the given ID or peer,
-        or raises ``KeyError`` on any error (i.e. cannot be found).
-        """
-        if not isinstance(item, int) or item < 0:
-            try:
-                return self.__dict__[utils.get_peer_id(item)]
-            except TypeError:
-                raise KeyError('Invalid key will not have entity') from None
-
-        for cls in (types.PeerUser, types.PeerChat, types.PeerChannel):
-            result = self.__dict__.get(utils.get_peer_id(cls(item)))
-            if result:
-                return result
-
-        raise KeyError('No cached entity for the given key')
-
-    def clear(self):
-        """
-        Clear the entity cache.
-        """
-        self.__dict__.clear()
-
-    def ensure_cached(
-            self,
-            update,
-            has_user_id=frozenset(_has_field[('user_id', int)]),
-            has_chat_id=frozenset(_has_field[('chat_id', int)]),
-            has_channel_id=frozenset(_has_field[('channel_id', int)]),
-            has_peer=frozenset(_has_field[('peer', 'TypePeer')] + _has_field[('peer', 'TypeDialogPeer')]),
-            has_message=frozenset(_has_field[('message', 'TypeMessage')])
-    ):
-        """
-        Ensures that all the relevant entities in the given update are cached.
-        """
-        # This method is called pretty often and we want it to have the lowest
-        # overhead possible. For that, we avoid `isinstance` and constantly
-        # getting attributes out of `types.` by "caching" the constructor IDs
-        # in sets inside the arguments, and using local variables.
-        dct = self.__dict__
-        cid = update.CONSTRUCTOR_ID
-        if cid in has_user_id and \
-                update.user_id not in dct:
-            return False
-
-        if cid in has_chat_id and \
-                utils.get_peer_id(types.PeerChat(update.chat_id)) not in dct:
-            return False
-
-        if cid in has_channel_id and \
-                utils.get_peer_id(types.PeerChannel(update.channel_id)) not in dct:
-            return False
-
-        if cid in has_peer and \
-                utils.get_peer_id(update.peer) not in dct:
-            return False
-
-        if cid in has_message:
-            x = update.message
-            y = getattr(x, 'peer_id', None)  # handle MessageEmpty
-            if y and utils.get_peer_id(y) not in dct:
-                return False
-
-            y = getattr(x, 'from_id', None)
-            if y and utils.get_peer_id(y) not in dct:
-                return False
-
-            # We don't quite worry about entities anywhere else.
-            # This is enough.
-
-        return True

telethon/errors/common.py

@@ -19,8 +19,8 @@ class TypeNotFoundError(Exception):
     def __init__(self, invalid_constructor_id, remaining):
         super().__init__(
             'Could not find a matching Constructor ID for the TLObject '
-            'that was supposed to be read with ID {:08x}. Most likely, '
+            'that was supposed to be read with ID {:08x}. See the FAQ '
-            'a TLObject was trying to be read when it should not be read. '
+            'for more details. '
             'Remaining bytes: {!r}'.format(invalid_constructor_id, remaining))
 
         self.invalid_constructor_id = invalid_constructor_id

telethon/events/album.py

@@ -160,7 +160,7 @@ class Album(EventBuilder):
         def _set_client(self, client):
             super()._set_client(client)
             self._sender, self._input_sender = utils._get_entity_pair(
-                self.sender_id, self._entities, client._entity_cache)
+                self.sender_id, self._entities, client._mb_entity_cache)
 
             for msg in self.messages:
                 msg._finish_init(client, self._entities, None)

telethon/events/callbackquery.py

@@ -151,7 +151,7 @@ class CallbackQuery(EventBuilder):
         def _set_client(self, client):
             super()._set_client(client)
             self._sender, self._input_sender = utils._get_entity_pair(
-                self.sender_id, self._entities, client._entity_cache)
+                self.sender_id, self._entities, client._mb_entity_cache)
 
         @property
         def id(self):
@@ -208,8 +208,9 @@ class CallbackQuery(EventBuilder):
             if not getattr(self._input_sender, 'access_hash', True):
                 # getattr with True to handle the InputPeerSelf() case
                 try:
-                    self._input_sender = self._client._entity_cache[self._sender_id]
+                    self._input_sender = self._client._mb_entity_cache.get(
-                except KeyError:
+                        utils.resolve_id(self._sender_id)[0])._as_input_peer()
+                except AttributeError:
                     m = await self.get_message()
                     if m:
                         self._sender = m._sender
@@ -299,7 +300,7 @@ class CallbackQuery(EventBuilder):
             """
             Edits the message. Shorthand for
             `telethon.client.messages.MessageMethods.edit_message` with
-            the ``entity`` set to the correct :tl:`InputBotInlineMessageID`.
+            the ``entity`` set to the correct :tl:`InputBotInlineMessageID` or :tl:`InputBotInlineMessageID64`.
 
             Returns `True` if the edit was successful.
 
@@ -312,7 +313,7 @@ class CallbackQuery(EventBuilder):
                 since the message object is normally not present.
             """
             self._client.loop.create_task(self.answer())
-            if isinstance(self.query.msg_id, types.InputBotInlineMessageID):
+            if isinstance(self.query.msg_id, (types.InputBotInlineMessageID, types.InputBotInlineMessageID64)):
                 return await self._client.edit_message(
                     self.query.msg_id, *args, **kwargs
                 )
@@ -337,6 +338,8 @@ class CallbackQuery(EventBuilder):
             This method will likely fail if `via_inline` is `True`.
             """
             self._client.loop.create_task(self.answer())
+            if isinstance(self.query.msg_id, (types.InputBotInlineMessageID, types.InputBotInlineMessageID64)):
+                raise TypeError('Inline messages cannot be deleted as there is no API request available to do so')
             return await self._client.delete_messages(
                 await self.get_input_chat(), [self.query.msg_id],
                 *args, **kwargs

telethon/events/chataction.py

@@ -108,6 +108,17 @@ class ChatAction(EventBuilder):
                 return cls.Event(msg,
                                  new_score=action.score)
 
+        elif isinstance(update, types.UpdateChannelParticipant) \
+                and bool(update.new_participant) != bool(update.prev_participant):
+            # If members are hidden, bots will receive this update instead,
+            # as there won't be a service message. Promotions and demotions
+            # seem to have both new and prev participant, which are ignored
+            # by this event.
+            return cls.Event(types.PeerChannel(update.channel_id),
+                             users=update.user_id,
+                             added_by=update.actor_id if update.new_participant else None,
+                             kicked_by=update.actor_id if update.prev_participant else None)
+
     class Event(EventCommon):
         """
         Represents the event of a new chat action.
@@ -414,9 +425,10 @@ class ChatAction(EventBuilder):
 
                     # If missing, try from the entity cache
                     try:
-                        self._input_users.append(self._client._entity_cache[user_id])
+                        self._input_users.append(self._client._mb_entity_cache.get(
+                            utils.resolve_id(user_id)[0])._as_input_peer())
                         continue
-                    except KeyError:
+                    except AttributeError:
                         pass
 
             return self._input_users or []

telethon/events/common.py

@@ -154,7 +154,7 @@ class EventCommon(ChatGetter, abc.ABC):
         self._client = client
         if self._chat_peer:
             self._chat, self._input_chat = utils._get_entity_pair(
-                self.chat_id, self._entities, client._entity_cache)
+                self.chat_id, self._entities, client._mb_entity_cache)
         else:
             self._chat = self._input_chat = None
 

telethon/events/inlinequery.py

@@ -4,7 +4,7 @@ import re
 import asyncio
 
 from .common import EventBuilder, EventCommon, name_inner_event
-from .. import utils
+from .. import utils, helpers
 from ..tl import types, functions, custom
 from ..tl.custom.sendergetter import SenderGetter
 
@@ -99,7 +99,7 @@ class InlineQuery(EventBuilder):
         def _set_client(self, client):
             super()._set_client(client)
             self._sender, self._input_sender = utils._get_entity_pair(
-                self.sender_id, self._entities, client._entity_cache)
+                self.sender_id, self._entities, client._mb_entity_cache)
 
         @property
         def id(self):
@@ -242,6 +242,6 @@ class InlineQuery(EventBuilder):
             if inspect.isawaitable(obj):
                 return asyncio.ensure_future(obj)
 
-            f = asyncio.get_event_loop().create_future()
+            f = helpers.get_running_loop().create_future()
             f.set_result(obj)
             return f

telethon/events/userupdate.py

@@ -14,7 +14,7 @@ from ..tl.custom.sendergetter import SenderGetter
 #      in a single place will make it annoying to use (since
 #      the user needs to check for the existence of `None`).
 #
-# TODO Handle UpdateUserBlocked, UpdateUserName, UpdateUserPhone, UpdateUserPhoto
+# TODO Handle UpdateUserBlocked, UpdateUserName, UpdateUserPhone, UpdateUser
 
 def _requires_action(function):
     @functools.wraps(function)
@@ -95,7 +95,7 @@ class UserUpdate(EventBuilder):
         def _set_client(self, client):
             super()._set_client(client)
             self._sender, self._input_sender = utils._get_entity_pair(
-                self.sender_id, self._entities, client._entity_cache)
+                self.sender_id, self._entities, client._mb_entity_cache)
 
         @property
         def user(self):
@@ -246,7 +246,7 @@ class UserUpdate(EventBuilder):
             return isinstance(self.action, types.SendMessageUploadPhotoAction)
 
         @property
-        @_requires_action
+        @_requires_status
         def last_seen(self):
             """
             Exact `datetime.datetime` when the user was last seen if known.

telethon/extensions/binaryreader.py

@@ -1,11 +1,9 @@
 """
 This module contains the BinaryReader utility class.
 """
-import os
+import struct
 import time
-from datetime import datetime, timezone, timedelta
+from datetime import datetime, timedelta, timezone
-from io import BytesIO
-from struct import unpack
 
 from ..errors import TypeNotFoundError
 from ..tl.alltlobjects import tlobjects
@@ -21,7 +19,8 @@ class BinaryReader:
     """
 
     def __init__(self, data):
-        self.stream = BytesIO(data)
+        self.stream = data or b''
+        self.position = 0
         self._last = None  # Should come in handy to spot -404 errors
 
     # region Reading
@@ -30,23 +29,35 @@ class BinaryReader:
     # https://core.telegram.org/mtproto
     def read_byte(self):
         """Reads a single byte value."""
-        return self.read(1)[0]
+        value, = struct.unpack_from("<B", self.stream, self.position)
+        self.position += 1
+        return value
 
     def read_int(self, signed=True):
         """Reads an integer (4 bytes) value."""
-        return int.from_bytes(self.read(4), byteorder='little', signed=signed)
+        fmt = '<i' if signed else '<I'
+        value, = struct.unpack_from(fmt, self.stream, self.position)
+        self.position += 4
+        return value
 
     def read_long(self, signed=True):
         """Reads a long integer (8 bytes) value."""
-        return int.from_bytes(self.read(8), byteorder='little', signed=signed)
+        fmt = '<q' if signed else '<Q'
+        value, = struct.unpack_from(fmt, self.stream, self.position)
+        self.position += 8
+        return value
 
     def read_float(self):
         """Reads a real floating point (4 bytes) value."""
-        return unpack('<f', self.read(4))[0]
+        value, = struct.unpack_from("<f", self.stream, self.position)
+        self.position += 4
+        return value
 
     def read_double(self):
         """Reads a real floating point (8 bytes) value."""
-        return unpack('<d', self.read(8))[0]
+        value, = struct.unpack_from("<d", self.stream, self.position)
+        self.position += 8
+        return value
 
     def read_large_int(self, bits, signed=True):
         """Reads a n-bits long integer value."""
@@ -55,7 +66,12 @@ class BinaryReader:
 
     def read(self, length=-1):
         """Read the given amount of bytes, or -1 to read all remaining."""
-        result = self.stream.read(length)
+        if length >= 0:
+            result = self.stream[self.position:self.position + length]
+            self.position += length
+        else:
+            result = self.stream[self.position:]
+            self.position += len(result)
         if (length >= 0) and (len(result) != length):
             raise BufferError(
                 'No more data left to read (need {}, got {}: {}); last read {}'
@@ -67,7 +83,7 @@ class BinaryReader:
 
     def get_bytes(self):
         """Gets the byte array representing the current buffer as a whole."""
-        return self.stream.getvalue()
+        return self.stream
 
     # endregion
 
@@ -153,24 +169,24 @@ class BinaryReader:
 
     def close(self):
         """Closes the reader, freeing the BytesIO stream."""
-        self.stream.close()
+        self.stream = b''
 
     # region Position related
 
     def tell_position(self):
         """Tells the current position on the stream."""
-        return self.stream.tell()
+        return self.position
 
     def set_position(self, position):
         """Sets the current position on the stream."""
-        self.stream.seek(position)
+        self.position = position
 
     def seek(self, offset):
         """
         Seeks the stream position given an offset from the current position.
         The offset may be negative.
         """
-        self.stream.seek(offset, os.SEEK_CUR)
+        self.position += offset
 
     # endregion
 

telethon/extensions/html.py

@@ -1,34 +1,22 @@
 """
 Simple HTML -> Telegram entity parser.
 """
-import struct
 from collections import deque
 from html import escape
 from html.parser import HTMLParser
-from typing import Iterable, Optional, Tuple, List
+from typing import Iterable, Tuple, List
 
-from .. import helpers
+from ..helpers import add_surrogate, del_surrogate, within_surrogate, strip_text
+from ..tl import TLObject
 from ..tl.types import (
     MessageEntityBold, MessageEntityItalic, MessageEntityCode,
     MessageEntityPre, MessageEntityEmail, MessageEntityUrl,
     MessageEntityTextUrl, MessageEntityMentionName,
     MessageEntityUnderline, MessageEntityStrike, MessageEntityBlockquote,
-    TypeMessageEntity
+    MessageEntityCustomEmoji, TypeMessageEntity
 )
 
 
-# Helpers from markdown.py
-def _add_surrogate(text):
-    return ''.join(
-        ''.join(chr(y) for y in struct.unpack('<HH', x.encode('utf-16le')))
-        if (0x10000 <= ord(x) <= 0x10FFFF) else x for x in text
-    )
-
-
-def _del_surrogate(text):
-    return text.encode('utf-16', 'surrogatepass').decode('utf-16')
-
-
 class HTMLToTelegramParser(HTMLParser):
     def __init__(self):
         super().__init__()
@@ -86,10 +74,18 @@ class HTMLToTelegramParser(HTMLParser):
                     EntityType = MessageEntityUrl
                 else:
                     EntityType = MessageEntityTextUrl
-                    args['url'] = _del_surrogate(url)
+                    args['url'] = del_surrogate(url)
                     url = None
             self._open_tags_meta.popleft()
             self._open_tags_meta.appendleft(url)
+        elif tag == 'tg-emoji':
+            try:
+                emoji_id = int(attrs['emoji-id'])
+            except (KeyError, ValueError):
+                return
+
+            EntityType = MessageEntityCustomEmoji
+            args['document_id'] = emoji_id
             
         if EntityType and tag not in self._building_entities:
             self._building_entities[tag] = EntityType(
@@ -133,13 +129,36 @@ def parse(html: str) -> Tuple[str, List[TypeMessageEntity]]:
         return html, []
 
     parser = HTMLToTelegramParser()
-    parser.feed(_add_surrogate(html))
+    parser.feed(add_surrogate(html))
-    text = helpers.strip_text(parser.text, parser.entities)
+    text = strip_text(parser.text, parser.entities)
-    return _del_surrogate(text), parser.entities
+    parser.entities.reverse()
+    parser.entities.sort(key=lambda entity: entity.offset)
+    return del_surrogate(text), parser.entities
 
 
-def unparse(text: str, entities: Iterable[TypeMessageEntity], _offset: int = 0,
+ENTITY_TO_FORMATTER = {
-            _length: Optional[int] = None) -> str:
+    MessageEntityBold: ('<strong>', '</strong>'),
+    MessageEntityItalic: ('<em>', '</em>'),
+    MessageEntityCode: ('<code>', '</code>'),
+    MessageEntityUnderline: ('<u>', '</u>'),
+    MessageEntityStrike: ('<del>', '</del>'),
+    MessageEntityBlockquote: ('<blockquote>', '</blockquote>'),
+    MessageEntityPre: lambda e, _: (
+        "<pre>\n"
+        "    <code class='language-{}'>\n"
+        "        ".format(e.language), "{}\n"
+        "    </code>\n"
+        "</pre>"
+    ),
+    MessageEntityEmail: lambda _, t: ('<a href="mailto:{}">'.format(t), '</a>'),
+    MessageEntityUrl: lambda _, t: ('<a href="{}">'.format(t), '</a>'),
+    MessageEntityTextUrl: lambda e, _: ('<a href="{}">'.format(escape(e.url)), '</a>'),
+    MessageEntityMentionName: lambda e, _: ('<a href="tg://user?id={}">'.format(e.user_id), '</a>'),
+    MessageEntityCustomEmoji: lambda e, _: ('<tg-emoji emoji-id="{}">'.format(e.document_id), '</tg-emoji>'),
+}
+
+
+def unparse(text: str, entities: Iterable[TypeMessageEntity]) -> str:
     """
     Performs the reverse operation to .parse(), effectively returning HTML
     given a normal text and its MessageEntity's.
@@ -153,77 +172,32 @@ def unparse(text: str, entities: Iterable[TypeMessageEntity], _offset: int = 0,
     elif not entities:
         return escape(text)
 
-    text = _add_surrogate(text)
+    if isinstance(entities, TLObject):
-    if _length is None:
+        entities = (entities,)
-        _length = len(text)
+
-    html = []
+    text = add_surrogate(text)
-    last_offset = 0
+    insert_at = []
     for i, entity in enumerate(entities):
-        if entity.offset >= _offset + _length:
+        s = entity.offset
-            break
+        e = entity.offset + entity.length
-        relative_offset = entity.offset - _offset
+        delimiter = ENTITY_TO_FORMATTER.get(type(entity), None)
-        if relative_offset > last_offset:
+        if delimiter:
-            html.append(escape(text[last_offset:relative_offset]))
+            if callable(delimiter):
-        elif relative_offset < last_offset:
+                delimiter = delimiter(entity, text[s:e])
-            continue
+            insert_at.append((s, i, delimiter[0]))
+            insert_at.append((e, -i, delimiter[1]))
 
-        skip_entity = False
+    insert_at.sort(key=lambda t: (t[0], t[1]))
-        length = entity.length
+    next_escape_bound = len(text)
+    while insert_at:
+        # Same logic as markdown.py
+        at, _, what = insert_at.pop()
+        while within_surrogate(text, at):
+            at += 1
 
-        # If we are in the middle of a surrogate nudge the position by +1.
+        text = text[:at] + what + escape(text[at:next_escape_bound]) + text[next_escape_bound:]
-        # Otherwise we would end up with malformed text and fail to encode.
+        next_escape_bound = at
-        # For example of bad input: "Hi \ud83d\ude1c"
-        # https://en.wikipedia.org/wiki/UTF-16#U+010000_to_U+10FFFF
-        while helpers.within_surrogate(text, relative_offset, length=_length):
-            relative_offset += 1
 
-        while helpers.within_surrogate(text, relative_offset + length, length=_length):
+    text = escape(text[:next_escape_bound]) + text[next_escape_bound:]
-            length += 1
 
-        entity_text = unparse(text=text[relative_offset:relative_offset + length],
+    return del_surrogate(text)
-                              entities=entities[i + 1:],
-                              _offset=entity.offset, _length=length)
-        entity_type = type(entity)
-
-        if entity_type == MessageEntityBold:
-            html.append('<strong>{}</strong>'.format(entity_text))
-        elif entity_type == MessageEntityItalic:
-            html.append('<em>{}</em>'.format(entity_text))
-        elif entity_type == MessageEntityCode:
-            html.append('<code>{}</code>'.format(entity_text))
-        elif entity_type == MessageEntityUnderline:
-            html.append('<u>{}</u>'.format(entity_text))
-        elif entity_type == MessageEntityStrike:
-            html.append('<del>{}</del>'.format(entity_text))
-        elif entity_type == MessageEntityBlockquote:
-            html.append('<blockquote>{}</blockquote>'.format(entity_text))
-        elif entity_type == MessageEntityPre:
-            if entity.language:
-                html.append(
-                    "<pre>\n"
-                    "    <code class='language-{}'>\n"
-                    "        {}\n"
-                    "    </code>\n"
-                    "</pre>".format(entity.language, entity_text))
-            else:
-                html.append('<pre><code>{}</code></pre>'
-                            .format(entity_text))
-        elif entity_type == MessageEntityEmail:
-            html.append('<a href="mailto:{0}">{0}</a>'.format(entity_text))
-        elif entity_type == MessageEntityUrl:
-            html.append('<a href="{0}">{0}</a>'.format(entity_text))
-        elif entity_type == MessageEntityTextUrl:
-            html.append('<a href="{}">{}</a>'
-                        .format(escape(entity.url), entity_text))
-        elif entity_type == MessageEntityMentionName:
-            html.append('<a href="tg://user?id={}">{}</a>'
-                        .format(entity.user_id, entity_text))
-        else:
-            skip_entity = True
-        last_offset = relative_offset + (0 if skip_entity else length)
-
-    while helpers.within_surrogate(text, last_offset, length=_length):
-        last_offset += 1
-
-    html.append(escape(text[last_offset:]))
-    return _del_surrogate(''.join(html))

telethon/extensions/markdown.py

@@ -22,14 +22,10 @@ DEFAULT_DELIMITERS = {
     '```': MessageEntityPre
 }
 
-DEFAULT_URL_RE = re.compile(r'\[([^\]]+)\]\(([^)]+)\)')
+DEFAULT_URL_RE = re.compile(r'\[([^]]*?)\]\(([\s\S]*?)\)')
 DEFAULT_URL_FORMAT = '[{0}]({1})'
 
 
-def overlap(a, b, x, y):
-    return max(a, x) < min(b, y)
-
-
 def parse(message, delimiters=None, url_re=None):
     """
     Parses the given markdown message and returns its stripped representation
@@ -90,8 +86,8 @@ def parse(message, delimiters=None, url_re=None):
                 for ent in result:
                     # If the end is after our start, it is affected
                     if ent.offset + ent.length > i:
-                        # If the old start is also before ours, it is fully enclosed
+                        # If the old start is before ours and the old end is after ours, we are fully enclosed
-                        if ent.offset <= i:
+                        if ent.offset <= i and ent.offset + ent.length >= end + len(delim):
                             ent.length -= len(delim) * 2
                         else:
                             ent.length -= len(delim)
@@ -119,7 +115,7 @@ def parse(message, delimiters=None, url_re=None):
                     message[m.end():]
                 ))
 
-                delim_size = m.end() - m.start() - len(m.group())
+                delim_size = m.end() - m.start() - len(m.group(1))
                 for ent in result:
                     # If the end is after our start, it is affected
                     if ent.offset + ent.length > m.start():
@@ -164,13 +160,13 @@ def unparse(text, entities, delimiters=None, url_fmt=None):
     text = add_surrogate(text)
     delimiters = {v: k for k, v in delimiters.items()}
     insert_at = []
-    for entity in entities:
+    for i, entity in enumerate(entities):
         s = entity.offset
         e = entity.offset + entity.length
         delimiter = delimiters.get(type(entity), None)
         if delimiter:
-            insert_at.append((s, delimiter))
+            insert_at.append((s, i, delimiter))
-            insert_at.append((e, delimiter))
+            insert_at.append((e, -i, delimiter))
         else:
             url = None
             if isinstance(entity, MessageEntityTextUrl):
@@ -178,12 +174,12 @@ def unparse(text, entities, delimiters=None, url_fmt=None):
             elif isinstance(entity, MessageEntityMentionName):
                 url = 'tg://user?id={}'.format(entity.user_id)
             if url:
-                insert_at.append((s, '['))
+                insert_at.append((s, i, '['))
-                insert_at.append((e, ']({})'.format(url)))
+                insert_at.append((e, -i, ']({})'.format(url)))
 
-    insert_at.sort(key=lambda t: t[0])
+    insert_at.sort(key=lambda t: (t[0], t[1]))
     while insert_at:
-        at, what = insert_at.pop()
+        at, _, what = insert_at.pop()
 
         # If we are in the middle of a surrogate nudge the position by -1.
         # Otherwise we would end up with malformed text and fail to encode.

telethon/helpers.py

@@ -7,6 +7,7 @@ import struct
 import inspect
 import logging
 import functools
+import sys
 from pathlib import Path
 from hashlib import sha1
 
@@ -57,7 +58,7 @@ def within_surrogate(text, index, *, length=None):
 
     return (
             1 < index < len(text) and  # in bounds
-            '\ud800' <= text[index - 1] <= '\udfff' and  # previous is
+            '\ud800' <= text[index - 1] <= '\udbff' and  # previous is
             '\ud800' <= text[index] <= '\udfff'  # current is
     )
 
@@ -347,21 +348,22 @@ class _FileStream(io.IOBase):
             self._size = os.path.getsize(self._file)
             self._stream = open(self._file, 'rb')
             self._close_stream = True
+            return self
 
-        elif isinstance(self._file, bytes):
+        if isinstance(self._file, bytes):
             self._size = len(self._file)
             self._stream = io.BytesIO(self._file)
             self._close_stream = True
+            return self
 
-        elif not callable(getattr(self._file, 'read', None)):
+        if not callable(getattr(self._file, 'read', None)):
             raise TypeError('file description should have a `read` method')
 
-        elif self._size is not None:
         self._name = getattr(self._file, 'name', None)
         self._stream = self._file
         self._close_stream = False
 
-        else:
+        if self._size is None:
             if callable(getattr(self._file, 'seekable', None)):
                 seekable = await _maybe_await(self._file.seekable())
             else:
@@ -372,8 +374,6 @@ class _FileStream(io.IOBase):
                 await _maybe_await(self._file.seek(0, os.SEEK_END))
                 self._size = await _maybe_await(self._file.tell())
                 await _maybe_await(self._file.seek(pos, os.SEEK_SET))
-                self._stream = self._file
-                self._close_stream = False
             else:
                 _log.warning(
                     'Could not determine file size beforehand so the entire '
@@ -423,3 +423,12 @@ class _FileStream(io.IOBase):
         pass
 
 # endregion
+
+def get_running_loop():
+    if sys.version_info >= (3, 7):
+        try:
+            return asyncio.get_running_loop()
+        except RuntimeError:
+            return asyncio.get_event_loop_policy().get_event_loop()
+    else:
+        return asyncio.get_event_loop()

telethon/hints.py

@@ -44,7 +44,12 @@ FileLike = typing.Union[
     typing.BinaryIO,
     types.TypeMessageMedia,
     types.TypeInputFile,
-    types.TypeInputFileLocation
+    types.TypeInputFileLocation,
+    types.TypeInputMedia,
+    types.TypePhoto,
+    types.TypeInputPhoto,
+    types.TypeDocument,
+    types.TypeInputDocument
 ]
 
 # Can't use `typing.Type` in Python 3.5.2

telethon/network/connection/connection.py

@@ -13,7 +13,7 @@ try:
 except ImportError:
     python_socks = None
 
-from ...errors import InvalidChecksumError
+from ...errors import InvalidChecksumError, InvalidBufferError
 from ... import helpers
 
 
@@ -116,9 +116,15 @@ class Connection(abc.ABC):
             # python_socks internal errors are not inherited from
             # builtin IOError (just from Exception). Instead of adding those
             # in exceptions clauses everywhere through the code, we
-            # rather monkey-patch them in place.
+            # rather monkey-patch them in place. Keep in mind that
+            # ProxyError takes error_code as keyword argument.
 
-            python_socks._errors.ProxyError = ConnectionError
+            class ConnectionErrorExtra(ConnectionError):
+                def __init__(self, message, error_code=None):
+                    super().__init__(message)
+                    self.error_code = error_code
+
+            python_socks._errors.ProxyError = ConnectionErrorExtra
             python_socks._errors.ProxyConnectionError = ConnectionError
             python_socks._errors.ProxyTimeoutError = ConnectionError
 
@@ -155,7 +161,7 @@ class Connection(abc.ABC):
 
                 # Actual TCP connection is performed here.
                 await asyncio.wait_for(
-                    asyncio.get_event_loop().sock_connect(sock=sock, address=address),
+                    helpers.get_running_loop().sock_connect(sock=sock, address=address),
                     timeout=timeout
                 )
 
@@ -190,7 +196,7 @@ class Connection(abc.ABC):
 
             # Actual TCP connection and negotiation performed here.
             await asyncio.wait_for(
-                asyncio.get_event_loop().sock_connect(sock=sock, address=address),
+                helpers.get_running_loop().sock_connect(sock=sock, address=address),
                 timeout=timeout
             )
 
@@ -244,7 +250,7 @@ class Connection(abc.ABC):
         await self._connect(timeout=timeout, ssl=ssl)
         self._connected = True
 
-        loop = asyncio.get_event_loop()
+        loop = helpers.get_running_loop()
         self._send_task = loop.create_task(self._send_loop())
         self._recv_task = loop.create_task(self._recv_loop())
 
@@ -253,6 +259,9 @@ class Connection(abc.ABC):
         Disconnects from the server, and clears
         pending outgoing and incoming messages.
         """
+        if not self._connected:
+            return
+
         self._connected = False
 
         await helpers._cancel(
@@ -296,8 +305,10 @@ class Connection(abc.ABC):
         This method returns a coroutine.
         """
         while self._connected:
-            result = await self._recv_queue.get()
+            result, err = await self._recv_queue.get()
-            if result:  # None = sentinel value = keep trying
+            if err:
+                raise err
+            if result:
                 return result
 
         raise ConnectionError('Not connected')
@@ -324,34 +335,31 @@ class Connection(abc.ABC):
         """
         This loop is constantly putting items on the queue as they're read.
         """
+        try:
             while self._connected:
                 try:
                     data = await self._recv()
                 except asyncio.CancelledError:
                     break
+                except (IOError, asyncio.IncompleteReadError) as e:
+                    self._log.warning('Server closed the connection: %s', e)
+                    await self._recv_queue.put((None, e))
+                    await self.disconnect()
+                except InvalidChecksumError as e:
+                    self._log.warning('Server response had invalid checksum: %s', e)
+                    await self._recv_queue.put((None, e))
+                except InvalidBufferError as e:
+                    self._log.warning('Server response had invalid buffer: %s', e)
+                    await self._recv_queue.put((None, e))
                 except Exception as e:
-                if isinstance(e, (IOError, asyncio.IncompleteReadError)):
+                    self._log.exception('Unexpected exception in the receive loop')
-                    msg = 'The server closed the connection'
+                    await self._recv_queue.put((None, e))
-                    self._log.info(msg)
+                    await self.disconnect()
-                elif isinstance(e, InvalidChecksumError):
-                    msg = 'The server response had an invalid checksum'
-                    self._log.info(msg)
                 else:
-                    msg = 'Unexpected exception in the receive loop'
+                    await self._recv_queue.put((data, None))
-                    self._log.exception(msg)
+        finally:
-
             await self.disconnect()
 
-                # Add a sentinel value to unstuck recv
-                if self._recv_queue.empty():
-                    self._recv_queue.put_nowait(None)
-
-                break
-
-            try:
-                await self._recv_queue.put(data)
-            except asyncio.CancelledError:
-                break
 
     def _init_conn(self):
         """

telethon/network/connection/tcpfull.py

@@ -2,7 +2,7 @@ import struct
 from zlib import crc32
 
 from .connection import Connection, PacketCodec
-from ...errors import InvalidChecksumError
+from ...errors import InvalidChecksumError, InvalidBufferError
 
 
 class FullPacketCodec(PacketCodec):
@@ -24,6 +24,18 @@ class FullPacketCodec(PacketCodec):
     async def read_packet(self, reader):
         packet_len_seq = await reader.readexactly(8)  # 4 and 4
         packet_len, seq = struct.unpack('<ii', packet_len_seq)
+        if packet_len < 0 and seq < 0:
+            # It has been observed that the length and seq can be -429,
+            # followed by the body of 4 bytes also being -429.
+            # See https://github.com/LonamiWebs/Telethon/issues/4042.
+            body = await reader.readexactly(4)
+            raise InvalidBufferError(body)
+        elif packet_len < 8:
+            # Currently unknown why packet_len may be less than 8 but not negative.
+            # Attempting to `readexactly` with less than 0 fails without saying what
+            # the number was which is less helpful.
+            raise InvalidBufferError(packet_len_seq)
+
         body = await reader.readexactly(packet_len - 8)
         checksum = struct.unpack('<I', body[-4:])[0]
         body = body[:-4]

telethon/network/connection/tcpmtproxy.py

@@ -1,5 +1,6 @@
 import asyncio
 import hashlib
+import base64
 import os
 
 from .connection import ObfuscatedConnection
@@ -98,7 +99,7 @@ class TcpMTProxy(ObfuscatedConnection):
     def __init__(self, ip, port, dc_id, *, loggers, proxy=None, local_addr=None):
         # connect to proxy's host and port instead of telegram's ones
         proxy_host, proxy_port = self.address_info(proxy)
-        self._secret = bytes.fromhex(proxy[2])
+        self._secret = self.normalize_secret(proxy[2])
         super().__init__(
             proxy_host, proxy_port, dc_id, loggers=loggers)
 
@@ -130,6 +131,18 @@ class TcpMTProxy(ObfuscatedConnection):
             raise ValueError("No proxy info specified for MTProxy connection")
         return proxy_info[:2]
 
+    @staticmethod
+    def normalize_secret(secret):
+        if secret[:2] in ("ee", "dd"):  # Remove extra bytes
+            secret = secret[2:]
+
+        try:
+            secret_bytes = bytes.fromhex(secret)
+        except ValueError:
+            secret = secret + '=' * (-len(secret) % 4)
+            secret_bytes = base64.b64decode(secret.encode())
+
+        return secret_bytes[:16]  # Remove the domain from the secret (until domain support is added)
 
 class ConnectionTcpMTProxyAbridged(TcpMTProxy):
     """

telethon/network/mtprotosender.py

@@ -1,6 +1,8 @@
 import asyncio
 import collections
 import struct
+import datetime
+import time
 
 from . import authenticator
 from ..extensions.messagepacker import MessagePacker
@@ -16,11 +18,12 @@ from ..errors import (
 from ..extensions import BinaryReader
 from ..tl.core import RpcResult, MessageContainer, GzipPacked
 from ..tl.functions.auth import LogOutRequest
-from ..tl.functions import PingRequest, DestroySessionRequest
+from ..tl.functions import PingRequest, DestroySessionRequest, DestroyAuthKeyRequest
 from ..tl.types import (
     MsgsAck, Pong, BadServerSalt, BadMsgNotification, FutureSalts,
     MsgNewDetailedInfo, NewSessionCreated, MsgDetailedInfo, MsgsStateReq,
     MsgsStateInfo, MsgsAllInfo, MsgResendReq, upload, DestroySessionOk, DestroySessionNone,
+    DestroyAuthKeyOk, DestroyAuthKeyNone, DestroyAuthKeyFail
 )
 from ..tl import types as _tl
 from ..crypto import AuthKey
@@ -67,7 +70,7 @@ class MTProtoSender:
         # pending futures should be cancelled.
         self._user_connected = False
         self._reconnecting = False
-        self._disconnected = asyncio.get_event_loop().create_future()
+        self._disconnected = helpers.get_running_loop().create_future()
         self._disconnected.set_result(None)
 
         # We need to join the loops upon disconnection
@@ -109,8 +112,11 @@ class MTProtoSender:
             MsgsStateReq.CONSTRUCTOR_ID: self._handle_state_forgotten,
             MsgResendReq.CONSTRUCTOR_ID: self._handle_state_forgotten,
             MsgsAllInfo.CONSTRUCTOR_ID: self._handle_msg_all,
-            DestroySessionOk: self._handle_destroy_session,
+            DestroySessionOk.CONSTRUCTOR_ID: self._handle_destroy_session,
-            DestroySessionNone: self._handle_destroy_session,
+            DestroySessionNone.CONSTRUCTOR_ID: self._handle_destroy_session,
+            DestroyAuthKeyOk.CONSTRUCTOR_ID: self._handle_destroy_auth_key,
+            DestroyAuthKeyNone.CONSTRUCTOR_ID: self._handle_destroy_auth_key,
+            DestroyAuthKeyFail.CONSTRUCTOR_ID: self._handle_destroy_auth_key,
         }
 
     # Public API
@@ -257,7 +263,7 @@ class MTProtoSender:
             await self._disconnect(error=e)
             raise e
 
-        loop = asyncio.get_event_loop()
+        loop = helpers.get_running_loop()
         self._log.debug('Starting send loop')
         self._send_loop_handle = loop.create_task(self._send_loop())
 
@@ -296,7 +302,7 @@ class MTProtoSender:
             # notify whenever we change it. This is crucial when we
             # switch to different data centers.
             if self._auth_key_callback:
-                self._auth_key_callback(self.auth_key)
+                await self._auth_key_callback(self.auth_key)
 
             self._log.debug('auth_key generation success!')
             return True
@@ -396,7 +402,7 @@ class MTProtoSender:
                 self._pending_state.clear()
 
                 if self._auto_reconnect_callback:
-                    asyncio.get_event_loop().create_task(self._auto_reconnect_callback())
+                    helpers.get_running_loop().create_task(self._auto_reconnect_callback())
 
                 break
         else:
@@ -421,7 +427,7 @@ class MTProtoSender:
             # gets stuck.
             # TODO It still gets stuck? Investigate where and why.
             self._reconnecting = True
-            asyncio.get_event_loop().create_task(self._reconnect(error))
+            helpers.get_running_loop().create_task(self._reconnect(error))
 
     def _keepalive_ping(self, rnd_id):
         """
@@ -499,8 +505,22 @@ class MTProtoSender:
             self._log.debug('Receiving items from the network...')
             try:
                 body = await self._connection.recv()
-            except IOError as e:
+            except asyncio.CancelledError:
-                self._log.info('Connection closed while receiving data')
+                raise  # bypass except Exception
+            except (IOError, asyncio.IncompleteReadError) as e:
+                self._log.info('Connection closed while receiving data: %s', e)
+                self._start_reconnect(e)
+                return
+            except InvalidBufferError as e:
+                if e.code == 429:
+                    self._log.warning('Server indicated flood error at transport level: %s', e)
+                    await self._disconnect(error=e)
+                else:
+                    self._log.exception('Server sent invalid buffer')
+                    self._start_reconnect(e)
+                return
+            except Exception as e:
+                self._log.exception('Unhandled error while receiving data')
                 self._start_reconnect(e)
                 return
 
@@ -528,7 +548,7 @@ class MTProtoSender:
                     self._start_reconnect(e)
                 return
             except Exception as e:
-                self._log.exception('Unhandled error while receiving data')
+                self._log.exception('Unhandled error while decrypting data')
                 self._start_reconnect(e)
                 return
 
@@ -653,7 +673,11 @@ class MTProtoSender:
         try:
             assert message.obj.SUBCLASS_OF_ID == 0x8af52aac  # crc32(b'Updates')
         except AssertionError:
-            self._log.warning('Note: %s is not an update, not dispatching it %s', message.obj)
+            self._log.warning(
+                'Note: %s is not an update, not dispatching it %s',
+                message.obj.__class__.__name__,
+                message.obj
+            )
             return
 
         self._log.debug('Handling update %s', message.obj.__class__.__name__)
@@ -666,11 +690,35 @@ class MTProtoSender:
         _tl.UpdatesCombined.CONSTRUCTOR_ID,
         _tl.Updates.CONSTRUCTOR_ID,
         _tl.UpdateShortSentMessage.CONSTRUCTOR_ID,
+    )), _update_like_ids=frozenset((
+        _tl.messages.AffectedHistory.CONSTRUCTOR_ID,
+        _tl.messages.AffectedMessages.CONSTRUCTOR_ID,
+        _tl.messages.AffectedFoundMessages.CONSTRUCTOR_ID,
     ))):
         try:
             if obj.CONSTRUCTOR_ID in _update_ids:
                 obj._self_outgoing = True  # flag to only process, but not dispatch these
                 self._updates_queue.put_nowait(obj)
+            elif obj.CONSTRUCTOR_ID in _update_like_ids:
+                # Ugly "hack" (?) - otherwise bots reliably detect gaps when deleting messages.
+                #
+                # Note: the `date` being `None` is used to check for `updatesTooLong`, so epoch
+                # is used instead. It is still not read, because `updateShort` has no `seq`.
+                #
+                # Some requests, such as `readHistory`, also return these types. But the `pts_count`
+                # seems to be zero, so while this will produce some bogus `updateDeleteMessages`,
+                # it's still one of the "cleaner" approaches to handling the new `pts`.
+                # `updateDeleteMessages` is probably the "least-invasive" update that can be used.
+                upd = _tl.UpdateShort(
+                    _tl.UpdateDeleteMessages([], obj.pts, obj.pts_count),
+                    datetime.datetime(*time.gmtime(0)[:6]).replace(tzinfo=datetime.timezone.utc)
+                )
+                upd._self_outgoing = True
+                self._updates_queue.put_nowait(upd)
+            elif obj.CONSTRUCTOR_ID == _tl.messages.InvitedUsers.CONSTRUCTOR_ID:
+                obj.updates._self_outgoing = True
+                self._updates_queue.put_nowait(obj.updates)
+
         except AttributeError:
             pass
 
@@ -845,3 +893,26 @@ class MTProtoSender:
         del self._pending_state[msg_id]
         if not state.future.cancelled():
             state.future.set_result(message.obj)
+
+    async def _handle_destroy_auth_key(self, message):
+        """
+        Handles :tl:`DestroyAuthKeyFail`, :tl:`DestroyAuthKeyNone`, and :tl:`DestroyAuthKeyOk`.
+
+        :tl:`DestroyAuthKey` is not intended for users to use, but they still
+        might, and the response won't come in `rpc_result`, so thhat's worked
+        around here.
+        """
+        self._log.debug('Handling destroy auth key %s', message.obj)
+        for msg_id, state in list(self._pending_state.items()):
+            if isinstance(state.request, DestroyAuthKeyRequest):
+                del self._pending_state[msg_id]
+                if not state.future.cancelled():
+                    state.future.set_result(message.obj)
+
+        # If the auth key has been destroyed, that pretty much means the
+        # library can't continue as our auth key will no longer be found
+        # on the server.
+        # Even if the library didn't disconnect, the server would (and then
+        # the library would reconnect and learn about auth key being invalid).
+        if isinstance(message.obj, DestroyAuthKeyOk):
+            await self._disconnect(error=AuthKeyNotFound())

telethon/network/mtprotostate.py

@@ -152,7 +152,7 @@ class MTProtoState:
         """
         Inverse of `encrypt_message_data` for incoming server messages.
         """
-        now = time.time() + self.time_offset  # get the time as early as possible, even if other checks make it go unused
+        now = time.time()  # get the time as early as possible, even if other checks make it go unused
 
         if len(body) < 8:
             raise InvalidBufferError(body)
@@ -176,7 +176,7 @@ class MTProtoState:
         reader = BinaryReader(body)
         reader.read_long()  # remote_salt
         if reader.read_long() != self.id:
-            raise SecurityError('Server replied with a wrong session ID')
+            raise SecurityError('Server replied with a wrong session ID (see FAQ for details)')
 
         remote_msg_id = reader.read_long()
 
@@ -203,17 +203,23 @@ class MTProtoState:
         # messages to change server_salt and notifications about invalid time on the client."
         #
         # This means we skip the time check for certain types of messages.
-        if obj.CONSTRUCTOR_ID not in (BadServerSalt.CONSTRUCTOR_ID, BadMsgNotification.CONSTRUCTOR_ID):
+        if obj.CONSTRUCTOR_ID in (BadServerSalt.CONSTRUCTOR_ID, BadMsgNotification.CONSTRUCTOR_ID):
+            if not self._highest_remote_id and not self.time_offset:
+                # If the first message we receive is a bad notification, take this opportunity
+                # to adjust the time offset. Assume it will remain stable afterwards. Updating
+                # the offset unconditionally would make the next checks pointless.
+                self.update_time_offset(remote_msg_id)
+        else:
             remote_msg_time = remote_msg_id >> 32
-            time_delta = now - remote_msg_time
+            time_delta = (now + self.time_offset) - remote_msg_time
 
             if time_delta > MSG_TOO_OLD_DELTA:
-                self._log.warning('Server sent a very old message with ID %d, ignoring', remote_msg_id)
+                self._log.warning('Server sent a very old message with ID %d, ignoring (see FAQ for details)', remote_msg_id)
                 self._count_ignored()
                 return None
 
             if -time_delta > MSG_TOO_NEW_DELTA:
-                self._log.warning('Server sent a very new message with ID %d, ignoring', remote_msg_id)
+                self._log.warning('Server sent a very new message with ID %d, ignoring (see FAQ for details)', remote_msg_id)
                 self._count_ignored()
                 return None
 

telethon/sessions/memory.py

@@ -174,7 +174,7 @@ class MemorySession(Session):
     def get_entity_rows_by_id(self, id, exact=True):
         try:
             if exact:
-                return next((id, hash) for found_id, hash, _, _, _
+                return next((found_id, hash) for found_id, hash, _, _, _
                             in self._entities if found_id == id)
             else:
                 ids = (
@@ -182,7 +182,7 @@ class MemorySession(Session):
                     utils.get_peer_id(PeerChat(id)),
                     utils.get_peer_id(PeerChannel(id))
                 )
-                return next((id, hash) for found_id, hash, _, _, _
+                return next((found_id, hash) for found_id, hash, _, _, _
                             in self._entities if found_id in ids)
         except StopIteration:
             pass

telethon/sessions/sqlite.py

@@ -2,7 +2,7 @@ import datetime
 import os
 import time
 
-from telethon.tl import types
+from ..tl import types
 from .memory import MemorySession, _SentFileType
 from .. import utils
 from ..crypto import AuthKey

telethon/sync.py

@@ -14,7 +14,7 @@ import asyncio
 import functools
 import inspect
 
-from . import events, errors, utils, connection
+from . import events, errors, utils, connection, helpers
 from .client.account import _TakeoutClient
 from .client.telegramclient import TelegramClient
 from .tl import types, functions, custom
@@ -32,7 +32,7 @@ def _syncify_wrap(t, method_name):
     @functools.wraps(method)
     def syncified(*args, **kwargs):
         coro = method(*args, **kwargs)
-        loop = asyncio.get_event_loop()
+        loop = helpers.get_running_loop()
         if loop.is_running():
             return coro
         else:

telethon/tl/core/gzippacked.py

@@ -1,3 +1,6 @@
+try:
+    from isal import igzip as gzip
+except ImportError:
     import gzip
 import struct
 

telethon/tl/custom/button.py

@@ -37,11 +37,14 @@ class Button:
     to 128 characters and add the ellipsis (…) character as
     the 129.
     """
-    def __init__(self, button, *, resize, single_use, selective):
+    def __init__(self, button, *, resize, single_use, selective,
+                 persistent, placeholder):
         self.button = button
         self.resize = resize
         self.single_use = single_use
         self.selective = selective
+        self.persistent = persistent
+        self.placeholder = placeholder
 
     @staticmethod
     def _is_inline(button):
@@ -49,12 +52,14 @@ class Button:
         Returns `True` if the button belongs to an inline keyboard.
         """
         return isinstance(button, (
+            types.KeyboardButtonCopy,
             types.KeyboardButtonBuy,
             types.KeyboardButtonCallback,
             types.KeyboardButtonGame,
             types.KeyboardButtonSwitchInline,
             types.KeyboardButtonUrl,
-            types.InputKeyboardButtonUrlAuth
+            types.InputKeyboardButtonUrlAuth,
+            types.KeyboardButtonWebView,
         ))
 
     @staticmethod
@@ -166,11 +171,15 @@ class Button:
         )
 
     @classmethod
-    def text(cls, text, *, resize=None, single_use=None, selective=None):
+    def text(cls, text, *, resize=None, single_use=None, selective=None,
+             persistent=None, placeholder=None):
         """
         Creates a new keyboard button with the given text.
 
         Args:
+            text (`str`):
+                The title of the button.
+
             resize (`bool`):
                 If present, the entire keyboard will be reconfigured to
                 be resized and be smaller if there are not many buttons.
@@ -185,48 +194,77 @@ class Button:
                 users. It will target users that are @mentioned in the text
                 of the message or to the sender of the message you reply to.
 
+            persistent (`bool`):
+                If present, always show the keyboard when the regular keyboard
+                is hidden. Defaults to false, in which case the custom keyboard
+                can be hidden and revealed via the keyboard icon.
+
+            placeholder (`str`):
+                The placeholder to be shown in the input field when the keyboard is active;
+                1-64 characters
+
         When the user clicks this button, a text message with the same text
         as the button will be sent, and can be handled with `events.NewMessage
         <telethon.events.newmessage.NewMessage>`. You cannot distinguish
         between a button press and the user typing and sending exactly the
         same text on their own.
         """
-        return cls(types.KeyboardButton(text),
+        return cls(
-                   resize=resize, single_use=single_use, selective=selective)
+            types.KeyboardButton(text),
+            resize=resize,
+            single_use=single_use,
+            selective=selective,
+            persistent=persistent,
+            placeholder=placeholder
+        )
 
     @classmethod
-    def request_location(cls, text, *,
+    def request_location(cls, text, *, resize=None, single_use=None, selective=None,
-                         resize=None, single_use=None, selective=None):
+                         persistent=None, placeholder=None):
         """
         Creates a new keyboard button to request the user's location on click.
 
-        ``resize``, ``single_use`` and ``selective`` are documented in `text`.
+        ``resize``, ``single_use``, ``selective``, ``persistent`` and ``placeholder``
+         are documented in `text`.
 
         When the user clicks this button, a confirmation box will be shown
         to the user asking whether they want to share their location with the
         bot, and if confirmed a message with geo media will be sent.
         """
-        return cls(types.KeyboardButtonRequestGeoLocation(text),
+        return cls(
-                   resize=resize, single_use=single_use, selective=selective)
+            types.KeyboardButtonRequestGeoLocation(text),
+            resize=resize,
+            single_use=single_use,
+            selective=selective,
+            persistent=persistent,
+            placeholder=placeholder
+        )
 
     @classmethod
-    def request_phone(cls, text, *,
+    def request_phone(cls, text, *, resize=None, single_use=None,
-                      resize=None, single_use=None, selective=None):
+                      selective=None, persistent=None, placeholder=None):
         """
         Creates a new keyboard button to request the user's phone on click.
 
-        ``resize``, ``single_use`` and ``selective`` are documented in `text`.
+        ``resize``, ``single_use``, ``selective``, ``persistent`` and ``placeholder``
+         are documented in `text`.
 
         When the user clicks this button, a confirmation box will be shown
         to the user asking whether they want to share their phone with the
         bot, and if confirmed a message with contact media will be sent.
         """
-        return cls(types.KeyboardButtonRequestPhone(text),
+        return cls(
-                   resize=resize, single_use=single_use, selective=selective)
+            types.KeyboardButtonRequestPhone(text),
+            resize=resize,
+            single_use=single_use,
+            selective=selective,
+            placeholder=placeholder,
+            persistent=persistent
+        )
 
     @classmethod
-    def request_poll(cls, text, *, force_quiz=False,
+    def request_poll(cls, text, *, force_quiz=False, resize=None, single_use=None,
-                     resize=None, single_use=None, selective=None):
+                     selective=None, persistent=None, placeholder=None):
         """
         Creates a new keyboard button to request the user to create a poll.
 
@@ -238,13 +276,20 @@ class Button:
         the votes cannot be retracted. Otherwise, users can vote and retract
         the vote, and the pol might be multiple choice.
 
-        ``resize``, ``single_use`` and ``selective`` are documented in `text`.
+        ``resize``, ``single_use``, ``selective``, ``persistent`` and ``placeholder``
+         are documented in `text`.
 
         When the user clicks this button, a screen letting the user create a
         poll will be shown, and if they do create one, the poll will be sent.
         """
-        return cls(types.KeyboardButtonRequestPoll(text, quiz=force_quiz),
+        return cls(
-                   resize=resize, single_use=single_use, selective=selective)
+            types.KeyboardButtonRequestPoll(text, quiz=force_quiz),
+            resize=resize,
+            single_use=single_use,
+            selective=selective,
+            persistent=persistent,
+            placeholder=placeholder
+        )
 
     @staticmethod
     def clear(selective=None):
@@ -263,15 +308,8 @@ class Button:
         Forces a reply to the message with this markup. If used,
         no other button should be present or it will be ignored.
 
-        ``single_use`` and ``selective`` are as documented in `text`.
+        ``single_use``, ``selective`` and ``placeholder`` are as documented in `text`.
 
-        Args:
-            placeholder (str):
-                text to show the user at typing place of message.
-
-                If the placeholder is too long, Telegram applications will
-                crop the text (for example, to 64 characters and adding an
-                ellipsis (…) character as the 65th).
         """
         return types.ReplyKeyboardForceReply(
             single_use=single_use,

telethon/tl/custom/chatgetter.py

@@ -66,8 +66,9 @@ class ChatGetter(abc.ABC):
         """
         if self._input_chat is None and self._chat_peer and self._client:
             try:
-                self._input_chat = self._client._entity_cache[self._chat_peer]
+                self._input_chat = self._client._mb_entity_cache.get(
-            except KeyError:
+                        utils.get_peer_id(self._chat_peer, add_mark=False))._as_input_peer()
+            except AttributeError:
                 pass
 
         return self._input_chat

telethon/tl/custom/draft.py

@@ -1,11 +1,11 @@
 import datetime
 
-from .. import TLObject
+from .. import TLObject, types
 from ..functions.messages import SaveDraftRequest
 from ..types import DraftMessage
 from ...errors import RPCError
 from ...extensions import markdown
-from ...utils import get_input_peer, get_peer
+from ...utils import get_input_peer, get_peer, get_peer_id
 
 
 class Draft:
@@ -37,7 +37,7 @@ class Draft:
         self._raw_text = draft.message
         self.date = draft.date
         self.link_preview = not draft.no_webpage
-        self.reply_to_msg_id = draft.reply_to_msg_id
+        self.reply_to_msg_id = draft.reply_to.reply_to_msg_id if isinstance(draft.reply_to, types.InputReplyToMessage) else None
 
     @property
     def entity(self):
@@ -53,8 +53,9 @@ class Draft:
         """
         if not self._input_entity:
             try:
-                self._input_entity = self._client._entity_cache[self._peer]
+                self._input_entity = self._client._mb_entity_cache.get(
-            except KeyError:
+                        get_peer_id(self._peer, add_mark=False))._as_input_peer()
+            except AttributeError:
                 pass
 
         return self._input_entity
@@ -138,7 +139,7 @@ class Draft:
             peer=self._peer,
             message=raw_text,
             no_webpage=not link_preview,
-            reply_to_msg_id=reply_to,
+            reply_to=None if reply_to is None else types.InputReplyToMessage(reply_to),
             entities=entities
         ))
 

telethon/tl/custom/file.py

@@ -21,7 +21,12 @@ class File:
     @property
     def id(self):
         """
-        The bot-API style ``file_id`` representing this file.
+        The old bot-API style ``file_id`` representing this file.
+
+        .. warning::
+
+            This feature has not been maintained for a long time and
+            may not work. It will be removed in future versions.
 
         .. note::
 

telethon/tl/custom/forward.py

@@ -36,12 +36,12 @@ class Forward(ChatGetter, SenderGetter):
             if ty == helpers._EntityType.USER:
                 sender_id = utils.get_peer_id(original.from_id)
                 sender, input_sender = utils._get_entity_pair(
-                    sender_id, entities, client._entity_cache)
+                    sender_id, entities, client._mb_entity_cache)
 
             elif ty in (helpers._EntityType.CHAT, helpers._EntityType.CHANNEL):
                 peer = original.from_id
                 chat, input_chat = utils._get_entity_pair(
-                    utils.get_peer_id(peer), entities, client._entity_cache)
+                    utils.get_peer_id(peer), entities, client._mb_entity_cache)
 
         # This call resets the client
         ChatGetter.__init__(self, peer, chat=chat, input_chat=input_chat)

telethon/tl/custom/inlinebuilder.py

@@ -391,7 +391,7 @@ class InlineBuilder:
                     'text geo contact game'.split(), args) if x[1]) or 'none')
             )
 
-        markup = self._client.build_reply_markup(buttons, inline_only=True)
+        markup = self._client.build_reply_markup(buttons)
         if text is not None:
             text, msg_entities = await self._client._parse_message_text(
                 text, parse_mode

telethon/tl/custom/inlineresult.py

@@ -158,7 +158,7 @@ class InlineResult:
             background=background,
             clear_draft=clear_draft,
             hide_via=hide_via,
-            reply_to_msg_id=reply_id
+            reply_to=None if reply_id is None else types.InputReplyToMessage(reply_id)
         )
         return self._client._get_response_message(
             req, await self._client(req), entity)

telethon/tl/custom/message.py

@@ -68,6 +68,12 @@ class Message(ChatGetter, SenderGetter, TLObject):
         noforwards (`bool`):
             Whether this message can be forwarded or not.
 
+        invert_media (`bool`):
+            Whether the media in this message should be inverted.
+            
+        offline (`bool`):
+            Whether the message was sent by an implicit action, for example, as an away or a greeting business message, or as a scheduled message.
+
         id (`int`):
             The ID of this message. This field is *always* present.
             Any other member is optional and may be `None`.
@@ -90,7 +96,7 @@ class Message(ChatGetter, SenderGetter, TLObject):
             The ID of the bot used to send this message
             through its inline mode (e.g. "via @like").
 
-        reply_to (:tl:`MessageReplyHeader`):
+        reply_to (:tl:`MessageReplyHeader` | :tl:`MessageReplyStoryHeader`):
             The original reply header if this message is replying to another.
 
         date (`datetime`):
@@ -160,55 +166,68 @@ class Message(ChatGetter, SenderGetter, TLObject):
         action (:tl:`MessageAction`):
             The message action object of the message for :tl:`MessageService`
             instances, which will be `None` for other types of messages.
+
+        saved_peer_id (:tl:`Peer`)
     """
 
     # region Initialization
 
     def __init__(
-            # Common to all
+        self,
-            self, id: int,
+        id: int,
-
+        peer_id: types.TypePeer,
-            # Common to Message and MessageService (mandatory)
-            peer_id: types.TypePeer = None,
         date: Optional[datetime] = None,
-
+        message: Optional[str] = None,
-            # Common to Message and MessageService (flags)
+        # Copied from Message.__init__ signature
         out: Optional[bool] = None,
         mentioned: Optional[bool] = None,
         media_unread: Optional[bool] = None,
         silent: Optional[bool] = None,
         post: Optional[bool] = None,
-            from_id: Optional[types.TypePeer] = None,
-            reply_to: Optional[types.TypeMessageReplyHeader] = None,
-            ttl_period: Optional[int] = None,
-
-            # For Message (mandatory)
-            message: Optional[str] = None,
-
-            # For Message (flags)
-            fwd_from: Optional[types.TypeMessageFwdHeader] = None,
-            via_bot_id: Optional[int] = None,
-            media: Optional[types.TypeMessageMedia] = None,
-            reply_markup: Optional[types.TypeReplyMarkup] = None,
-            entities: Optional[List[types.TypeMessageEntity]] = None,
-            views: Optional[int] = None,
-            edit_date: Optional[datetime] = None,
-            post_author: Optional[str] = None,
-            grouped_id: Optional[int] = None,
         from_scheduled: Optional[bool] = None,
         legacy: Optional[bool] = None,
         edit_hide: Optional[bool] = None,
         pinned: Optional[bool] = None,
         noforwards: Optional[bool] = None,
-            reactions: Optional[types.TypeMessageReactions] = None,
+        invert_media: Optional[bool] = None,
-            restriction_reason: Optional[types.TypeRestrictionReason] = None,
+        offline: Optional[bool] = None,
+        video_processing_pending: Optional[bool] = None,
+        paid_suggested_post_stars: Optional[bool] = None,
+        paid_suggested_post_ton: Optional[bool] = None,
+        from_id: Optional[types.TypePeer] = None,
+        from_boosts_applied: Optional[int] = None,
+        saved_peer_id: Optional[types.TypePeer] = None,
+        fwd_from: Optional[types.TypeMessageFwdHeader] = None,
+        via_bot_id: Optional[int] = None,
+        via_business_bot_id: Optional[int] = None,
+        reply_to: Optional[types.TypeMessageReplyHeader] = None,
+        media: Optional[types.TypeMessageMedia] = None,
+        reply_markup: Optional[types.TypeReplyMarkup] = None,
+        entities: Optional[List[types.TypeMessageEntity]] = None,
+        views: Optional[int] = None,
         forwards: Optional[int] = None,
         replies: Optional[types.TypeMessageReplies] = None,
-
+        edit_date: Optional[datetime] = None,
-            # For MessageAction (mandatory)
+        post_author: Optional[str] = None,
-            action: Optional[types.TypeMessageAction] = None
+        grouped_id: Optional[int] = None,
+        reactions: Optional[types.TypeMessageReactions] = None,
+        restriction_reason: Optional[List[types.TypeRestrictionReason]] = None,
+        ttl_period: Optional[int] = None,
+        quick_reply_shortcut_id: Optional[int] = None,
+        effect: Optional[int] = None,
+        factcheck: Optional[types.TypeFactCheck] = None,
+        report_delivery_until_date: Optional[datetime] = None,
+        paid_message_stars: Optional[int] = None,
+        suggested_post: Optional[types.TypeSuggestedPost] = None,
+        # Copied from MessageService.__init__ signature
+        action: Optional[types.TypeMessageAction] = None,
+        reactions_are_possible: Optional[bool] = None,
     ):
-        # Common properties to messages, then to service (in the order they're defined in the `.tl`)
+        # Copied from Message.__init__ body
+        self.id = id
+        self.peer_id = peer_id
+        self.date = date
+        self.message = message
         self.out = bool(out)
         self.mentioned = mentioned
         self.media_unread = media_unread
@@ -217,14 +236,20 @@ class Message(ChatGetter, SenderGetter, TLObject):
         self.from_scheduled = from_scheduled
         self.legacy = legacy
         self.edit_hide = edit_hide
-        self.id = id
+        self.pinned = pinned
+        self.noforwards = noforwards
+        self.invert_media = invert_media
+        self.offline = offline
+        self.video_processing_pending = video_processing_pending
+        self.paid_suggested_post_stars = paid_suggested_post_stars
+        self.paid_suggested_post_ton = paid_suggested_post_ton
         self.from_id = from_id
-        self.peer_id = peer_id
+        self.from_boosts_applied = from_boosts_applied
+        self.saved_peer_id = saved_peer_id
         self.fwd_from = fwd_from
         self.via_bot_id = via_bot_id
+        self.via_business_bot_id = via_business_bot_id
         self.reply_to = reply_to
-        self.date = date
-        self.message = message
         self.media = None if isinstance(media, types.MessageMediaEmpty) else media
         self.reply_markup = reply_markup
         self.entities = entities
@@ -232,14 +257,20 @@ class Message(ChatGetter, SenderGetter, TLObject):
         self.forwards = forwards
         self.replies = replies
         self.edit_date = edit_date
-        self.pinned = pinned
-        self.noforwards = noforwards
         self.post_author = post_author
         self.grouped_id = grouped_id
         self.reactions = reactions
         self.restriction_reason = restriction_reason
         self.ttl_period = ttl_period
+        self.quick_reply_shortcut_id = quick_reply_shortcut_id
+        self.effect = effect
+        self.factcheck = factcheck
+        self.report_delivery_until_date = report_delivery_until_date
+        self.paid_message_stars = paid_message_stars
+        self.suggested_post = suggested_post
+        # Copied from MessageService.__init__ body
         self.action = action
+        self.reactions_are_possible = reactions_are_possible
 
         # Convenient storage for custom functions
         # TODO This is becoming a bit of bloat
@@ -271,6 +302,8 @@ class Message(ChatGetter, SenderGetter, TLObject):
         SenderGetter.__init__(self, sender_id)
 
         self._forward = None
+        self._reply_to_chat = None
+        self._reply_to_sender = None
 
     def _finish_init(self, client, entities, input_chat):
         """
@@ -285,7 +318,7 @@ class Message(ChatGetter, SenderGetter, TLObject):
         if self.peer_id == types.PeerUser(client._self_id) and not self.fwd_from:
             self.out = True
 
-        cache = client._entity_cache
+        cache = client._mb_entity_cache
 
         self._sender, self._input_sender = utils._get_entity_pair(
             self.sender_id, entities, cache)
@@ -324,6 +357,14 @@ class Message(ChatGetter, SenderGetter, TLObject):
             self._linked_chat = entities.get(utils.get_peer_id(
                     types.PeerChannel(self.replies.channel_id)))
         
+        if isinstance(self.reply_to, types.MessageReplyHeader):
+            if self.reply_to.reply_to_peer_id:
+                self._reply_to_chat = entities.get(utils.get_peer_id(self.reply_to.reply_to_peer_id))
+            if self.reply_to.reply_from:
+                if self.reply_to.reply_from.from_id:
+                    self._reply_to_sender = entities.get(utils.get_peer_id(self.reply_to.reply_from.from_id))
+
+
 
     # endregion Initialization
 
@@ -383,10 +424,11 @@ class Message(ChatGetter, SenderGetter, TLObject):
     @property
     def is_reply(self):
         """
-        `True` if the message is a reply to some other message.
+        `True` if the message is a reply to some other message or story.
 
-        Remember that you can access the ID of the message
+        Remember that if the replied-to is a message, 
-        this one is replying to through `reply_to.reply_to_msg_id`,
+        you can access the ID of the message this one is 
+        replying to through `reply_to.reply_to_msg_id`,
         and the `Message` object with `get_reply_message()`.
         """
         return self.reply_to is not None
@@ -399,6 +441,22 @@ class Message(ChatGetter, SenderGetter, TLObject):
         """
         return self._forward
 
+    @property
+    def reply_to_chat(self):
+        """
+        The :tl:`Channel` in which the replied-to message was sent,
+        if this message is a reply in another chat
+        """
+        return self._reply_to_chat
+
+    @property
+    def reply_to_sender(self):
+        """
+        The :tl:`User`, :tl:`Channel`, or whatever other entity that
+        sent the replied-to message, if this message is a reply in another chat.
+        """
+        return self._reply_to_sender
+
     @property
     def buttons(self):
         """
@@ -659,7 +717,11 @@ class Message(ChatGetter, SenderGetter, TLObject):
         Returns the message ID this message is replying to, if any.
         This is equivalent to accessing ``.reply_to.reply_to_msg_id``.
         """
-        return self.reply_to.reply_to_msg_id if self.reply_to else None
+        return (
+            self.reply_to.reply_to_msg_id
+            if isinstance(self.reply_to, types.MessageReplyHeader)
+            else None
+        )
 
     @property
     def to_id(self):
@@ -725,7 +787,7 @@ class Message(ChatGetter, SenderGetter, TLObject):
         The result will be cached after its first use.
         """
         if self._reply_message is None and self._client:
-            if not self.reply_to:
+            if not isinstance(self.reply_to, types.MessageReplyHeader):
                 return None
 
             # Bots cannot access other bots' messages by their ID.
@@ -784,12 +846,26 @@ class Message(ChatGetter, SenderGetter, TLObject):
 
     async def edit(self, *args, **kwargs):
         """
-        Edits the message iff it's outgoing. Shorthand for
+        Edits the message if it's outgoing. Shorthand for
         `telethon.client.messages.MessageMethods.edit_message`
         with both ``entity`` and ``message`` already set.
 
-        Returns `None` if the message was incoming,
+        Returns
-        or the edited `Message` otherwise.
+            The edited `Message <telethon.tl.custom.message.Message>`,
+            unless `entity` was a :tl:`InputBotInlineMessageID` or :tl:`InputBotInlineMessageID64` in which
+            case this method returns a boolean.
+
+        Raises
+            ``MessageAuthorRequiredError`` if you're not the author of the
+            message but tried editing it anyway.
+
+            ``MessageNotModifiedError`` if the contents of the message were
+            not modified at all.
+
+            ``MessageIdInvalidError`` if the ID of the message is invalid
+            (the ID itself may be correct, but the message with that ID
+            cannot be edited). For example, when trying to edit messages
+            with a reply markup (or clear markup) this error will be raised.
 
         .. note::
 
@@ -803,9 +879,6 @@ class Message(ChatGetter, SenderGetter, TLObject):
             This is generally the most desired and convenient behaviour,
             and will work for link previews and message buttons.
         """
-        if self.fwd_from or not self.out or not self._client:
-            return None  # We assume self.out was patched for our chat
-
         if 'link_preview' not in kwargs:
             kwargs['link_preview'] = bool(self.web_preview)
 
@@ -848,7 +921,7 @@ class Message(ChatGetter, SenderGetter, TLObject):
 
     async def click(self, i=None, j=None,
                     *, text=None, filter=None, data=None, share_phone=None,
-                    share_geo=None, password=None):
+                    share_geo=None, password=None, open_url=None):
         """
         Calls :tl:`SendVote` with the specified poll option
         or `button.click <telethon.tl.custom.messagebutton.MessageButton.click>`
@@ -933,6 +1006,12 @@ class Message(ChatGetter, SenderGetter, TLObject):
                 you need to provide your account's password. Otherwise,
                 `teltehon.errors.PasswordHashInvalidError` is raised.
             
+            open_url (`bool`):
+                When clicking on an inline keyboard URL button :tl:`KeyboardButtonUrl`
+                By default it will return URL of the button, passing ``click(open_url=True)``
+                will lunch the default browser with given URL of the button and 
+                return `True` on success.
+                
             Example:
 
                 .. code-block:: python
@@ -962,7 +1041,7 @@ class Message(ChatGetter, SenderGetter, TLObject):
 
             but = types.KeyboardButtonCallback('', data)
             return await MessageButton(self._client, but, chat, None, self.id).click(
-                share_phone=share_phone, share_geo=share_geo, password=password)
+                share_phone=share_phone, share_geo=share_geo, password=password, open_url=open_url)
 
         if sum(int(x is not None) for x in (i, text, filter)) >= 2:
             raise ValueError('You can only set either of i, text or filter')
@@ -1035,7 +1114,7 @@ class Message(ChatGetter, SenderGetter, TLObject):
         button = find_button()
         if button:
             return await button.click(
-                share_phone=share_phone, share_geo=share_geo, password=password)
+                share_phone=share_phone, share_geo=share_geo, password=password, open_url=open_url)
 
     async def mark_read(self):
         """
@@ -1138,8 +1217,9 @@ class Message(ChatGetter, SenderGetter, TLObject):
                         return bot
                     else:
                         try:
-                            return self._client._entity_cache[self.via_bot_id]
+                            return self._client._mb_entity_cache.get(
-                        except KeyError:
+                                utils.resolve_id(self.via_bot_id)[0])._as_input_peer()
+                        except AttributeError:
                             raise ValueError('No input sender') from None
 
     def _document_by_attribute(self, kind, condition=None):

telethon/tl/custom/messagebutton.py

@@ -1,7 +1,11 @@
 from .. import types, functions
 from ... import password as pwd_mod
 from ...errors import BotResponseTimeoutError
+try:
     import webbrowser
+except ModuleNotFoundError:
+    pass
+import sys
 import os
 
 
@@ -61,7 +65,7 @@ class MessageButton:
         if isinstance(self.button, types.KeyboardButtonUrl):
             return self.button.url
 
-    async def click(self, share_phone=None, share_geo=None, *, password=None):
+    async def click(self, share_phone=None, share_geo=None, *, password=None, open_url=None):
         """
         Emulates the behaviour of clicking this button.
 
@@ -75,7 +79,8 @@ class MessageButton:
         :tl:`StartBotRequest` will be invoked and the resulting updates
         returned.
 
-        If it's a :tl:`KeyboardButtonUrl`, the URL of the button will
+        If it's a :tl:`KeyboardButtonUrl`, the ``URL`` of the button will
+        be returned. If you pass ``open_url=True`` the URL of the button will
         be passed to ``webbrowser.open`` and return `True` on success.
 
         If it's a :tl:`KeyboardButtonRequestPhone`, you must indicate that you
@@ -112,7 +117,10 @@ class MessageButton:
                 bot=self._bot, peer=self._chat, start_param=self.button.query
             ))
         elif isinstance(self.button, types.KeyboardButtonUrl):
+            if open_url:
+                if "webbrowser" in sys.modules:
                     return webbrowser.open(self.button.url)
+            return self.button.url
         elif isinstance(self.button, types.KeyboardButtonGame):
             req = functions.messages.GetBotCallbackAnswerRequest(
                 peer=self._chat, msg_id=self._msg_id, game=True

telethon/tl/custom/sendergetter.py

@@ -1,5 +1,7 @@
 import abc
 
+from ... import utils
+
 
 class SenderGetter(abc.ABC):
     """
@@ -46,6 +48,9 @@ class SenderGetter(abc.ABC):
         # cached information, they may use the property instead.
         if (self._sender is None or getattr(self._sender, 'min', None)) \
                 and await self.get_input_sender():
+            # self.get_input_sender may refresh in which case the sender may no longer be min
+            # However it could still incur a cost so the cheap check is done twice instead.
+            if self._sender is None or getattr(self._sender, 'min', None):
                 try:
                     self._sender =\
                         await self._client.get_entity(self._input_sender)
@@ -66,9 +71,9 @@ class SenderGetter(abc.ABC):
         """
         if self._input_sender is None and self._sender_id and self._client:
             try:
-                self._input_sender = \
+                self._input_sender = self._client._mb_entity_cache.get(
-                    self._client._entity_cache[self._sender_id]
+                        utils.resolve_id(self._sender_id)[0])._as_input_peer()
-            except KeyError:
+            except AttributeError:
                 pass
         return self._input_sender
 

telethon/utils.py

@@ -4,7 +4,6 @@ to convert between an entity like a User, Chat, etc. into its Input version)
 """
 import base64
 import binascii
-import imghdr
 import inspect
 import io
 import itertools
@@ -15,6 +14,7 @@ import os
 import pathlib
 import re
 import struct
+import warnings
 from collections import namedtuple
 from mimetypes import guess_extension
 from types import GeneratorType
@@ -60,14 +60,8 @@ TG_JOIN_RE = re.compile(
     r'tg://(join)\?invite='
 )
 
-# The only shorter-than-five-characters usernames are those used for some
-# special, very well known bots. This list may be incomplete though:
-#    "[...] @gif, @vid, @pic, @bing, @wiki, @imdb and @bold [...]"
-#
-# See https://telegram.org/blog/inline-bots#how-does-it-work
 VALID_USERNAME_RE = re.compile(
-    r'^([a-z](?:(?!__)\w){3,30}[a-z\d]'
+    r'^[a-z](?:(?!__)\w){1,30}[a-z\d]$',
-    r'|gif|vid|pic|bing|wiki|imdb|bold|vote|like|coub)$',
     re.IGNORECASE
 )
 
@@ -102,7 +96,8 @@ def get_display_name(entity):
         else:
             return ''
 
-    elif isinstance(entity, (types.Chat, types.ChatForbidden, types.Channel)):
+    elif isinstance(entity, (
+            types.Chat, types.ChatForbidden, types.Channel, types.ChannelForbidden)):
         return entity.title
 
     return ''
@@ -443,15 +438,16 @@ def get_input_media(
         if media.SUBCLASS_OF_ID == 0xfaf846f4:  # crc32(b'InputMedia')
             return media
         elif media.SUBCLASS_OF_ID == 0x846363e0:  # crc32(b'InputPhoto')
-            return types.InputMediaPhoto(media, ttl_seconds=ttl)
+            return types.InputMediaPhoto(media, ttl_seconds=ttl, spoiler=media.spoiler)
         elif media.SUBCLASS_OF_ID == 0xf33fdb68:  # crc32(b'InputDocument')
-            return types.InputMediaDocument(media, ttl_seconds=ttl)
+            return types.InputMediaDocument(media, ttl_seconds=ttl, spoiler=media.spoiler)
     except AttributeError:
         _raise_cast_fail(media, 'InputMedia')
 
     if isinstance(media, types.MessageMediaPhoto):
         return types.InputMediaPhoto(
             id=get_input_photo(media.photo),
+            spoiler=media.spoiler,
             ttl_seconds=ttl or media.ttl_seconds
         )
 
@@ -506,6 +502,14 @@ def get_input_media(
     if isinstance(media, types.MessageMediaGeo):
         return types.InputMediaGeoPoint(geo_point=get_input_geo(media.geo))
 
+    if isinstance(media, types.MessageMediaGeoLive):
+        return types.InputMediaGeoLive(
+            geo_point=get_input_geo(media.geo),
+            period=media.period,
+            heading=media.heading,
+            proximity_notification_radius=media.proximity_notification_radius,
+        )
+
     if isinstance(media, types.MessageMediaVenue):
         return types.InputMediaVenue(
             geo_point=get_input_geo(media.geo),
@@ -583,11 +587,14 @@ def _get_entity_pair(entity_id, entities, cache,
     """
     Returns ``(entity, input_entity)`` for the given entity ID.
     """
+    if not entity_id:
+        return None, None
+
     entity = entities.get(entity_id)
     try:
-        input_entity = cache[entity_id]
+        input_entity = cache.get(resolve_id(entity_id)[0])._as_input_peer()
-    except KeyError:
+    except AttributeError:
-        # KeyError is unlikely, so another TypeError won't hurt
+        # AttributeError is unlikely, so another TypeError won't hurt
         try:
             input_entity = get_input_peer(entity)
         except TypeError:
@@ -604,6 +611,9 @@ def get_message_id(message):
     if isinstance(message, int):
         return message
 
+    if isinstance(message, types.InputMessageID):
+        return message.id
+
     try:
         if message.SUBCLASS_OF_ID == 0x790009e3:
             # hex(crc32(b'Message')) = 0x790009e3
@@ -760,7 +770,10 @@ def sanitize_parse_mode(mode):
     if not mode:
         return None
 
-    if callable(mode):
+    if (all(hasattr(mode, x) for x in ('parse', 'unparse'))
+          and all(callable(x) for x in (mode.parse, mode.unparse))):
+        return mode
+    elif callable(mode):
         class CustomMode:
             @staticmethod
             def unparse(text, entities):
@@ -768,9 +781,6 @@ def sanitize_parse_mode(mode):
 
         CustomMode.parse = mode
         return CustomMode
-    elif (all(hasattr(mode, x) for x in ('parse', 'unparse'))
-          and all(callable(x) for x in (mode.parse, mode.unparse))):
-        return mode
     elif isinstance(mode, str):
         try:
             return {
@@ -838,12 +848,6 @@ def _get_extension(file):
         return os.path.splitext(file)[-1]
     elif isinstance(file, pathlib.Path):
         return file.suffix
-    elif isinstance(file, bytes):
-        kind = imghdr.what(io.BytesIO(file))
-        return ('.' + kind) if kind else ''
-    elif isinstance(file, io.IOBase) and not isinstance(file, io.TextIOBase) and file.seekable():
-        kind = imghdr.what(file)
-        return ('.' + kind) if kind is not None else ''
     elif getattr(file, 'name', None):
         # Note: ``file.name`` works for :tl:`InputFile` and some `IOBase`
         return _get_extension(file.name)
@@ -906,7 +910,7 @@ def is_list_like(obj):
     enough. Things like ``open()`` are also iterable (and probably many
     other things), so just support the commonly known list-like objects.
     """
-    return isinstance(obj, (list, tuple, set, dict, GeneratorType))
+    return isinstance(obj, (list, tuple, set, dict, range, GeneratorType))
 
 
 def parse_phone(phone):
@@ -1554,3 +1558,11 @@ def _photo_size_byte_count(size):
         return max(size.sizes)
     else:
         return None
+
+
+async def maybe_async(coro):
+    result = coro
+    if inspect.isawaitable(result):
+        warnings.warn('Using async sessions support is an experimental feature')
+        result = await result
+    return result

telethon/version.py

@@ -1,3 +1,3 @@
 # Versions should comply with PEP440.
 # This line is parsed in setup.py:
-__version__ = '1.25.1'
+__version__ = '1.40.0'

telethon_examples/gui.py

@@ -53,7 +53,7 @@ def callback(func):
     def wrapped(*args, **kwargs):
         result = func(*args, **kwargs)
         if inspect.iscoroutine(result):
-            aio_loop.create_task(result)
+            asyncio.create_task(result)
 
     return wrapped
 
@@ -369,10 +369,4 @@ async def main(interval=0.05):
 
 
 if __name__ == "__main__":
-    # Some boilerplate code to set up the main method
+    asyncio.run(main())
-    aio_loop = asyncio.get_event_loop()
-    try:
-        aio_loop.run_until_complete(main())
-    finally:
-        if not aio_loop.is_closed():
-            aio_loop.close()

telethon_examples/interactive_telegram_client.py

@@ -9,9 +9,6 @@ from telethon.errors import SessionPasswordNeededError
 from telethon.network import ConnectionTcpAbridged
 from telethon.utils import get_display_name
 
-# Create a global variable to hold the loop we will be using
-loop = asyncio.get_event_loop()
-
 
 def sprint(string, *args, **kwargs):
     """Safe Print (handle UnicodeEncodeErrors on some terminals)"""
@@ -50,7 +47,7 @@ async def async_input(prompt):
     let the loop run while we wait for input.
     """
     print(prompt, end='', flush=True)
-    return (await loop.run_in_executor(None, sys.stdin.readline)).rstrip()
+    return (await asyncio.get_running_loop().run_in_executor(None, sys.stdin.readline)).rstrip()
 
 
 def get_env(name, message, cast=str):
@@ -109,34 +106,34 @@ class InteractiveTelegramClient(TelegramClient):
         # media known the message ID, for every message having media.
         self.found_media = {}
 
+    async def init(self):
         # Calling .connect() may raise a connection error False, so you need
         # to except those before continuing. Otherwise you may want to retry
         # as done here.
         print('Connecting to Telegram servers...')
         try:
-            loop.run_until_complete(self.connect())
+            await self.connect()
         except IOError:
             # We handle IOError and not ConnectionError because
             # PySocks' errors do not subclass ConnectionError
             # (so this will work with and without proxies).
             print('Initial connection failed. Retrying...')
-            loop.run_until_complete(self.connect())
+            await self.connect()
 
-        # If the user hasn't called .sign_in() or .sign_up() yet, they won't
+        # If the user hasn't called .sign_in() yet, they won't
         # be authorized. The first thing you must do is authorize. Calling
         # .sign_in() should only be done once as the information is saved on
         # the *.session file so you don't need to enter the code every time.
-        if not loop.run_until_complete(self.is_user_authorized()):
+        if not await self.is_user_authorized():
             print('First run. Sending code request...')
             user_phone = input('Enter your phone: ')
-            loop.run_until_complete(self.sign_in(user_phone))
+            await self.sign_in(user_phone)
 
             self_user = None
             while self_user is None:
                 code = input('Enter the code you just received: ')
                 try:
-                    self_user =\
+                    self_user = await self.sign_in(code=code)
-                        loop.run_until_complete(self.sign_in(code=code))
 
                 # Two-step verification may be enabled, and .sign_in will
                 # raise this error. If that's the case ask for the password.
@@ -146,8 +143,7 @@ class InteractiveTelegramClient(TelegramClient):
                     pw = getpass('Two step verification is enabled. '
                                  'Please enter your password: ')
 
-                    self_user =\
+                    self_user = await self.sign_in(password=pw)
-                        loop.run_until_complete(self.sign_in(password=pw))
 
     async def run(self):
         """Main loop of the TelegramClient, will wait for user action"""
@@ -397,9 +393,14 @@ class InteractiveTelegramClient(TelegramClient):
                 ))
 
 
-if __name__ == '__main__':
+async def main():
     SESSION = os.environ.get('TG_SESSION', 'interactive')
     API_ID = get_env('TG_API_ID', 'Enter your API ID: ', int)
     API_HASH = get_env('TG_API_HASH', 'Enter your API hash: ')
     client = InteractiveTelegramClient(SESSION, API_ID, API_HASH)
-    loop.run_until_complete(client.run())
+    await client.init()
+    await client.run()
+
+
+if __name__ == '__main__':
+    asyncio.run()

telethon_examples/payment.py

@@ -7,8 +7,6 @@ import os
 import time
 import sys
 
-loop = asyncio.get_event_loop()
-
 """
 Provider token can be obtained via @BotFather. more info at https://core.telegram.org/bots/payments#getting-a-token
 
@@ -85,9 +83,9 @@ async def payment_received_handler(event):
         payment: types.MessageActionPaymentSentMe = event.message.action
         # do something after payment was received
         if payment.payload.decode('UTF-8') == 'product A':
-            await bot.send_message(event.message.from_id, 'Thank you for buying product A!')
+            await bot.send_message(event.message.peer_id.user_id, 'Thank you for buying product A!')
         elif payment.payload.decode('UTF-8') == 'product B':
-            await bot.send_message(event.message.from_id, 'Thank you for buying product B!')
+            await bot.send_message(event.message.peer_id.user_id, 'Thank you for buying product B!')
         raise events.StopPropagation
 
 
@@ -180,4 +178,4 @@ if __name__ == '__main__':
     if not provider_token:
         logger.error("No provider token supplied.")
         exit(1)
-    loop.run_until_complete(main())
+    asyncio.run(main())

telethon_examples/quart_login.py

@@ -1,7 +1,6 @@
 import base64
 import os
 
-import hypercorn.asyncio
 from quart import Quart, render_template_string, request
 
 from telethon import TelegramClient, utils
@@ -82,6 +81,8 @@ async def format_message(message):
 # Connect the client before we start serving with Quart
 @app.before_serving
 async def startup():
+    # After connecting, the client will create additional asyncio tasks that run until it's disconnected again.
+    # Be careful to not mix different asyncio loops during a client's lifetime, or things won't work properly!
     await client.connect()
 
 
@@ -129,24 +130,11 @@ async def root():
     return await render_template_string(BASE_TEMPLATE, content=CODE_FORM)
 
 
-async def main():
-    await hypercorn.asyncio.serve(app, hypercorn.Config())
-
-
 # By default, `Quart.run` uses `asyncio.run()`, which creates a new asyncio
-# event loop. If we create the `TelegramClient` before, `telethon` will
+# event loop. If we had connected the `TelegramClient` before, `telethon` will
-# use `asyncio.get_event_loop()`, which is the implicit loop in the main
+# use `asyncio.get_running_loop()` to create some additional tasks. If these
-# thread. These two loops are different, and it won't work.
+# loops are different, it won't work.
 #
-# So, we have to manually pass the same `loop` to both applications to
+# To keep things simple, be sure to not create multiple asyncio loops!
-# make 100% sure it works and to avoid headaches.
-#
-# To run Quart inside `async def`, we must use `hypercorn.asyncio.serve()`
-# directly.
-#
-# This example creates a global client outside of Quart handlers.
-# If you create the client inside the handlers (common case), you
-# won't have to worry about any of this, but it's still good to be
-# explicit about the event loop.
 if __name__ == '__main__':
-    client.loop.run_until_complete(main())
+    app.run()

telethon_generator/data/errors.csv

@@ -5,14 +5,15 @@ ACCESS_TOKEN_EXPIRED,400,Bot token expired
 ACCESS_TOKEN_INVALID,400,The provided token is not valid
 ACTIVE_USER_REQUIRED,401,The method is only available to already activated users
 ADMINS_TOO_MUCH,400,Too many admins
+ADMIN_ID_INVALID,400,The specified admin ID is invalid
 ADMIN_RANK_EMOJI_NOT_ALLOWED,400,Emoji are not allowed in admin titles or ranks
 ADMIN_RANK_INVALID,400,The given admin title or rank was invalid (possibly larger than 16 characters)
 ALBUM_PHOTOS_TOO_MANY,400,Too many photos were included in the album
 API_ID_INVALID,400,The api_id/api_hash combination is invalid
 API_ID_PUBLISHED_FLOOD,400,"This API id was published somewhere, you can't use it now"
 ARTICLE_TITLE_EMPTY,400,The title of the article is empty
+AUDIO_CONTENT_URL_EMPTY,400,The remote URL specified in the content field is empty
 AUDIO_TITLE_EMPTY,400,The title attribute of the audio must be non-empty
-AUDIO_CONTENT_URL_EMPTY,400,
 AUTH_BYTES_INVALID,400,The provided authorization is invalid
 AUTH_KEY_DUPLICATED,406,"The authorization key (session file) was used under two different IP addresses simultaneously, and can no longer be used. Use the same session exclusively, or use different sessions"
 AUTH_KEY_INVALID,401,The key is invalid
@@ -20,18 +21,19 @@ AUTH_KEY_PERM_EMPTY,401,"The method is unavailable for temporary authorization k
 AUTH_KEY_UNREGISTERED,401,The key is not registered in the system
 AUTH_RESTART,500,Restart the authorization process
 AUTH_TOKEN_ALREADY_ACCEPTED,400,The authorization token was already used
+AUTH_TOKEN_EXCEPTION,400,An error occurred while importing the auth token
 AUTH_TOKEN_EXPIRED,400,The provided authorization token has expired and the updated QR-code must be re-scanned
 AUTH_TOKEN_INVALID,400,An invalid authorization token was provided
 AUTH_TOKEN_INVALID2,400,An invalid authorization token was provided
+AUTH_TOKEN_INVALIDX,400,The specified auth token is invalid
 AUTOARCHIVE_NOT_AVAILABLE,400,You cannot use this feature yet
 BANK_CARD_NUMBER_INVALID,400,Incorrect credit card number
-BASE_PORT_LOC_INVALID,400,Base port location invalid
 BANNED_RIGHTS_INVALID,400,"You cannot use that set of permissions in this request, i.e. restricting view_messages as a default"
+BASE_PORT_LOC_INVALID,400,Base port location invalid
 BOTS_TOO_MUCH,400,There are too many bots in this chat/channel
-BOT_ONESIDE_NOT_AVAIL,400,
 BOT_CHANNELS_NA,400,Bots can't edit admin privileges
 BOT_COMMAND_DESCRIPTION_INVALID,400,"The command description was empty, too long or had invalid characters used"
-BOT_COMMAND_INVALID,400,
+BOT_COMMAND_INVALID,400,The specified command is invalid
 BOT_DOMAIN_INVALID,400,The domain used for the auth button does not match the one configured in @BotFather
 BOT_GAMES_DISABLED,400,Bot games cannot be used in this type of chat
 BOT_GROUPS_BLOCKED,400,This bot can't be added to groups
@@ -39,73 +41,100 @@ BOT_INLINE_DISABLED,400,This bot can't be used in inline mode
 BOT_INVALID,400,This is not a valid bot
 BOT_METHOD_INVALID,400,The API access for bot users is restricted. The method you tried to invoke cannot be executed as a bot
 BOT_MISSING,400,This method can only be run by a bot
+BOT_ONESIDE_NOT_AVAIL,400,Bots can't pin messages in PM just for themselves
 BOT_PAYMENTS_DISABLED,400,This method can only be run by a bot
 BOT_POLLS_DISABLED,400,You cannot create polls under a bot account
 BOT_RESPONSE_TIMEOUT,400,The bot did not answer to the callback query in time
+BOT_SCORE_NOT_MODIFIED,400,The score wasn't modified
 BROADCAST_CALLS_DISABLED,400,
 BROADCAST_FORBIDDEN,403,The request cannot be used in broadcast channels
 BROADCAST_ID_INVALID,400,The channel is invalid
 BROADCAST_PUBLIC_VOTERS_FORBIDDEN,400,You cannot broadcast polls where the voters are public
 BROADCAST_REQUIRED,400,The request can only be used with a broadcast channel
 BUTTON_DATA_INVALID,400,The provided button data is invalid
+BUTTON_TEXT_INVALID,400,The specified button text is invalid
 BUTTON_TYPE_INVALID,400,The type of one of the buttons you provided is invalid
 BUTTON_URL_INVALID,400,Button URL invalid
+BUTTON_USER_PRIVACY_RESTRICTED,400,The privacy setting of the user specified in a [inputKeyboardButtonUserProfile](/constructor/inputKeyboardButtonUserProfile) button do not allow creating such a button
 CALL_ALREADY_ACCEPTED,400,The call was already accepted
 CALL_ALREADY_DECLINED,400,The call was already declined
 CALL_OCCUPY_FAILED,500,The call failed because the user is already making another call
 CALL_PEER_INVALID,400,The provided call peer object is invalid
 CALL_PROTOCOL_FLAGS_INVALID,400,Call protocol flags invalid
 CDN_METHOD_INVALID,400,This method cannot be invoked on a CDN server. Refer to https://core.telegram.org/cdn#schema for available methods
+CDN_UPLOAD_TIMEOUT,500,A server-side timeout occurred while reuploading the file to the CDN DC
+CHANNELS_ADMIN_LOCATED_TOO_MUCH,400,The user has reached the limit of public geogroups
 CHANNELS_ADMIN_PUBLIC_TOO_MUCH,400,"You're admin of too many public channels, make some channels private to change the username of this channel"
 CHANNELS_TOO_MUCH,400,You have joined too many channels/supergroups
 CHANNEL_BANNED,400,The channel is banned
+CHANNEL_FORUM_MISSING,400,
+CHANNEL_ID_INVALID,400,The specified supergroup ID is invalid
 CHANNEL_INVALID,400,"Invalid channel object. Make sure to pass the right types, for instance making sure that the request is designed for channels or otherwise look for a different one more suited"
-CHANNEL_PRIVATE,400,The channel specified is private and you lack permission to access it. Another reason may be that you were banned from it
+CHANNEL_PARICIPANT_MISSING,400,The current user is not in the channel
+CHANNEL_PRIVATE,400 406,The channel specified is private and you lack permission to access it. Another reason may be that you were banned from it
 CHANNEL_PUBLIC_GROUP_NA,403,channel/supergroup not available
-CHANNEL_TOO_LARGE,406,
+CHANNEL_TOO_BIG,400,
+CHANNEL_TOO_LARGE,400 406,Channel is too large to be deleted; this error is issued when trying to delete channels with more than 1000 members (subject to change)
 CHAT_ABOUT_NOT_MODIFIED,400,About text has not changed
 CHAT_ABOUT_TOO_LONG,400,Chat about too long
 CHAT_ADMIN_INVITE_REQUIRED,403,You do not have the rights to do this
-CHAT_ADMIN_REQUIRED,400,"Chat admin privileges are required to do that in the specified chat (for example, to send a message in a channel which is not yours), or invalid permissions used for the channel or group"
+CHAT_ADMIN_REQUIRED,400 403,"Chat admin privileges are required to do that in the specified chat (for example, to send a message in a channel which is not yours), or invalid permissions used for the channel or group"
+CHAT_DISCUSSION_UNALLOWED,400,
 CHAT_FORBIDDEN,403,You cannot write in this chat
+CHAT_FORWARDS_RESTRICTED,400 406,You can't forward messages from a protected chat
+CHAT_GET_FAILED,500,
+CHAT_GUEST_SEND_FORBIDDEN,403,"You join the discussion group before commenting, see [here](/api/discussion#requiring-users-to-join-the-group) for more info"
 CHAT_ID_EMPTY,400,The provided chat ID is empty
+CHAT_ID_GENERATE_FAILED,500,Failure while generating the chat ID
 CHAT_ID_INVALID,400,"Invalid object ID for a chat. Make sure to pass the right types, for instance making sure that the request is designed for chats (not channels/megagroups) or otherwise look for a different one more suited\nAn example working with a megagroup and AddChatUserRequest, it will fail because megagroups are channels. Use InviteToChannelRequest instead"
 CHAT_INVALID,400,The chat is invalid for this request
+CHAT_INVITE_PERMANENT,400,You can't set an expiration date on permanent invite links
 CHAT_LINK_EXISTS,400,The chat is linked to a channel and cannot be used in that request
 CHAT_NOT_MODIFIED,400,"The chat or channel wasn't modified (title, invites, username, admins, etc. are the same)"
 CHAT_RESTRICTED,400,The chat is restricted and cannot be used in that request
+CHAT_REVOKE_DATE_UNSUPPORTED,400,`min_date` and `max_date` are not available for using with non-user peers
+CHAT_SEND_GAME_FORBIDDEN,403,You can't send a game to this chat
 CHAT_SEND_GIFS_FORBIDDEN,403,You can't send gifs in this chat
-CHAT_SEND_INLINE_FORBIDDEN,400,You cannot send inline results in this chat
+CHAT_SEND_INLINE_FORBIDDEN,400 403,You cannot send inline results in this chat
 CHAT_SEND_MEDIA_FORBIDDEN,403,You can't send media in this chat
+CHAT_SEND_POLL_FORBIDDEN,403,You can't send polls in this chat
 CHAT_SEND_STICKERS_FORBIDDEN,403,You can't send stickers in this chat
 CHAT_TITLE_EMPTY,400,No chat title provided
-CHAT_TOO_BIG,400,
+CHAT_TOO_BIG,400,"This method is not available for groups with more than `chat_read_mark_size_threshold` members, [see client configuration](https://core.telegram.org/api/config#client-configuration)"
 CHAT_WRITE_FORBIDDEN,403,You can't write in this chat
 CHP_CALL_FAIL,500,The statistics cannot be retrieved at this time
 CODE_EMPTY,400,The provided code is empty
 CODE_HASH_INVALID,400,Code hash invalid
 CODE_INVALID,400,Code invalid (i.e. from email)
 CONNECTION_API_ID_INVALID,400,The provided API id is invalid
+CONNECTION_APP_VERSION_EMPTY,400,App version is empty
 CONNECTION_DEVICE_MODEL_EMPTY,400,Device model empty
 CONNECTION_LANG_PACK_INVALID,400,"The specified language pack is not valid. This is meant to be used by official applications only so far, leave it empty"
 CONNECTION_LAYER_INVALID,400,The very first request must always be InvokeWithLayerRequest
 CONNECTION_NOT_INITED,400,Connection not initialized
 CONNECTION_SYSTEM_EMPTY,400,Connection system empty
 CONNECTION_SYSTEM_LANG_CODE_EMPTY,400,The system language string was empty during connection
+CONTACT_ADD_MISSING,400,Contact to add is missing
 CONTACT_ID_INVALID,400,The provided contact ID is invalid
 CONTACT_NAME_EMPTY,400,The provided contact name cannot be empty
-CURRENCY_TOTAL_AMOUNT_INVALID,400,
+CONTACT_REQ_MISSING,400,Missing contact request
+CREATE_CALL_FAILED,400,An error occurred while creating the call
+CURRENCY_TOTAL_AMOUNT_INVALID,400,The total amount of all prices is invalid
 DATA_INVALID,400,Encrypted data invalid
 DATA_JSON_INVALID,400,The provided JSON data is invalid
+DATA_TOO_LONG,400,Data too long
 DATE_EMPTY,400,Date empty
 DC_ID_INVALID,400,This occurs when an authorization is tried to be exported for the same data center one is currently connected to
 DH_G_A_INVALID,400,g_a invalid
 DOCUMENT_INVALID,400,The document file was invalid and can't be used in inline mode
+EDIT_BOT_INVITE_FORBIDDEN,403,Normal users can't edit invites that were created by bots
 EMAIL_HASH_EXPIRED,400,The email hash expired and cannot be used to verify it
 EMAIL_INVALID,400,The given email is invalid
+EMAIL_UNCONFIRMED,400,Email unconfirmed
 EMAIL_UNCONFIRMED_X,400,"Email unconfirmed, the length of the code must be {code_length}"
-EMOJI_INVALID,400,
+EMAIL_VERIFY_EXPIRED,400,The verification email has expired
-EMOJI_NOT_MODIFIED,400,
+EMOJI_INVALID,400,The specified theme emoji is valid
+EMOJI_NOT_MODIFIED,400,The theme wasn't changed
 EMOTICON_EMPTY,400,The emoticon field cannot be empty
 EMOTICON_INVALID,400,The specified emoticon cannot be used or was not a emoticon
 EMOTICON_STICKERPACK_MISSING,400,The emoticon sticker pack you are trying to get is missing
@@ -119,13 +148,15 @@ ENTITIES_TOO_LONG,400,It is no longer possible to send such long data inside ent
 ENTITY_BOUNDS_INVALID,400,Some of provided entities have invalid bounds (length is zero or out of the boundaries of the string)
 ENTITY_MENTION_USER_INVALID,400,You can't use this entity
 ERROR_TEXT_EMPTY,400,The provided error message is empty
+EXPIRE_DATE_INVALID,400,The specified expiration date is invalid
 EXPIRE_FORBIDDEN,400,
 EXPORT_CARD_INVALID,400,Provided card is invalid
 EXTERNAL_URL_INVALID,400,External URL invalid
 FIELD_NAME_EMPTY,400,The field with the name FIELD_NAME is missing
 FIELD_NAME_INVALID,400,The field with the name FIELD_NAME is invalid
 FILEREF_UPGRADE_NEEDED,406,The file reference needs to be refreshed before being used again
-FILE_CONTENT_TYPE_INVALID,400,
+FILE_CONTENT_TYPE_INVALID,400,File content-type is invalid
+FILE_EMTPY,400,An empty file was provided
 FILE_ID_INVALID,400,"The provided file id is invalid. Make sure all parameters are present, have the correct type and are not empty (ID, access hash, file reference, thumb size ...)"
 FILE_MIGRATE_X,303,The file to be accessed is currently stored in DC {new_dc}
 FILE_PARTS_INVALID,400,The number of file parts is invalid
@@ -135,40 +166,51 @@ FILE_PART_INVALID,400,The file part number is invalid
 FILE_PART_LENGTH_INVALID,400,The length of a file part is invalid
 FILE_PART_SIZE_CHANGED,400,The file part size (chunk size) cannot change during upload
 FILE_PART_SIZE_INVALID,400,The provided file part size is invalid
+FILE_PART_TOO_BIG,400,The uploaded file part is too big
 FILE_PART_X_MISSING,400,Part {which} of the file is missing from storage
 FILE_REFERENCE_EMPTY,400,The file reference must exist to access the media and it cannot be empty
 FILE_REFERENCE_EXPIRED,400,The file reference has expired and is no longer valid or it belongs to self-destructing media and cannot be resent
 FILE_REFERENCE_INVALID,400,The file reference is invalid or you can't do that operation on such message
-FILE_TITLE_EMPTY,400,
+FILE_TITLE_EMPTY,400,An empty file title was specified
+FILTER_ID_INVALID,400,The specified filter ID is invalid
+FILTER_INCLUDE_EMPTY,400,The include_peers vector of the filter is empty
+FILTER_NOT_SUPPORTED,400,The specified filter cannot be used in this context
+FILTER_TITLE_EMPTY,400,The title field of the filter is empty
 FIRSTNAME_INVALID,400,The first name is invalid
 FLOOD_TEST_PHONE_WAIT_X,420,A wait of {seconds} seconds is required in the test servers
 FLOOD_WAIT_X,420,A wait of {seconds} seconds is required
+FLOOD_PREMIUM_WAIT_X,420,A wait of {seconds} seconds is required in non-premium accounts
 FOLDER_ID_EMPTY,400,The folder you tried to delete was already empty
 FOLDER_ID_INVALID,400,The folder you tried to use was not valid
-FRESH_CHANGE_ADMINS_FORBIDDEN,400,Recently logged-in users cannot add or change admins
+FRESH_CHANGE_ADMINS_FORBIDDEN,400 406,Recently logged-in users cannot add or change admins
 FRESH_CHANGE_PHONE_FORBIDDEN,406,Recently logged-in users cannot use this request
 FRESH_RESET_AUTHORISATION_FORBIDDEN,406,The current session is too new and cannot be used to reset other authorisations yet
+FROM_MESSAGE_BOT_DISABLED,400,Bots can't use fromMessage min constructors
 FROM_PEER_INVALID,400,The given from_user peer cannot be used for the parameter
 GAME_BOT_INVALID,400,You cannot send that game with the current bot
-GIF_CONTENT_TYPE_INVALID,400,
+GEO_POINT_INVALID,400,Invalid geoposition provided
+GIF_CONTENT_TYPE_INVALID,400,GIF content-type invalid
 GIF_ID_INVALID,400,The provided GIF ID is invalid
-GRAPH_INVALID_RELOAD,400,
+GRAPH_EXPIRED_RELOAD,400,"This graph has expired, please obtain a new graph token"
+GRAPH_INVALID_RELOAD,400,"Invalid graph token provided, please reload the stats and provide the updated token"
 GRAPH_OUTDATED_RELOAD,400,"Data can't be used for the channel statistics, graphs outdated"
 GROUPCALL_ADD_PARTICIPANTS_FAILED,500,
-GROUPCALL_ALREADY_DISCARDED,400,
+GROUPCALL_ALREADY_DISCARDED,400,The group call was already discarded
-GROUPCALL_FORBIDDEN,403,
+GROUPCALL_ALREADY_STARTED,403,"The groupcall has already started, you can join directly using [phone.joinGroupCall](https://core.telegram.org/method/phone.joinGroupCall)"
-GROUPCALL_JOIN_MISSING,400,
+GROUPCALL_FORBIDDEN,403,The group call has already ended
-GROUPCALL_SSRC_DUPLICATE_MUCH,400,
+GROUPCALL_INVALID,400,The specified group call is invalid
-GROUPCALL_NOT_MODIFIED,400,
+GROUPCALL_JOIN_MISSING,400,You haven't joined this group call
+GROUPCALL_NOT_MODIFIED,400,Group call settings weren't modified
+GROUPCALL_SSRC_DUPLICATE_MUCH,400,The app needs to retry joining the group call with a new SSRC value
 GROUPED_MEDIA_INVALID,400,Invalid grouped media
 GROUP_CALL_INVALID,400,Group call invalid
 HASH_INVALID,400,The provided hash is invalid
-HIDE_REQUESTER_MISSING,400,
+HIDE_REQUESTER_MISSING,400,The join request was missing or was already handled
 HISTORY_GET_FAILED,500,Fetching of history failed
 IMAGE_PROCESS_FAILED,400,Failure while processing image
 IMPORT_FILE_INVALID,400,The file is too large to be imported
 IMPORT_FORMAT_UNRECOGNIZED,400,Unknown import format
-IMPORT_ID_INVALID,400,
+IMPORT_ID_INVALID,400,The specified import ID is invalid
 INLINE_BOT_REQUIRED,403,The action must be performed through an inline bot callback
 INLINE_RESULT_EXPIRED,400,The inline query expired
 INPUT_CONSTRUCTOR_INVALID,400,The provided constructor is invalid
@@ -178,26 +220,32 @@ INPUT_FILTER_INVALID,400,The search query filter is invalid
 INPUT_LAYER_INVALID,400,The provided layer is invalid
 INPUT_METHOD_INVALID,400,The invoked method does not exist anymore or has never existed
 INPUT_REQUEST_TOO_LONG,400,The input request was too long. This may be a bug in the library as it can occur when serializing more bytes than it should (like appending the vector constructor code at the end of a message)
+INPUT_TEXT_EMPTY,400,The specified text is empty
 INPUT_USER_DEACTIVATED,400,The specified user was deleted
 INTERDC_X_CALL_ERROR,500,An error occurred while communicating with DC {dc}
 INTERDC_X_CALL_RICH_ERROR,500,A rich error occurred while communicating with DC {dc}
-INVITE_FORBIDDEN_WITH_JOINAS,400,
+INVITE_FORBIDDEN_WITH_JOINAS,400,"If the user has anonymously joined a group call as a channel, they can't invite other users to the group call because that would cause deanonymization, because the invite would be sent using the original user ID, not the anonymized channel ID"
 INVITE_HASH_EMPTY,400,The invite hash is empty
-INVITE_HASH_EXPIRED,400,The chat the user tried to join has expired and is not valid anymore
+INVITE_HASH_EXPIRED,400 406,The chat the user tried to join has expired and is not valid anymore
 INVITE_HASH_INVALID,400,The invite hash is invalid
 INVITE_REQUEST_SENT,400,You have successfully requested to join this chat or channel
-LANG_CODE_INVALID,400,
+INVITE_REVOKED_MISSING,400,The specified invite link was already revoked or is invalid
+INVOICE_PAYLOAD_INVALID,400,The specified invoice payload is invalid
+JOIN_AS_PEER_INVALID,400,The specified peer cannot be used to join a group call
+LANG_CODE_INVALID,400,The specified language code is invalid
+LANG_CODE_NOT_SUPPORTED,400,The specified language code is not supported
 LANG_PACK_INVALID,400,The provided language pack is invalid
 LASTNAME_INVALID,400,The last name is invalid
 LIMIT_INVALID,400,An invalid limit was provided. See https://core.telegram.org/api/files#downloading-files
 LINK_NOT_MODIFIED,400,The channel is already linked to this group
 LOCATION_INVALID,400,The location given for a file was invalid. See https://core.telegram.org/api/files#downloading-files
+MAX_DATE_INVALID,400,The specified maximum date is invalid
 MAX_ID_INVALID,400,The provided max ID is invalid
 MAX_QTS_INVALID,400,The provided QTS were invalid
 MD5_CHECKSUM_INVALID,400,The MD5 check-sums do not match
 MEDIA_CAPTION_TOO_LONG,400,The caption is too long
 MEDIA_EMPTY,400,The provided media object is invalid or the current account may not be able to send it (such as games as users)
-MEDIA_GROUPED_INVALID,400,
+MEDIA_GROUPED_INVALID,400,You tried to send media of different types in an album
 MEDIA_INVALID,400,Media invalid
 MEDIA_NEW_INVALID,400,The new media to edit the message with is invalid (such as stickers or voice notes)
 MEDIA_PREV_INVALID,400,The old media cannot be edited with anything else (such as stickers or voice notes)
@@ -212,13 +260,15 @@ MESSAGE_DELETE_FORBIDDEN,403,"You can't delete one of the messages you tried to
 MESSAGE_EDIT_TIME_EXPIRED,400,"You can't edit this message anymore, too much time has passed since its creation."
 MESSAGE_EMPTY,400,Empty or invalid UTF-8 message was sent
 MESSAGE_IDS_EMPTY,400,No message ids were provided
-MESSAGE_ID_INVALID,400,"The specified message ID is invalid or you can't do that operation on such message"
+MESSAGE_ID_INVALID,400,The specified message ID is invalid or you can't do that operation on such message
 MESSAGE_NOT_MODIFIED,400,Content of the message was not modified
 MESSAGE_POLL_CLOSED,400,The poll was closed and can no longer be voted on
 MESSAGE_TOO_LONG,400,Message was too long
 METHOD_INVALID,400,The API method is invalid and cannot be used
+MIN_DATE_INVALID,400,The specified minimum date is invalid
 MSGID_DECREASE_RETRY,500,The request should be retried with a lower message ID
 MSG_ID_INVALID,400,The message ID used in the peer was invalid
+MSG_TOO_OLD,400,"[`chat_read_mark_expire_period` seconds](https://core.telegram.org/api/config#chat-read-mark-expire-period) have passed since the message was sent, read receipts were deleted"
 MSG_WAIT_FAILED,400,A waiting call returned an error
 MT_SEND_QUEUE_TOO_LONG,500,
 MULTI_MEDIA_TOO_LONG,400,Too many media files were included in the same album
@@ -226,27 +276,32 @@ NEED_CHAT_INVALID,500,The provided chat is invalid
 NEED_MEMBER_INVALID,500,The provided member is invalid or does not exist (for example a thumb size)
 NETWORK_MIGRATE_X,303,The source IP address is associated with DC {new_dc}
 NEW_SALT_INVALID,400,The new salt is invalid
+NEW_SETTINGS_EMPTY,400,"No password is set on the current account, and no new password was specified in `new_settings`"
 NEW_SETTINGS_INVALID,400,The new settings are invalid
 NEXT_OFFSET_INVALID,400,The value for next_offset is invalid. Check that it has normal characters and is not too long
+NOT_ALLOWED,403,
 OFFSET_INVALID,400,"The given offset was invalid, it must be divisible by 1KB. See https://core.telegram.org/api/files#downloading-files"
 OFFSET_PEER_ID_INVALID,400,The provided offset peer is invalid
 OPTIONS_TOO_MUCH,400,You defined too many options for the poll
 OPTION_INVALID,400,The option specified is invalid and does not exist in the target poll
 PACK_SHORT_NAME_INVALID,400,"Invalid sticker pack name. It must begin with a letter, can't contain consecutive underscores and must end in ""_by_<bot username>""."
 PACK_SHORT_NAME_OCCUPIED,400,A stickerpack with this name already exists
+PACK_TITLE_INVALID,400,The stickerpack title is invalid
 PARTICIPANTS_TOO_FEW,400,Not enough participants
 PARTICIPANT_CALL_FAILED,500,Failure while making call
-PARTICIPANT_JOIN_MISSING,403,
+PARTICIPANT_ID_INVALID,400,The specified participant ID is invalid
+PARTICIPANT_JOIN_MISSING,400 403,"Trying to enable a presentation, when the user hasn't joined the Video Chat with [phone.joinGroupCall](https://core.telegram.org/method/phone.joinGroupCall)"
 PARTICIPANT_VERSION_OUTDATED,400,The other participant does not use an up to date telegram client with support for calls
 PASSWORD_EMPTY,400,The provided password is empty
 PASSWORD_HASH_INVALID,400,The password (and thus its hash value) you entered is invalid
 PASSWORD_MISSING,400,The account must have 2-factor authentication enabled (a password) before this method can be used
-PASSWORD_RECOVERY_EXPIRED,400,
+PASSWORD_RECOVERY_EXPIRED,400,The recovery code has expired
-PASSWORD_RECOVERY_NA,400,
+PASSWORD_RECOVERY_NA,400,"No email was set, can't recover password via email"
 PASSWORD_REQUIRED,400,The account must have 2-factor authentication enabled (a password) before this method can be used
 PASSWORD_TOO_FRESH_X,400,The password was added too recently and {seconds} seconds must pass before using the method
 PAYMENT_PROVIDER_INVALID,400,The payment provider was not recognised or its token was invalid
 PEER_FLOOD,400,Too many requests
+PEER_HISTORY_EMPTY,400,
 PEER_ID_INVALID,400,"An invalid Peer was used. Make sure to pass the right peer type and that the value is valid (for instance, bots cannot start conversations)"
 PEER_ID_NOT_SUPPORTED,400,The provided peer ID is not supported
 PERSISTENT_TIMESTAMP_EMPTY,400,Persistent timestamp empty
@@ -256,7 +311,9 @@ PHONE_CODE_EMPTY,400,The phone code is missing
 PHONE_CODE_EXPIRED,400,The confirmation code has expired
 PHONE_CODE_HASH_EMPTY,400,The phone code hash is missing
 PHONE_CODE_INVALID,400,The phone code entered was invalid
+PHONE_HASH_EXPIRED,400,An invalid or expired `phone_code_hash` was provided
 PHONE_MIGRATE_X,303,The phone number a user is trying to use for authorization is associated with DC {new_dc}
+PHONE_NOT_OCCUPIED,400,No user is associated to the specified phone number
 PHONE_NUMBER_APP_SIGNUP_FORBIDDEN,400,You can't sign up using this app
 PHONE_NUMBER_BANNED,400,The used phone number has been banned from Telegram and cannot be used anymore. Maybe check https://www.telegram.org/faq_spam
 PHONE_NUMBER_FLOOD,400,You asked for the code too many times.
@@ -265,66 +322,81 @@ PHONE_NUMBER_OCCUPIED,400,The phone number is already in use
 PHONE_NUMBER_UNOCCUPIED,400,The phone number is not yet being used
 PHONE_PASSWORD_FLOOD,406,You have tried logging in too many times
 PHONE_PASSWORD_PROTECTED,400,This phone is password protected
-PHOTO_CONTENT_TYPE_INVALID,400,
+PHOTO_CONTENT_TYPE_INVALID,400,Photo mime-type invalid
 PHOTO_CONTENT_URL_EMPTY,400,The content from the URL used as a photo appears to be empty or has caused another HTTP error
+PHOTO_CROP_FILE_MISSING,400,Photo crop file missing
 PHOTO_CROP_SIZE_SMALL,400,Photo is too small
 PHOTO_EXT_INVALID,400,The extension of the photo is invalid
+PHOTO_FILE_MISSING,400,Profile photo file missing
 PHOTO_ID_INVALID,400,Photo id is invalid
 PHOTO_INVALID,400,Photo invalid
 PHOTO_INVALID_DIMENSIONS,400,The photo dimensions are invalid (hint: `pip install pillow` for `send_file` to resize images)
 PHOTO_SAVE_FILE_INVALID,400,The photo you tried to send cannot be saved by Telegram. A reason may be that it exceeds 10MB. Try resizing it locally
 PHOTO_THUMB_URL_EMPTY,400,The URL used as a thumbnail appears to be empty or has caused another HTTP error
+PINNED_DIALOGS_TOO_MUCH,400,Too many pinned dialogs
 PIN_RESTRICTED,400,You can't pin messages in private chats with other people
-PINNED_DIALOGS_TOO_MUCH,400,
 POLL_ANSWERS_INVALID,400,The poll did not have enough answers or had too many
+POLL_ANSWER_INVALID,400,One of the poll answers is not acceptable
 POLL_OPTION_DUPLICATE,400,A duplicate option was sent in the same poll
 POLL_OPTION_INVALID,400,A poll option used invalid data (the data may be too long)
 POLL_QUESTION_INVALID,400,The poll question was either empty or too long
 POLL_UNSUPPORTED,400,This layer does not support polls in the issued method
-POLL_VOTE_REQUIRED,403,
+POLL_VOTE_REQUIRED,403,Cast a vote in the poll before calling this method
 POSTPONED_TIMEOUT,500,The postponed call has timed out
-PREMIUM_ACCOUNT_REQUIRED,403,
+PREMIUM_ACCOUNT_REQUIRED,403,A premium account is required to execute this action
+PREMIUM_CURRENTLY_UNAVAILABLE,406,
 PREVIOUS_CHAT_IMPORT_ACTIVE_WAIT_XMIN,406,"Similar to a flood wait, must wait {minutes} minutes"
 PRIVACY_KEY_INVALID,400,The privacy key is invalid
 PRIVACY_TOO_LONG,400,Cannot add that many entities in a single request
 PRIVACY_VALUE_INVALID,400,The privacy value is invalid
 PTS_CHANGE_EMPTY,500,No PTS change
-PUBLIC_KEY_REQUIRED,400,
+PUBLIC_CHANNEL_MISSING,403,You can only export group call invite links for public chats or channels
+PUBLIC_KEY_REQUIRED,400,A public key is required
 QUERY_ID_EMPTY,400,The query ID is empty
 QUERY_ID_INVALID,400,The query ID is invalid
 QUERY_TOO_SHORT,400,The query string is too short
-QUIZ_ANSWER_MISSING,400,
+QUIZ_ANSWER_MISSING,400,You can forward a quiz while hiding the original author only after choosing an option in the quiz
 QUIZ_CORRECT_ANSWERS_EMPTY,400,A quiz must specify one correct answer
 QUIZ_CORRECT_ANSWERS_TOO_MUCH,400,There can only be one correct answer
 QUIZ_CORRECT_ANSWER_INVALID,400,The correct answer is not an existing answer
 QUIZ_MULTIPLE_INVALID,400,A poll cannot be both multiple choice and quiz
 RANDOM_ID_DUPLICATE,500,You provided a random ID that was already used
+RANDOM_ID_EMPTY,400,Random ID empty
 RANDOM_ID_INVALID,400,A provided random ID is invalid
 RANDOM_LENGTH_INVALID,400,Random length invalid
 RANGES_INVALID,400,Invalid range provided
+REACTIONS_TOO_MANY,400,"The message already has exactly `reactions_uniq_max` reaction emojis, you can't react with a new emoji, see [the docs for more info](/api/config#client-configuration)"
 REACTION_EMPTY,400,No reaction provided
 REACTION_INVALID,400,Invalid reaction provided (only emoji are allowed)
 REFLECTOR_NOT_AVAILABLE,400,Invalid call reflector server
 REG_ID_GENERATE_FAILED,500,Failure while generating registration ID
+REPLY_MARKUP_BUY_EMPTY,400,Reply markup for buy button empty
 REPLY_MARKUP_GAME_EMPTY,400,The provided reply markup for the game is empty
 REPLY_MARKUP_INVALID,400,The provided reply markup is invalid
 REPLY_MARKUP_TOO_LONG,400,The data embedded in the reply markup buttons was too much
-RESET_REQUEST_MISSING,400,
+RESET_REQUEST_MISSING,400,No password reset is in progress
 RESULTS_TOO_MUCH,400,"You sent too many results, see https://core.telegram.org/bots/api#answerinlinequery for the current limit"
 RESULT_ID_DUPLICATE,400,Duplicated IDs on the sent results. Make sure to use unique IDs
+RESULT_ID_EMPTY,400,Result ID empty
 RESULT_ID_INVALID,400,The given result cannot be used to send the selection to the bot
 RESULT_TYPE_INVALID,400,Result type invalid
+REVOTE_NOT_ALLOWED,400,You cannot change your vote
+RIGHTS_NOT_MODIFIED,400,"The new admin rights are equal to the old rights, no change was made"
 RIGHT_FORBIDDEN,403,Either your admin rights do not allow you to do this or you passed the wrong rights combination (some rights only apply to channels and vice versa)
 RPC_CALL_FAIL,500,"Telegram is having internal issues, please try again later."
 RPC_MCGET_FAIL,500,"Telegram is having internal issues, please try again later."
 RSA_DECRYPT_FAILED,400,Internal RSA decryption failed
 SCHEDULE_BOT_NOT_ALLOWED,400,Bots are not allowed to schedule messages
-SCHEDULE_DATE_INVALID,400,
+SCHEDULE_DATE_INVALID,400,Invalid schedule date provided
 SCHEDULE_DATE_TOO_LATE,400,The date you tried to schedule is too far in the future (last known limit of 1 year and a few hours)
 SCHEDULE_STATUS_PRIVATE,400,You cannot schedule a message until the person comes online if their privacy does not show this information
 SCHEDULE_TOO_MUCH,400,You cannot schedule more messages in this chat (last known limit of 100 per chat)
+SCORE_INVALID,400,The specified game score is invalid
 SEARCH_QUERY_EMPTY,400,The search query is empty
+SEARCH_WITH_LINK_NOT_SUPPORTED,400,You cannot provide a search query and an invite link at the same time
 SECONDS_INVALID,400,"Slow mode only supports certain values (e.g. 0, 10s, 30s, 1m, 5m, 15m and 1h)"
+SEND_AS_PEER_INVALID,400,You can't send messages as the specified peer
+SEND_CODE_UNAVAILABLE,406,"Returned when all available options for this type of number were already used (e.g. flash-call, then SMS, then this error might be returned to trigger a second resend)"
 SEND_MESSAGE_MEDIA_INVALID,400,The message media was invalid or not specified
 SEND_MESSAGE_TYPE_INVALID,400,The message type is invalid
 SENSITIVE_CHANGE_FORBIDDEN,403,Your sensitive content settings cannot be changed at this time
@@ -332,72 +404,99 @@ SESSION_EXPIRED,401,The authorization has expired
 SESSION_PASSWORD_NEEDED,401,Two-steps verification is enabled and a password is required
 SESSION_REVOKED,401,"The authorization has been invalidated, because of the user terminating all sessions"
 SESSION_TOO_FRESH_X,400,The session logged in too recently and {seconds} seconds must pass before calling the method
+SETTINGS_INVALID,400,Invalid settings were provided
 SHA256_HASH_INVALID,400,The provided SHA256 hash is invalid
 SHORTNAME_OCCUPY_FAILED,400,An error occurred when trying to register the short-name used for the sticker pack. Try a different name
-SHORT_NAME_INVALID,400,
+SHORT_NAME_INVALID,400,The specified short name is invalid
-SHORT_NAME_OCCUPIED,400,
+SHORT_NAME_OCCUPIED,400,The specified short name is already in use
+SIGN_IN_FAILED,500,Failure while signing in
+SLOWMODE_MULTI_MSGS_DISABLED,400,"Slowmode is enabled, you cannot forward multiple messages to this group"
 SLOWMODE_WAIT_X,420,A wait of {seconds} seconds is required before sending another message in this chat
-SRP_ID_INVALID,400,
+SMS_CODE_CREATE_FAILED,400,An error occurred while creating the SMS code
+SRP_ID_INVALID,400,Invalid SRP ID provided
+SRP_PASSWORD_CHANGED,400,Password has changed
 START_PARAM_EMPTY,400,The start parameter is empty
 START_PARAM_INVALID,400,Start parameter invalid
+START_PARAM_TOO_LONG,400,Start parameter is too long
 STATS_MIGRATE_X,303,The channel statistics must be fetched from DC {dc}
-STICKERSET_INVALID,400,The provided sticker set is invalid
+STICKERPACK_STICKERS_TOO_MUCH,400,"There are too many stickers in this stickerpack, you can't add any more"
+STICKERSET_INVALID,400 406,The provided sticker set is invalid
 STICKERSET_OWNER_ANONYMOUS,406,This sticker set can't be used as the group's official stickers because it was created by one of its anonymous admins
 STICKERS_EMPTY,400,No sticker provided
+STICKERS_TOO_MUCH,400,"There are too many stickers in this stickerpack, you can't add any more"
 STICKER_DOCUMENT_INVALID,400,"The sticker file was invalid (this file has failed Telegram internal checks, make sure to use the correct format and comply with https://core.telegram.org/animated_stickers)"
 STICKER_EMOJI_INVALID,400,Sticker emoji invalid
 STICKER_FILE_INVALID,400,Sticker file invalid
+STICKER_GIF_DIMENSIONS,400,The specified video sticker has invalid dimensions
 STICKER_ID_INVALID,400,The provided sticker ID is invalid
 STICKER_INVALID,400,The provided sticker is invalid
+STICKER_MIME_INVALID,400,Make sure to pass a valid image file for the right InputFile parameter
 STICKER_PNG_DIMENSIONS,400,Sticker png dimensions invalid
 STICKER_PNG_NOPNG,400,Stickers must be a png file but the used image was not a png
-STICKER_TGS_NODOC,400,
+STICKER_TGS_NODOC,400,You must send the animated sticker as a document
 STICKER_TGS_NOTGS,400,Stickers must be a tgs file but the used file was not a tgs
 STICKER_THUMB_PNG_NOPNG,400,Stickerset thumb must be a png file but the used file was not png
 STICKER_THUMB_TGS_NOTGS,400,Stickerset thumb must be a tgs file but the used file was not tgs
+STICKER_VIDEO_BIG,400,The specified video sticker is too big
+STICKER_VIDEO_NODOC,400,You must send the video sticker as a document
+STICKER_VIDEO_NOWEBM,400,The specified video sticker is not in webm format
 STORAGE_CHECK_FAILED,500,Server storage check failed
 STORE_INVALID_SCALAR_TYPE,500,
+SWITCH_PM_TEXT_EMPTY,400,The switch_pm.text field was empty
 TAKEOUT_INIT_DELAY_X,420,A wait of {seconds} seconds is required before being able to initiate the takeout
 TAKEOUT_INVALID,400,The takeout session has been invalidated by another data export session
-TAKEOUT_REQUIRED,400,You must initialize a takeout request first
+TAKEOUT_REQUIRED,400 403,You must initialize a takeout request first
+TEMP_AUTH_KEY_ALREADY_BOUND,400,The passed temporary key is already bound to another **perm_auth_key_id**
 TEMP_AUTH_KEY_EMPTY,400,No temporary auth key provided
-TIMEOUT,500,A timeout occurred while fetching data from the worker
+THEME_FILE_INVALID,400,Invalid theme file provided
-TITLE_INVALID,400,
+THEME_FORMAT_INVALID,400,Invalid theme format provided
 THEME_INVALID,400,Theme invalid
 THEME_MIME_INVALID,400,"You cannot create this theme, the mime-type is invalid"
+THEME_TITLE_INVALID,400,The specified theme title is invalid
+TIMEOUT,500,A timeout occurred while fetching data from the worker
+TITLE_INVALID,400,The specified stickerpack title is invalid
 TMP_PASSWORD_DISABLED,400,The temporary password is disabled
 TMP_PASSWORD_INVALID,400,Password auth needs to be regenerated
 TOKEN_INVALID,400,The provided token is invalid
+TOPIC_DELETED,400,The topic was deleted
+TO_LANG_INVALID,400,The specified destination language is invalid
 TTL_DAYS_INVALID,400,The provided TTL is invalid
 TTL_MEDIA_INVALID,400,The provided media cannot be used with a TTL
 TTL_PERIOD_INVALID,400,The provided TTL Period is invalid
 TYPES_EMPTY,400,The types field is empty
 TYPE_CONSTRUCTOR_INVALID,400,The type constructor is invalid
+Timedout,-503,Timeout while fetching data
+Timeout,-503,Timeout while fetching data
 UNKNOWN_ERROR,400,
 UNKNOWN_METHOD,500,The method you tried to call cannot be called on non-CDN DCs
 UNTIL_DATE_INVALID,400,That date cannot be specified in this request (try using None)
+UPDATE_APP_TO_LOGIN,406,
 URL_INVALID,400,The URL used was invalid (e.g. when answering a callback with a URL that's not t.me/yourbot or your game's URL)
-USER_VOLUME_INVALID,400,
+USAGE_LIMIT_INVALID,400,The specified usage limit is invalid
 USERNAME_INVALID,400,"Nobody is using this username, or the username is unacceptable. If the latter, it must match r""[a-zA-Z][\w\d]{3,30}[a-zA-Z\d]"""
 USERNAME_NOT_MODIFIED,400,The username is not different from the current username
 USERNAME_NOT_OCCUPIED,400,The username is not in use by anyone else yet
 USERNAME_OCCUPIED,400,The username is already taken
+USERNAME_PURCHASE_AVAILABLE,400,
+USERPIC_PRIVACY_REQUIRED,406,You need to disable privacy settings for your profile picture in order to make your geolocation public
+USERPIC_UPLOAD_REQUIRED,400 406,You must have a profile picture before using this method
 USERS_TOO_FEW,400,"Not enough users (to create a chat, for example)"
 USERS_TOO_MUCH,400,"The maximum number of users has been exceeded (to create a chat, for example)"
 USER_ADMIN_INVALID,400,Either you're not an admin or you tried to ban an admin that you didn't promote
-USER_ALREADY_INVITED,400,
+USER_ALREADY_INVITED,400,You have already invited this user
 USER_ALREADY_PARTICIPANT,400,The authenticated user is already a participant of the chat
 USER_BANNED_IN_CHANNEL,400,You're banned from sending messages in supergroups/channels
 USER_BLOCKED,400,User blocked
 USER_BOT,400,Bots can only be admins in channels.
 USER_BOT_INVALID,400 403,This method can only be called by a bot
 USER_BOT_REQUIRED,400,This method can only be called by a bot
-USER_CHANNELS_TOO_MUCH,403,One of the users you tried to add is already in too many channels/supergroups
+USER_CHANNELS_TOO_MUCH,400 403,One of the users you tried to add is already in too many channels/supergroups
 USER_CREATOR,400,"You can't leave this channel, because you're its creator"
 USER_DEACTIVATED,401,The user has been deleted/deactivated
 USER_DEACTIVATED_BAN,401,The user has been deleted/deactivated
+USER_DELETED,403,You can't send this secret message because the other participant deleted their account
 USER_ID_INVALID,400,"Invalid object ID for a user. Make sure to pass the right types, for instance making sure that the request is designed for users or otherwise look for a different one more suited"
-USER_INVALID,400,The given user was invalid
+USER_INVALID,400 403,The given user was invalid
 USER_IS_BLOCKED,400 403,User is blocked
 USER_IS_BOT,400,Bots can't send messages to other bots
 USER_KICKED,400,This user was kicked from this supergroup/channel
@@ -405,18 +504,26 @@ USER_MIGRATE_X,303,The user whose identity is being used to execute queries is a
 USER_NOT_MUTUAL_CONTACT,400 403,The provided user is not a mutual contact
 USER_NOT_PARTICIPANT,400,The target user is not a member of the specified megagroup or channel
 USER_PRIVACY_RESTRICTED,403,The user's privacy settings do not allow you to do this
-USER_RESTRICTED,403,"You're spamreported, you can't create channels or chats."
+USER_RESTRICTED,403 406,"You're spamreported, you can't create channels or chats."
-USERPIC_UPLOAD_REQUIRED,400,You must have a profile picture before using this method
+USER_VOLUME_INVALID,400,The specified user volume is invalid
 VIDEO_CONTENT_TYPE_INVALID,400,The video content type is not supported with the given parameters (i.e. supports_streaming)
 VIDEO_FILE_INVALID,400,The given video cannot be used
-VIDEO_TITLE_EMPTY,400,
+VIDEO_TITLE_EMPTY,400,The specified video title is empty
+VOICE_MESSAGES_FORBIDDEN,400,This user's privacy settings forbid you from sending voice messages
 WALLPAPER_FILE_INVALID,400,The given file cannot be used as a wallpaper
 WALLPAPER_INVALID,400,The input wallpaper was not valid
-WALLPAPER_MIME_INVALID,400,
+WALLPAPER_MIME_INVALID,400,The specified wallpaper MIME type is invalid
 WC_CONVERT_URL_INVALID,400,WC convert URL invalid
-WEBDOCUMENT_MIME_INVALID,400,
+WEBDOCUMENT_INVALID,400,Invalid webdocument URL provided
+WEBDOCUMENT_MIME_INVALID,400,Invalid webdocument mime type provided
+WEBDOCUMENT_SIZE_TOO_BIG,400,Webdocument is too big!
 WEBDOCUMENT_URL_INVALID,400,The given URL cannot be used
 WEBPAGE_CURL_FAILED,400,Failure while fetching the webpage with cURL
 WEBPAGE_MEDIA_EMPTY,400,Webpage media empty
+WEBPUSH_AUTH_INVALID,400,The specified web push authentication secret is invalid
+WEBPUSH_KEY_INVALID,400,The specified web push elliptic curve Diffie-Hellman public key is invalid
+WEBPUSH_TOKEN_INVALID,400,The specified web push token is invalid
 WORKER_BUSY_TOO_LONG_RETRY,500,Telegram workers are too busy to respond immediately
 YOU_BLOCKED_USER,400,You blocked this user
+FROZEN_METHOD_INVALID,420,You tried to use a method that is not available for frozen accounts
+FROZEN_PARTICIPANT_MISSING,400,Your account is frozen and can't access the chat

telethon_generator/data/friendly.csv

@@ -1,7 +1,6 @@
 ns,friendly,raw
 account.AccountMethods,takeout,invokeWithTakeout account.initTakeoutSession account.finishTakeoutSession
 auth.AuthMethods,sign_in,auth.signIn auth.importBotAuthorization
-auth.AuthMethods,sign_up,auth.signUp
 auth.AuthMethods,send_code_request,auth.sendCode auth.resendCode
 auth.AuthMethods,log_out,auth.logOut
 auth.AuthMethods,edit_2fa,account.updatePasswordSettings

telethon_generator/data/methods.csv

@@ -80,7 +80,7 @@ auth.importLoginToken,user,AUTH_TOKEN_ALREADY_ACCEPTED AUTH_TOKEN_EXPIRED AUTH_T
 auth.logOut,both,
 auth.recoverPassword,user,CODE_EMPTY NEW_SETTINGS_INVALID
 auth.requestPasswordRecovery,user,PASSWORD_EMPTY PASSWORD_RECOVERY_NA
-auth.resendCode,user,PHONE_NUMBER_INVALID
+auth.resendCode,user,PHONE_NUMBER_INVALID SEND_CODE_UNAVAILABLE
 auth.resetAuthorizations,user,TIMEOUT
 auth.sendCode,user,API_ID_INVALID API_ID_PUBLISHED_FLOOD AUTH_RESTART INPUT_REQUEST_TOO_LONG PHONE_NUMBER_APP_SIGNUP_FORBIDDEN PHONE_NUMBER_BANNED PHONE_NUMBER_FLOOD PHONE_NUMBER_INVALID PHONE_PASSWORD_FLOOD PHONE_PASSWORD_PROTECTED
 auth.signIn,user,PHONE_CODE_EMPTY PHONE_CODE_EXPIRED PHONE_CODE_INVALID PHONE_NUMBER_INVALID PHONE_NUMBER_UNOCCUPIED SESSION_PASSWORD_NEEDED
@@ -92,7 +92,7 @@ channels.checkUsername,user,CHANNEL_INVALID CHAT_ID_INVALID USERNAME_INVALID
 channels.convertToGigagroup,user,PARTICIPANTS_TOO_FEW
 channels.createChannel,user,CHAT_TITLE_EMPTY USER_RESTRICTED
 channels.deleteChannel,user,CHANNEL_INVALID CHANNEL_PRIVATE CHANNEL_TOO_LARGE
-channels.deleteHistory,user,
+channels.deleteHistory,user,CHANNEL_TOO_BIG
 channels.deleteMessages,both,CHANNEL_INVALID CHANNEL_PRIVATE MESSAGE_DELETE_FORBIDDEN
 channels.deleteUserHistory,user,CHANNEL_INVALID CHAT_ADMIN_REQUIRED
 channels.editAdmin,both,ADMINS_TOO_MUCH ADMIN_RANK_EMOJI_NOT_ALLOWED ADMIN_RANK_INVALID BOT_CHANNELS_NA CHANNEL_INVALID CHAT_ADMIN_INVITE_REQUIRED CHAT_ADMIN_REQUIRED FRESH_CHANGE_ADMINS_FORBIDDEN RIGHT_FORBIDDEN USER_CREATOR USER_ID_INVALID USER_NOT_MUTUAL_CONTACT USER_PRIVACY_RESTRICTED
@@ -113,17 +113,18 @@ channels.getMessages,both,CHANNEL_INVALID CHANNEL_PRIVATE MESSAGE_IDS_EMPTY
 channels.getParticipant,both,CHANNEL_INVALID CHANNEL_PRIVATE CHAT_ADMIN_REQUIRED USER_ID_INVALID USER_NOT_PARTICIPANT
 channels.getParticipants,both,CHANNEL_INVALID CHANNEL_PRIVATE CHAT_ADMIN_REQUIRED INPUT_CONSTRUCTOR_INVALID TIMEOUT
 channels.inviteToChannel,user,BOTS_TOO_MUCH BOT_GROUPS_BLOCKED CHANNEL_INVALID CHANNEL_PRIVATE CHAT_ADMIN_REQUIRED CHAT_INVALID CHAT_WRITE_FORBIDDEN INPUT_USER_DEACTIVATED USERS_TOO_MUCH USER_BANNED_IN_CHANNEL USER_BLOCKED USER_BOT USER_CHANNELS_TOO_MUCH USER_ID_INVALID USER_KICKED USER_NOT_MUTUAL_CONTACT USER_PRIVACY_RESTRICTED
-channels.joinChannel,user,CHANNELS_TOO_MUCH CHANNEL_INVALID CHANNEL_PRIVATE
+channels.joinChannel,user,CHANNELS_TOO_MUCH CHANNEL_INVALID CHANNEL_PRIVATE INVITE_REQUEST_SENT
 channels.leaveChannel,both,CHANNEL_INVALID CHANNEL_PRIVATE CHANNEL_PUBLIC_GROUP_NA USER_CREATOR USER_NOT_PARTICIPANT
 channels.readHistory,user,CHANNEL_INVALID CHANNEL_PRIVATE
 channels.readMessageContents,user,CHANNEL_INVALID CHANNEL_PRIVATE
 channels.reportSpam,user,CHANNEL_INVALID INPUT_USER_DEACTIVATED
 channels.setDiscussionGroup,user,BROADCAST_ID_INVALID LINK_NOT_MODIFIED MEGAGROUP_ID_INVALID MEGAGROUP_PREHISTORY_HIDDEN
 channels.setStickers,both,CHANNEL_INVALID PARTICIPANTS_TOO_FEW STICKERSET_OWNER_ANONYMOUS
+channels.toggleForum,user,CHAT_DISCUSSION_UNALLOWED
 channels.togglePreHistoryHidden,user,CHAT_LINK_EXISTS
 channels.toggleSignatures,user,CHANNEL_INVALID
 channels.toggleSlowMode,user,SECONDS_INVALID
-channels.updateUsername,user,CHANNELS_ADMIN_PUBLIC_TOO_MUCH CHANNEL_INVALID CHAT_ADMIN_REQUIRED USERNAME_INVALID USERNAME_OCCUPIED
+channels.updateUsername,user,CHANNELS_ADMIN_PUBLIC_TOO_MUCH CHANNEL_INVALID CHAT_ADMIN_REQUIRED USERNAME_INVALID USERNAME_OCCUPIED USERNAME_PURCHASE_AVAILABLE
 channels.viewSponsoredMessage,user,UNKNOWN_ERROR
 contacts.acceptContact,user,
 contacts.addContact,user,CONTACT_NAME_EMPTY
@@ -148,7 +149,7 @@ folders.deleteFolder,user,FOLDER_ID_EMPTY
 folders.editPeerFolders,user,FOLDER_ID_INVALID
 getFutureSalts,both,
 help.acceptTermsOfService,user,
-help.editUserInfo,user,USER_INVALID ENTITY_BOUNDS_INVALID
+help.editUserInfo,user,ENTITY_BOUNDS_INVALID USER_INVALID
 help.getAppChangelog,user,
 help.getAppConfig,user,
 help.getAppUpdate,user,
@@ -195,11 +196,11 @@ messages.editChatAdmin,user,CHAT_ID_INVALID
 messages.editChatDefaultBannedRights,both,BANNED_RIGHTS_INVALID
 messages.editChatPhoto,both,CHAT_ID_INVALID INPUT_CONSTRUCTOR_INVALID INPUT_FETCH_FAIL PEER_ID_INVALID PHOTO_EXT_INVALID
 messages.editChatTitle,both,CHAT_ID_INVALID NEED_CHAT_INVALID
-messages.editInlineBotMessage,both,MESSAGE_ID_INVALID MESSAGE_NOT_MODIFIED ENTITY_BOUNDS_INVALID
+messages.editInlineBotMessage,both,ENTITY_BOUNDS_INVALID MESSAGE_ID_INVALID MESSAGE_NOT_MODIFIED
-messages.editMessage,both,CHANNEL_INVALID CHANNEL_PRIVATE CHAT_WRITE_FORBIDDEN INLINE_BOT_REQUIRED INPUT_USER_DEACTIVATED MEDIA_GROUPED_INVALID MEDIA_NEW_INVALID MEDIA_PREV_INVALID MESSAGE_AUTHOR_REQUIRED MESSAGE_EDIT_TIME_EXPIRED MESSAGE_EMPTY MESSAGE_ID_INVALID MESSAGE_NOT_MODIFIED PEER_ID_INVALID ENTITY_BOUNDS_INVALID
+messages.editMessage,both,CHANNEL_INVALID CHANNEL_PRIVATE CHAT_WRITE_FORBIDDEN ENTITY_BOUNDS_INVALID INLINE_BOT_REQUIRED INPUT_USER_DEACTIVATED MEDIA_GROUPED_INVALID MEDIA_NEW_INVALID MEDIA_PREV_INVALID MESSAGE_AUTHOR_REQUIRED MESSAGE_EDIT_TIME_EXPIRED MESSAGE_EMPTY MESSAGE_ID_INVALID MESSAGE_NOT_MODIFIED PEER_ID_INVALID
-messages.exportChatInvite,both,CHAT_ID_INVALID
+messages.exportChatInvite,both,CHAT_ID_INVALID EXPIRE_DATE_INVALID
 messages.faveSticker,user,STICKER_ID_INVALID
-messages.forwardMessages,both,BROADCAST_PUBLIC_VOTERS_FORBIDDEN CHANNEL_INVALID CHANNEL_PRIVATE CHAT_ADMIN_REQUIRED CHAT_ID_INVALID CHAT_SEND_GIFS_FORBIDDEN CHAT_SEND_MEDIA_FORBIDDEN CHAT_SEND_STICKERS_FORBIDDEN CHAT_WRITE_FORBIDDEN GROUPED_MEDIA_INVALID INPUT_USER_DEACTIVATED MEDIA_EMPTY MESSAGE_IDS_EMPTY MESSAGE_ID_INVALID PEER_ID_INVALID PTS_CHANGE_EMPTY QUIZ_ANSWER_MISSING RANDOM_ID_DUPLICATE RANDOM_ID_INVALID SCHEDULE_DATE_TOO_LATE SCHEDULE_TOO_MUCH TIMEOUT USER_BANNED_IN_CHANNEL USER_IS_BLOCKED USER_IS_BOT YOU_BLOCKED_USER
+messages.forwardMessages,both,BROADCAST_PUBLIC_VOTERS_FORBIDDEN CHANNEL_INVALID CHANNEL_PRIVATE CHAT_ADMIN_REQUIRED CHAT_ID_INVALID CHAT_SEND_GIFS_FORBIDDEN CHAT_SEND_MEDIA_FORBIDDEN CHAT_SEND_STICKERS_FORBIDDEN CHAT_WRITE_FORBIDDEN GROUPED_MEDIA_INVALID INPUT_USER_DEACTIVATED MEDIA_EMPTY MESSAGE_IDS_EMPTY MESSAGE_ID_INVALID PEER_ID_INVALID PTS_CHANGE_EMPTY QUIZ_ANSWER_MISSING RANDOM_ID_DUPLICATE RANDOM_ID_INVALID SCHEDULE_DATE_TOO_LATE SCHEDULE_TOO_MUCH TIMEOUT TOPIC_DELETED USER_BANNED_IN_CHANNEL USER_IS_BLOCKED USER_IS_BOT YOU_BLOCKED_USER
 messages.getAllChats,user,
 messages.getAllDrafts,user,
 messages.getAllStickers,user,
@@ -251,7 +252,7 @@ messages.getWebPage,user,WC_CONVERT_URL_INVALID
 messages.getWebPagePreview,user,ENTITY_BOUNDS_INVALID
 messages.hideAllChatJoinRequests,user,HIDE_REQUESTER_MISSING
 messages.hidePeerSettingsBar,user,
-messages.importChatInvite,user,CHANNELS_TOO_MUCH INVITE_HASH_EMPTY INVITE_HASH_EXPIRED INVITE_HASH_INVALID SESSION_PASSWORD_NEEDED USERS_TOO_MUCH USER_ALREADY_PARTICIPANT
+messages.importChatInvite,user,CHANNELS_TOO_MUCH INVITE_HASH_EMPTY INVITE_HASH_EXPIRED INVITE_HASH_INVALID INVITE_REQUEST_SENT SESSION_PASSWORD_NEEDED USERS_TOO_MUCH USER_ALREADY_PARTICIPANT
 messages.initHistoryImport,user,IMPORT_FILE_INVALID IMPORT_FORMAT_UNRECOGNIZED PREVIOUS_CHAT_IMPORT_ACTIVE_WAIT_XMIN TIMEOUT
 messages.installStickerSet,user,STICKERSET_INVALID
 messages.markDialogUnread,user,
@@ -262,7 +263,7 @@ messages.readHistory,user,PEER_ID_INVALID TIMEOUT
 messages.readMentions,user,
 messages.readMessageContents,user,
 messages.receivedMessages,user,
-messages.receivedQueue,user,MSG_WAIT_FAILED MAX_QTS_INVALID
+messages.receivedQueue,user,MAX_QTS_INVALID MSG_WAIT_FAILED
 messages.reorderPinnedDialogs,user,PEER_ID_INVALID
 messages.reorderStickerSets,user,
 messages.report,user,
@@ -270,7 +271,7 @@ messages.reportEncryptedSpam,user,CHAT_ID_INVALID
 messages.reportSpam,user,PEER_ID_INVALID
 messages.requestEncryption,user,DH_G_A_INVALID USER_ID_INVALID
 messages.requestUrlAuth,user,
-messages.saveDraft,user,PEER_ID_INVALID ENTITY_BOUNDS_INVALID
+messages.saveDraft,user,ENTITY_BOUNDS_INVALID PEER_ID_INVALID
 messages.saveGif,user,GIF_ID_INVALID
 messages.saveRecentSticker,user,STICKER_ID_INVALID
 messages.search,user,CHAT_ADMIN_REQUIRED FROM_PEER_INVALID INPUT_CONSTRUCTOR_INVALID INPUT_FILTER_INVALID INPUT_USER_DEACTIVATED PEER_ID_INVALID PEER_ID_NOT_SUPPORTED SEARCH_QUERY_EMPTY USER_ID_INVALID
@@ -280,10 +281,10 @@ messages.searchStickerSets,user,
 messages.sendEncrypted,user,CHAT_ID_INVALID DATA_INVALID ENCRYPTION_DECLINED MSG_WAIT_FAILED
 messages.sendEncryptedFile,user,MSG_WAIT_FAILED
 messages.sendEncryptedService,user,DATA_INVALID ENCRYPTION_DECLINED MSG_WAIT_FAILED USER_IS_BLOCKED
-messages.sendInlineBotResult,user,CHAT_SEND_INLINE_FORBIDDEN CHAT_WRITE_FORBIDDEN INLINE_RESULT_EXPIRED PEER_ID_INVALID QUERY_ID_EMPTY SCHEDULE_DATE_TOO_LATE SCHEDULE_TOO_MUCH WEBPAGE_CURL_FAILED WEBPAGE_MEDIA_EMPTY ENTITY_BOUNDS_INVALID
+messages.sendInlineBotResult,user,CHAT_SEND_INLINE_FORBIDDEN CHAT_WRITE_FORBIDDEN ENTITY_BOUNDS_INVALID INLINE_RESULT_EXPIRED PEER_ID_INVALID QUERY_ID_EMPTY SCHEDULE_DATE_TOO_LATE SCHEDULE_TOO_MUCH TOPIC_DELETED WEBPAGE_CURL_FAILED WEBPAGE_MEDIA_EMPTY
-messages.sendMedia,both,BOT_PAYMENTS_DISABLED BOT_POLLS_DISABLED BROADCAST_PUBLIC_VOTERS_FORBIDDEN CHANNEL_INVALID CHANNEL_PRIVATE CHAT_ADMIN_REQUIRED CHAT_SEND_MEDIA_FORBIDDEN CHAT_WRITE_FORBIDDEN CURRENCY_TOTAL_AMOUNT_INVALID EMOTICON_INVALID EXTERNAL_URL_INVALID FILE_PARTS_INVALID FILE_PART_LENGTH_INVALID FILE_REFERENCE_EMPTY FILE_REFERENCE_EXPIRED GAME_BOT_INVALID INPUT_USER_DEACTIVATED MEDIA_CAPTION_TOO_LONG MEDIA_EMPTY PAYMENT_PROVIDER_INVALID PEER_ID_INVALID PHOTO_EXT_INVALID PHOTO_INVALID_DIMENSIONS PHOTO_SAVE_FILE_INVALID POLL_ANSWERS_INVALID POLL_OPTION_DUPLICATE POLL_QUESTION_INVALID QUIZ_CORRECT_ANSWERS_EMPTY QUIZ_CORRECT_ANSWERS_TOO_MUCH QUIZ_CORRECT_ANSWER_INVALID QUIZ_MULTIPLE_INVALID RANDOM_ID_DUPLICATE SCHEDULE_DATE_TOO_LATE SCHEDULE_TOO_MUCH STORAGE_CHECK_FAILED TIMEOUT USER_BANNED_IN_CHANNEL USER_IS_BLOCKED USER_IS_BOT VIDEO_CONTENT_TYPE_INVALID WEBPAGE_CURL_FAILED WEBPAGE_MEDIA_EMPTY POSTPONED_TIMEOUT ENTITY_BOUNDS_INVALID
+messages.sendMedia,both,BOT_PAYMENTS_DISABLED BOT_POLLS_DISABLED BROADCAST_PUBLIC_VOTERS_FORBIDDEN CHANNEL_INVALID CHANNEL_PRIVATE CHAT_ADMIN_REQUIRED CHAT_SEND_MEDIA_FORBIDDEN CHAT_WRITE_FORBIDDEN CURRENCY_TOTAL_AMOUNT_INVALID EMOTICON_INVALID ENTITY_BOUNDS_INVALID EXTERNAL_URL_INVALID FILE_PARTS_INVALID FILE_PART_LENGTH_INVALID FILE_REFERENCE_EMPTY FILE_REFERENCE_EXPIRED GAME_BOT_INVALID INPUT_USER_DEACTIVATED MEDIA_CAPTION_TOO_LONG MEDIA_EMPTY PAYMENT_PROVIDER_INVALID PEER_ID_INVALID PHOTO_EXT_INVALID PHOTO_INVALID_DIMENSIONS PHOTO_SAVE_FILE_INVALID POLL_ANSWERS_INVALID POLL_OPTION_DUPLICATE POLL_QUESTION_INVALID POSTPONED_TIMEOUT QUIZ_CORRECT_ANSWERS_EMPTY QUIZ_CORRECT_ANSWERS_TOO_MUCH QUIZ_CORRECT_ANSWER_INVALID QUIZ_MULTIPLE_INVALID RANDOM_ID_DUPLICATE SCHEDULE_DATE_TOO_LATE SCHEDULE_TOO_MUCH STORAGE_CHECK_FAILED TIMEOUT TOPIC_DELETED USER_BANNED_IN_CHANNEL USER_IS_BLOCKED USER_IS_BOT VIDEO_CONTENT_TYPE_INVALID WEBPAGE_CURL_FAILED WEBPAGE_MEDIA_EMPTY
-messages.sendMessage,both,AUTH_KEY_DUPLICATED BOT_DOMAIN_INVALID BUTTON_DATA_INVALID BUTTON_TYPE_INVALID BUTTON_URL_INVALID CHANNEL_INVALID CHANNEL_PRIVATE CHAT_ADMIN_REQUIRED CHAT_ID_INVALID CHAT_RESTRICTED CHAT_WRITE_FORBIDDEN ENTITIES_TOO_LONG ENTITY_MENTION_USER_INVALID INPUT_USER_DEACTIVATED MESSAGE_EMPTY MESSAGE_TOO_LONG MSG_ID_INVALID PEER_ID_INVALID POLL_OPTION_INVALID RANDOM_ID_DUPLICATE REPLY_MARKUP_INVALID REPLY_MARKUP_TOO_LONG SCHEDULE_BOT_NOT_ALLOWED SCHEDULE_DATE_TOO_LATE SCHEDULE_STATUS_PRIVATE SCHEDULE_TOO_MUCH TIMEOUT USER_BANNED_IN_CHANNEL USER_IS_BLOCKED USER_IS_BOT YOU_BLOCKED_USER ENTITY_BOUNDS_INVALID
+messages.sendMessage,both,AUTH_KEY_DUPLICATED BOT_DOMAIN_INVALID BUTTON_DATA_INVALID BUTTON_TYPE_INVALID BUTTON_URL_INVALID CHANNEL_INVALID CHANNEL_PRIVATE CHAT_ADMIN_REQUIRED CHAT_ID_INVALID CHAT_RESTRICTED CHAT_WRITE_FORBIDDEN ENTITIES_TOO_LONG ENTITY_BOUNDS_INVALID ENTITY_MENTION_USER_INVALID INPUT_USER_DEACTIVATED MESSAGE_EMPTY MESSAGE_TOO_LONG MSG_ID_INVALID PEER_ID_INVALID POLL_OPTION_INVALID RANDOM_ID_DUPLICATE REPLY_MARKUP_INVALID REPLY_MARKUP_TOO_LONG SCHEDULE_BOT_NOT_ALLOWED SCHEDULE_DATE_TOO_LATE SCHEDULE_STATUS_PRIVATE SCHEDULE_TOO_MUCH TIMEOUT TOPIC_DELETED USER_BANNED_IN_CHANNEL USER_IS_BLOCKED USER_IS_BOT YOU_BLOCKED_USER
-messages.sendMultiMedia,both,MULTI_MEDIA_TOO_LONG SCHEDULE_DATE_TOO_LATE SCHEDULE_TOO_MUCH ENTITY_BOUNDS_INVALID
+messages.sendMultiMedia,both,ENTITY_BOUNDS_INVALID MULTI_MEDIA_TOO_LONG SCHEDULE_DATE_TOO_LATE SCHEDULE_TOO_MUCH TOPIC_DELETED
 messages.sendScheduledMessages,user,
 messages.sendVote,user,MESSAGE_POLL_CLOSED OPTION_INVALID
 messages.setBotCallbackAnswer,both,QUERY_ID_INVALID URL_INVALID
@@ -298,7 +299,7 @@ messages.setInlineGameScore,bot,MESSAGE_ID_INVALID USER_BOT_REQUIRED
 messages.setTyping,both,CHANNEL_INVALID CHANNEL_PRIVATE CHAT_ID_INVALID CHAT_WRITE_FORBIDDEN PEER_ID_INVALID USER_BANNED_IN_CHANNEL USER_IS_BLOCKED USER_IS_BOT
 messages.startBot,user,BOT_INVALID PEER_ID_INVALID START_PARAM_EMPTY START_PARAM_INVALID
 messages.startHistoryImport,user,IMPORT_ID_INVALID
-messages.toggleDialogPin,user,PEER_ID_INVALID PINNED_DIALOGS_TOO_MUCH
+messages.toggleDialogPin,user,PEER_HISTORY_EMPTY PEER_ID_INVALID PINNED_DIALOGS_TOO_MUCH
 messages.toggleStickerSets,user,
 messages.uninstallStickerSet,user,STICKERSET_INVALID
 messages.updateDialogFilter,user,
@@ -320,7 +321,7 @@ phone.discardCall,user,CALL_ALREADY_ACCEPTED CALL_PEER_INVALID
 phone.discardGroupCallRequest,user,GROUPCALL_ALREADY_DISCARDED
 phone.editGroupCallParticipant,user,USER_VOLUME_INVALID
 phone.getCallConfig,user,
-phone.inviteToGroupCall,user,GROUPCALL_FORBIDDEN USER_ALREADY_INVITED INVITE_FORBIDDEN_WITH_JOINAS
+phone.inviteToGroupCall,user,GROUPCALL_FORBIDDEN INVITE_FORBIDDEN_WITH_JOINAS USER_ALREADY_INVITED
 phone.joinGroupCall,user,GROUPCALL_ADD_PARTICIPANTS_FAILED GROUPCALL_SSRC_DUPLICATE_MUCH
 phone.joinGroupCallPresentation,user,PARTICIPANT_JOIN_MISSING
 phone.receivedCall,user,CALL_ALREADY_DECLINED CALL_PEER_INVALID
@@ -331,7 +332,7 @@ phone.toggleGroupCallSettings,user,GROUPCALL_NOT_MODIFIED
 photos.deletePhotos,user,
 photos.getUserPhotos,both,MAX_ID_INVALID USER_ID_INVALID
 photos.updateProfilePhoto,user,PHOTO_ID_INVALID
-photos.uploadProfilePhoto,user,ALBUM_PHOTOS_TOO_MANY FILE_PARTS_INVALID IMAGE_PROCESS_FAILED PHOTO_CROP_SIZE_SMALL PHOTO_EXT_INVALID VIDEO_FILE_INVALID
+photos.uploadProfilePhoto,user,ALBUM_PHOTOS_TOO_MANY FILE_PARTS_INVALID IMAGE_PROCESS_FAILED PHOTO_CROP_SIZE_SMALL PHOTO_EXT_INVALID STICKER_MIME_INVALID VIDEO_FILE_INVALID
 ping,both,
 reqDHParams,both,
 reqPq,both,
@@ -341,10 +342,10 @@ setClientDHParams,both,
 stats.getBroadcastStats,user,BROADCAST_REQUIRED CHAT_ADMIN_REQUIRED CHP_CALL_FAIL STATS_MIGRATE_X
 stats.getMegagroupStats,user,CHAT_ADMIN_REQUIRED MEGAGROUP_REQUIRED STATS_MIGRATE_X
 stats.loadAsyncGraph,user,GRAPH_INVALID_RELOAD GRAPH_OUTDATED_RELOAD
-stickers.addStickerToSet,bot,BOT_MISSING STICKERSET_INVALID STICKER_PNG_NOPNG STICKER_TGS_NOTGS
+stickers.addStickerToSet,both,BOT_MISSING STICKERSET_INVALID STICKER_PNG_NOPNG STICKER_TGS_NOTGS
 stickers.changeStickerPosition,bot,BOT_MISSING STICKER_INVALID
 stickers.checkShortName,user,SHORT_NAME_INVALID SHORT_NAME_OCCUPIED
-stickers.createStickerSet,bot,BOT_MISSING PACK_SHORT_NAME_INVALID PACK_SHORT_NAME_OCCUPIED PEER_ID_INVALID SHORTNAME_OCCUPY_FAILED STICKERS_EMPTY STICKER_EMOJI_INVALID STICKER_FILE_INVALID STICKER_PNG_DIMENSIONS STICKER_PNG_NOPNG STICKER_TGS_NOTGS STICKER_THUMB_PNG_NOPNG STICKER_THUMB_TGS_NOTGS USER_ID_INVALID
+stickers.createStickerSet,both,BOT_MISSING PACK_SHORT_NAME_INVALID PACK_SHORT_NAME_OCCUPIED PEER_ID_INVALID SHORTNAME_OCCUPY_FAILED STICKERS_EMPTY STICKER_EMOJI_INVALID STICKER_FILE_INVALID STICKER_PNG_DIMENSIONS STICKER_PNG_NOPNG STICKER_TGS_NOTGS STICKER_THUMB_PNG_NOPNG STICKER_THUMB_TGS_NOTGS USER_ID_INVALID
 stickers.removeStickerFromSet,bot,BOT_MISSING STICKER_INVALID
 stickers.setStickerSetThumb,bot,STICKER_THUMB_PNG_NOPNG STICKER_THUMB_TGS_NOTGS
 stickers.suggestShortName,user,TITLE_INVALID

telethon_generator/generators/docs.py

@@ -117,11 +117,11 @@ def _generate_index(folder, paths,
                          .replace(os.path.sep, '/').title())
 
         if bots_index:
-            docs.write_text('These are the methods that you may be able to '
+            docs.write_text('These are the requests that you may be able to '
                             'use as a bot. Click <a href="{}">here</a> to '
                             'view them all.'.format(INDEX))
         else:
-            docs.write_text('Click <a href="{}">here</a> to view the methods '
+            docs.write_text('Click <a href="{}">here</a> to view the requests '
                             'that you can use as a bot.'.format(BOT_INDEX))
         if namespaces:
             docs.write_title('Namespaces', level=3)
@@ -268,7 +268,7 @@ def _write_html_pages(tlobjects, methods, layer, input_res):
                     start = \
                         'Both users and bots <strong>may</strong> be able to'
 
-                docs.write_text('{} use this method. <a href="#examples">'
+                docs.write_text('{} use this request. <a href="#examples">'
                                 'See code examples.</a>'.format(start))
 
             # Write the code definition for this TLObject
@@ -287,7 +287,7 @@ def _write_html_pages(tlobjects, methods, layer, input_res):
                 # We assume it's a function returning a generic type
                 generic_arg = next((arg.name for arg in tlobject.args
                                     if arg.is_generic))
-                docs.write_text('This function returns the result of whatever '
+                docs.write_text('This request returns the result of whatever '
                                 'the result from invoking the request passed '
                                 'through <i>{}</i> is.'.format(generic_arg))
             else:
@@ -465,15 +465,15 @@ def _write_html_pages(tlobjects, methods, layer, input_res):
             docs.end_table()
 
             # List all the methods which return this type
-            docs.write_title('Methods returning this type', level=3)
+            docs.write_title('Requests returning this type', level=3)
             functions = type_to_functions.get(t, [])
             if not functions:
-                docs.write_text('No method returns this type.')
+                docs.write_text('No request returns this type.')
             elif len(functions) == 1:
-                docs.write_text('Only the following method returns this type.')
+                docs.write_text('Only the following request returns this type.')
             else:
                 docs.write_text(
-                    'The following %d methods return this type as a result.' %
+                    'The following %d requests return this type as a result.' %
                     len(functions)
                 )
 
@@ -484,7 +484,7 @@ def _write_html_pages(tlobjects, methods, layer, input_res):
             docs.end_table()
 
             # List all the methods which take this type as input
-            docs.write_title('Methods accepting this type as input', level=3)
+            docs.write_title('Requests accepting this type as input', level=3)
             other_methods = sorted(
                 (u for u in tlobjects
                  if any(a.type == t for a in u.args) and u.is_function),
@@ -492,13 +492,13 @@ def _write_html_pages(tlobjects, methods, layer, input_res):
             )
             if not other_methods:
                 docs.write_text(
-                    'No methods accept this type as an input parameter.')
+                    'No request accepts this type as an input parameter.')
             elif len(other_methods) == 1:
                 docs.write_text(
-                    'Only this method has a parameter with this type.')
+                    'Only this request has a parameter with this type.')
             else:
                 docs.write_text(
-                    'The following %d methods accept this type as an input '
+                    'The following %d requests accept this type as an input '
                     'parameter.' % len(other_methods))
 
             docs.begin_table(2)

telethon_generator/generators/tlobject.py

@@ -197,7 +197,7 @@ def _write_class_init(tlobject, kind, type_constructors, builder):
     if not tlobject.real_args:
         return
 
-    if any(a.name in __builtins__ for a in tlobject.real_args):
+    if any(hasattr(__builtins__, a.name) for a in tlobject.real_args):
         builder.writeln('# noinspection PyShadowingBuiltins')
 
     builder.writeln("def __init__({}):", ', '.join(['self'] + args))
@@ -667,6 +667,7 @@ def _write_all_tlobjects(tlobjects, layer, builder):
     builder.current_indent += 1
 
     # Fill the dictionary (0x1a2b3c4f: tl.full.type.path.Class)
+    tlobjects.sort(key=lambda x: x.name)
     for tlobject in tlobjects:
         builder.write('{:#010x}: ', tlobject.id)
         builder.write('functions' if tlobject.is_function else 'types')

telethon_generator/parsers/errors.py

@@ -44,6 +44,7 @@ class Error:
         # Should these be split into different files or doesn't really matter?
         # Telegram isn't exactly consistent with returned errors anyway.
         self.int_code = codes[0]
+        self.int_codes = codes
         self.str_code = name
         self.subclass = _get_class_name(codes[0])
         self.subclass_exists = abs(codes[0]) in KNOWN_BASE_CLASSES

telethon_generator/parsers/methods.py

@@ -9,6 +9,15 @@ class Usability(enum.Enum):
     BOT = 2
     BOTH = 4
 
+    @property
+    def key(self):
+        return {
+            Usability.UNKNOWN: 'unknown',
+            Usability.USER: 'user',
+            Usability.BOT: 'bot',
+            Usability.BOTH: 'both',
+        }[self]
+
 
 class MethodInfo:
     def __init__(self, name, usability, errors, friendly):

telethon_generator/parsers/tlobject/tlarg.py

@@ -96,7 +96,12 @@ class TLArg:
         :param generic_definition: Is the argument a generic definition?
                                    (i.e. {X:Type})
         """
-        self.name = 'is_self' if name == 'self' else name
+        if name == 'self':
+            self.name = 'is_self'
+        elif name == 'from':
+            self.name = 'from_'
+        else:
+            self.name = name
 
         # Default values
         self.is_vector = False
@@ -204,17 +209,21 @@ class TLArg:
         return real_type
 
     def __str__(self):
+        name = self.orig_name()
         if self.generic_definition:
-            return '{{{}:{}}}'.format(self.name, self.real_type())
+            return '{{{}:{}}}'.format(name, self.real_type())
         else:
-            return '{}:{}'.format(self.name, self.real_type())
+            return '{}:{}'.format(name, self.real_type())
 
     def __repr__(self):
         return str(self).replace(':date', ':int').replace('?date', '?int')
 
+    def orig_name(self):
+        return self.name.replace('is_self', 'self').strip('_')
+
     def to_dict(self):
         return {
-            'name': self.name.replace('is_self', 'self'),
+            'name': self.orig_name(),
             'type': re.sub(r'\bdate$', 'int', self.real_type())
         }
 

telethon_generator/syncerrors.py

@@ -0,0 +1,55 @@
+# Should be fed with the JSON obtained from https://core.telegram.org/api/errors#error-database
+import re
+import csv
+import sys
+import json
+from pathlib import Path
+
+sys.path.insert(0, '..')
+
+from telethon_generator.parsers.errors import parse_errors, Error
+from telethon_generator.parsers.methods import parse_methods, MethodInfo
+
+ERRORS = Path('data/errors.csv')
+METHODS = Path('data/methods.csv')
+FRIENDLY = Path('data/friendly.csv')
+
+
+def main():
+    new_errors = []
+    new_methods = []
+
+    self_errors = {e.str_code: e for e in parse_errors(ERRORS)}
+    self_methods = {m.name: m for m in parse_methods(METHODS, FRIENDLY, self_errors)}
+
+    tg_data = json.load(sys.stdin)
+
+    def get_desc(code):
+        return re.sub(r'\s*&\w+;\s*', '', (tg_data['descriptions'].get(code) or '').rstrip('.'))
+
+    for int_code, errors in tg_data['errors'].items():
+        int_code = int(int_code)  # json does not support non-string keys
+        for code, methods in errors.items():
+            if not re.match(r'\w+', code):
+                continue  # skip, full code is unknown (contains asterisk or is multiple words)
+            str_code = code.replace('%d', 'X')
+            if error := self_errors.get(str_code):
+                error.int_codes.append(int_code)  # de-duplicated once later
+                if not error.description:  # prefer our descriptions
+                    if not error.has_captures:  # need descriptions with specific text if error has captures
+                        error.description = get_desc(code)
+            else:
+                self_errors[str_code] = Error([int_code], str_code, get_desc(code))
+
+    new_errors.extend((e.str_code, ' '.join(map(str, sorted(set(e.int_codes)))), e.description) for e in self_errors.values())
+    new_methods.extend((m.name, m.usability.key, ' '.join(sorted(e.str_code for e in m.errors))) for m in self_methods.values())
+
+    csv.register_dialect('plain', lineterminator='\n')
+    with ERRORS.open('w', encoding='utf-8', newline='') as fd:
+        csv.writer(fd, 'plain').writerows((('name', 'codes', 'description'), *sorted(new_errors)))
+    with METHODS.open('w', encoding='utf-8', newline='') as fd:
+        csv.writer(fd, 'plain').writerows((('method', 'usability', 'errors'), *sorted(new_methods)))
+
+
+if __name__ == '__main__':
+    main()

tests/readthedocs/quick_references/test_client_reference.py

@@ -10,5 +10,5 @@ def test_all_methods_present(docs_dir):
     assert len(present_methods) > 0
     for name in dir(TelegramClient):
         attr = getattr(TelegramClient, name)
-        if callable(attr) and not name.startswith('_'):
+        if callable(attr) and not name.startswith('_') and name != 'sign_up':
             assert name in present_methods

tests/telethon/client/test_messages.py

@@ -1,8 +1,12 @@
 import inspect
+from unittest import mock
+from unittest.mock import MagicMock
 
 import pytest
 
 from telethon import TelegramClient
+from telethon.client import MessageMethods
+from telethon.tl.types import PeerChat, MessageMediaDocument, Message, MessageEntityBold
 
 
 @pytest.mark.asyncio
@@ -38,3 +42,43 @@ async def test_send_message_with_file_forwards_args():
 
     client = MockedClient()
     assert (await client.send_message('a', file='b', **arguments)) == sentinel
+
+
+class TestMessageMethods:
+    @pytest.mark.asyncio
+    @pytest.mark.parametrize(
+        'formatting_entities',
+        ([MessageEntityBold(offset=0, length=0)], None)
+    )
+    async def test_send_msg_and_file(self, formatting_entities):
+        async def async_func(result): # AsyncMock was added only in 3.8
+            return result
+        msg_methods = MessageMethods()
+        expected_result = Message(
+            id=0, peer_id=PeerChat(chat_id=0), message='', date=None,
+        )
+        entity = 'test_entity'
+        message = Message(
+            id=1, peer_id=PeerChat(chat_id=0), message='expected_caption', date=None,
+            entities=[MessageEntityBold(offset=9, length=9)],
+        )
+        media_file = MessageMediaDocument()
+
+        with mock.patch.object(
+            target=MessageMethods, attribute='send_file',
+            new=MagicMock(return_value=async_func(expected_result)), create=True,
+        ) as mock_obj:
+            result = await msg_methods.send_message(
+                entity=entity, message=message, file=media_file,
+                formatting_entities=formatting_entities,
+            )
+            mock_obj.assert_called_once_with(
+                entity, media_file, caption=message.message,
+                formatting_entities=formatting_entities or message.entities,
+                reply_to=None, silent=None, attributes=None, parse_mode=(),
+                force_document=False, thumb=None, buttons=None,
+                clear_draft=False, schedule=None, supports_streaming=False,
+                comment_to=None, background=None, nosound_video=None,
+                send_as=None, message_effect_id=None,
+            )
+            assert result == expected_result

tests/telethon/extensions/test_html.py

@@ -53,6 +53,22 @@ def test_entities_together():
     assert text == original
 
 
+def test_nested_entities():
+    """
+    Test that an entity nested inside another one behaves well.
+    """
+    original = '<a href="https://example.com"><strong>Example</strong></a>'
+    original_entities = [MessageEntityTextUrl(0, 7, url='https://example.com'), MessageEntityBold(0, 7)]
+    stripped = 'Example'
+
+    text, entities = html.parse(original)
+    assert text == stripped
+    assert entities == original_entities
+
+    text = html.unparse(text, entities)
+    assert text == original
+
+
 def test_offset_at_emoji():
     """
     Tests that an entity starting at a emoji preserves the emoji.

tests/telethon/extensions/test_markdown.py

@@ -53,6 +53,21 @@ def test_entities_together():
     assert text == original
 
 
+def test_nested_entities():
+    """
+    Test that an entity nested inside another one behaves well.
+    """
+    original = '**[Example](https://example.com)**'
+    stripped = 'Example'
+
+    text, entities = markdown.parse(original)
+    assert text == stripped
+    assert entities == [MessageEntityBold(0, 7), MessageEntityTextUrl(0, 7, url='https://example.com')]
+
+    text = markdown.unparse(text, entities)
+    assert text == original
+
+
 def test_offset_at_emoji():
     """
     Tests that an entity starting at a emoji preserves the emoji.

tests/telethon/test_utils.py

@@ -34,9 +34,6 @@ def test_private_get_extension():
 
     assert utils._get_extension('foo.bar.baz') == '.baz'
     assert utils._get_extension(pathlib.Path('foo.bar.baz')) == '.baz'
-    assert utils._get_extension(png_header) == '.png'
-    assert utils._get_extension(png_buffer) == '.png'
-    assert utils._get_extension(png_buffer) == '.png'  # make sure it did seek back
     assert utils._get_extension(CustomFd('foo.bar.baz')) == '.baz'
 
     # Negative cases