Merge pull request #1 from LonamiWebs/master

merge upstream to master

.github/ISSUE_TEMPLATE.md

@@ -1,9 +1,10 @@
 <!--
-0. The library is Python 3.x, not Python 2.x.
-1. If you're posting an issue, make sure it's a bug in the library, not in your code.
-2. If you're posting a question, make sure you have read and tried enough things first.
-3. Show as much information as possible, including your failed attempts, and the full console output (to include the whole traceback with line numbers).
-4. Good looking issues are a lot more appealing. If you need help check out https://guides.github.com/features/mastering-markdown/.
+0. The library is Python >= 3.4, not Python 2.x.
+1. If you have a QUESTION, ask it on @TelethonChat (Telegram) or StackOverflow, not here. It will be closed immediatly with no explanation if you post it here.
+2. If you have an ISSUE or you are experiencing strange behaviour, make sure you're using the latest version (pip install -U telethon), and post as much information as possible here. Enhancement suggestions are welcome too.
 
-You may also want to watch "How (not) to ask a technical question" over https://youtu.be/53zkBvL4ZB4
+If you paste code, please put it between three backticks (`):
+```python
+code here
+```
 -->

.gitignore

@@ -1,5 +1,6 @@
 # Docs
 _build/
+docs/
 
 # Generated code
 telethon/tl/functions/

README.rst

@@ -52,7 +52,7 @@ Doing stuff
   client.send_file('username', '/home/myself/Pictures/holidays.jpg')
 
   client.download_profile_photo('me')
-  messages = client.get_message_history('username')
+  messages = client.get_messages('username')
   client.download_media(messages[0])
 
 

docs/.gitignore

@@ -1 +0,0 @@
-generated/

readthedocs/conf.py

@@ -42,6 +42,9 @@ extensions = [
     'custom_roles'
 ]
 
+# Change the default role so we can avoid prefixing everything with :obj:
+default_role = "py:obj"
+
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 
@@ -157,7 +160,7 @@ latex_elements = {
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
     (master_doc, 'Telethon.tex', 'Telethon Documentation',
-     'Jeff', 'manual'),
+     author, 'manual'),
 ]
 
 

readthedocs/extra/advanced-usage/accessing-the-full-api.rst

@@ -4,11 +4,17 @@
 Accessing the Full API
 ======================
 
+.. important::
 
-The ``TelegramClient`` doesn't offer a method for every single request
-the Telegram API supports. However, it's very simple to *call* or *invoke*
-any request. Whenever you need something, don't forget to `check the
-documentation`__ and look for the `method you need`__. There you can go
+    While you have access to this, you should always use the friendly
+    methods listed on :ref:`telethon-package` unless you have a better
+    reason not to, like a method not existing or you wanting more control.
+
+
+The `telethon.telegram_client.TelegramClient` doesn't offer a method for every
+single request the Telegram API supports. However, it's very simple to *call*
+or *invoke* any request. Whenever you need something, don't forget to `check
+the documentation`__ and look for the `method you need`__. There you can go
 through a sorted list of everything you can do.
 
 
@@ -24,9 +30,9 @@ You should also refer to the documentation to see what the objects
 (constructors) Telegram returns look like. Every constructor inherits
 from a common type, and that's the reason for this distinction.
 
-Say ``client.send_message()`` didn't exist, we could use the `search`__
-to look for "message". There we would find :tl:`SendMessageRequest`,
-which we can work with.
+Say `telethon.telegram_client.TelegramClient.send_message` didn't exist,
+we could use the `search`__ to look for "message". There we would find
+:tl:`SendMessageRequest`, which we can work with.
 
 Every request is a Python class, and has the parameters needed for you
 to invoke it. You can also call ``help(request)`` for information on
@@ -57,7 +63,7 @@ construct one, for instance:
 
         peer = InputPeerUser(user_id, user_hash)
 
-Or we call ``.get_input_entity()``:
+Or we call `telethon.telegram_client.TelegramClient.get_input_entity()`:
 
     .. code-block:: python
 
@@ -68,7 +74,7 @@ When you're going to invoke an API method, most require you to pass an
 ``.get_input_entity()`` is more straightforward (and often
 immediate, if you've seen the user before, know their ID, etc.).
 If you also need to have information about the whole user, use
-``.get_entity()`` instead:
+`telethon.telegram_client.TelegramClient.get_entity()` instead:
 
     .. code-block:: python
 
@@ -77,7 +83,7 @@ If you also need to have information about the whole user, use
 In the later case, when you use the entity, the library will cast it to
 its "input" version for you. If you already have the complete user and
 want to cache its input version so the library doesn't have to do this
-every time its used, simply call ``.get_input_peer``:
+every time its used, simply call `telethon.utils.get_input_peer`:
 
     .. code-block:: python
 
@@ -137,4 +143,4 @@ This can further be simplified to:
 
 __ https://lonamiwebs.github.io/Telethon
 __ https://lonamiwebs.github.io/Telethon/methods/index.html
-__ https://lonamiwebs.github.io/Telethon/?q=message
+__ https://lonamiwebs.github.io/Telethon/?q=message&redirect=no

readthedocs/extra/advanced-usage/sessions.rst

@@ -7,7 +7,15 @@ Session Files
 The first parameter you pass to the constructor of the ``TelegramClient`` is
 the ``session``, and defaults to be the session name (or full path). That is,
 if you create a ``TelegramClient('anon')`` instance and connect, an
-``anon.session`` file will be created on the working directory.
+``anon.session`` file will be created in the working directory.
+
+Note that if you pass a string it will be a file in the current working
+directory, although you can also pass absolute paths.
+
+The session file contains enough information for you to login without
+re-sending the code, so if you have to enter the code more than once,
+maybe you're changing the working directory, renaming or removing the
+file, or using random names.
 
 These database files using ``sqlite3`` contain the required information to
 talk to the Telegram servers, such as to which IP the client should connect,

readthedocs/extra/advanced-usage/update-modes.rst

@@ -44,7 +44,7 @@ That's it! This is the old way to listen for raw updates, with no further
 processing. If this feels annoying for you, remember that you can always
 use :ref:`working-with-updates` but maybe use this for some other cases.
 
-Now let's do something more interesting. Every time an user talks to use,
+Now let's do something more interesting. Every time an user talks to us,
 let's reply to them with the same text reversed:
 
     .. code-block:: python

readthedocs/extra/basic/creating-a-client.rst

@@ -39,6 +39,13 @@ Note that ``'some_name'`` will be used to save your session (persistent
 information such as access key and others) as ``'some_name.session'`` in
 your disk. This is by default a database file using Python's ``sqlite3``.
 
+.. note::
+
+    It's important that the library always accesses the same session file so
+    that you don't need to re-send the code over and over again. By default it
+    creates the file in your working directory, but absolute paths work too.
+
+
 Before using the client, you must be connected to Telegram.
 Doing so is very easy:
 
@@ -150,36 +157,44 @@ The mentioned ``.start()`` method will handle this for you as well, but
 you must set the ``password=`` parameter beforehand (it won't be asked).
 
 If you don't have 2FA enabled, but you would like to do so through the library,
-take as example the following code snippet:
+use ``client.edit_2fa()``.
+Be sure to know what you're doing when using this function and
+you won't run into any problems.
+Take note that if you want to set only the email/hint and leave 
+the current password unchanged, you need to "redo" the 2fa.
+
+See the examples below:
 
     .. code-block:: python
 
-        import os
-        from hashlib import sha256
-        from telethon.tl.functions import account
-        from telethon.tl.types.account import PasswordInputSettings
-
-        new_salt = client(account.GetPasswordRequest()).new_salt
-        salt = new_salt + os.urandom(8)  # new random salt
-
-        pw = 'secret'.encode('utf-8')  # type your new password here
-        hint = 'hint'
-
-        pw_salted = salt + pw + salt
-        pw_hash = sha256(pw_salted).digest()
-
-        result = client(account.UpdatePasswordSettingsRequest(
-            current_password_hash=salt,
-            new_settings=PasswordInputSettings(
-                new_salt=salt,
-                new_password_hash=pw_hash,
-                hint=hint
-            )
-        ))
-
-Thanks to `Issue 259 <https://github.com/LonamiWebs/Telethon/issues/259>`_
-for the tip!
-
+        from telethon.errors import EmailUnconfirmedError
+        
+        # Sets 2FA password for first time:
+        client.edit_2fa(new_password='supersecurepassword')
+        
+        # Changes password:
+        client.edit_2fa(current_password='supersecurepassword', 
+                        new_password='changedmymind')
+        
+        # Clears current password (i.e. removes 2FA):
+        client.edit_2fa(current_password='changedmymind', new_password=None)
+        
+        # Sets new password with recovery email:
+        try:
+            client.edit_2fa(new_password='memes and dreams', 
+                            email='JohnSmith@example.com')
+            # Raises error (you need to check your email to complete 2FA setup.)
+        except EmailUnconfirmedError:
+            # You can put email checking code here if desired.
+            pass
+            
+        # Also take note that unless you remove 2FA or explicitly
+        # give email parameter again it will keep the last used setting
+        
+        # Set hint after already setting password:
+        client.edit_2fa(current_password='memes and dreams',
+                        new_password='memes and dreams',
+                        hint='It keeps you alive')
 
 __ https://github.com/Anorov/PySocks#installation
 __ https://github.com/Anorov/PySocks#usage-1

readthedocs/extra/basic/entities.rst

@@ -20,6 +20,19 @@ in response to certain methods, such as :tl:`GetUsersRequest`.
     or even entire :tl:`User`, :tl:`Chat` and :tl:`Channel` objects and even
     phone numbers from people you have in your contacts.
 
+    To "encounter" an ID, you would have to "find it" like you would in the
+    normal app. If the peer is in your dialogs, you would need to
+    `client.get_dialogs() <telethon.telegram_client.TelegramClient.get_dialogs>`.
+    If the peer is someone in a group, you would similarly
+    `client.get_participants(group) <telethon.telegram_client.TelegramClient.get_participants>`.
+
+    Once you have encountered an ID, the library will (by default) have saved
+    their ``access_hash`` for you, which is needed to invoke most methods.
+    This is why sometimes you might encounter this error when working with
+    the library. You should ``except ValueError`` and run code that you know
+    should work to find the entity.
+
+
 Getting entities
 ****************
 
@@ -32,7 +45,7 @@ you're able to just do this:
         # Dialogs are the "conversations you have open".
         # This method returns a list of Dialog, which
         # has the .entity attribute and other information.
-        dialogs = client.get_dialogs(limit=200)
+        dialogs = client.get_dialogs()
 
         # All of these work and do the same.
         lonami = client.get_entity('lonami')
@@ -44,27 +57,18 @@ you're able to just do this:
         contact = client.get_entity('+34xxxxxxxxx')
         friend  = client.get_entity(friend_id)
 
-        # Using Peer/InputPeer (note that the API may return these)
-        # users, chats and channels may all have the same ID, so it's
-        # necessary to wrap (at least) chat and channels inside Peer.
-        #
-        # NOTICE how the IDs *must* be wrapped inside a Peer() so the
-        # library knows their type.
+        # Getting entities through their ID (User, Chat or Channel)
+        entity = client.get_entity(some_id)
+
+        # You can be more explicit about the type for said ID by wrapping
+        # it inside a Peer instance. This is recommended but not necessary.
         from telethon.tl.types import PeerUser, PeerChat, PeerChannel
+
         my_user    = client.get_entity(PeerUser(some_id))
         my_chat    = client.get_entity(PeerChat(some_id))
         my_channel = client.get_entity(PeerChannel(some_id))
 
 
-.. warning::
-
-    As it has been mentioned already, getting the entity of a channel
-    through e.g. ``client.get_entity(channel id)`` will **not** work.
-    You would use ``client.get_entity(types.PeerChannel(channel id))``.
-    Remember that supergroups are channels and normal groups are chats.
-    This is a common mistake!
-
-
 All methods in the :ref:`telegram-client` call ``.get_input_entity()`` prior
 to sending the requst to save you from the hassle of doing so manually.
 That way, convenience calls such as ``client.send_message('lonami', 'hi!')``
@@ -127,3 +131,21 @@ library, the raw requests you make to the API are also able to call
 The library will call the ``.resolve()`` method of the request, which will
 resolve ``'username'`` with the appropriated :tl:`InputPeer`. Don't worry if
 you don't get this yet, but remember some of the details here are important.
+
+
+Full entities
+*************
+
+In addition to :tl:`PeerUser`, :tl:`InputPeerUser`, :tl:`User` (and its
+variants for chats and channels), there is also the concept of :tl:`UserFull`.
+
+This full variant has additional information such as whether the user is
+blocked, its notification settings, the bio or about of the user, etc.
+
+There is also :tl:`messages.ChatFull` which is the equivalent of full entities
+for chats and channels, with also the about section of the channel. Note that
+the ``users`` field only contains bots for the channel (so that clients can
+suggest commands to use).
+
+You can get both of these by invoking :tl:`GetFullUser`, :tl:`GetFullChat`
+and :tl:`GetFullChannel` respectively.

readthedocs/extra/basic/getting-started.rst

@@ -48,7 +48,7 @@ Basic Usage
 
        # Retrieving messages from a chat
        from telethon import utils
-       for message in client.get_message_history('username', limit=10):
+       for message in client.iter_messages('username', limit=10):
            print(utils.get_display_name(message.sender), message.message)
 
        # Listing all the dialogs (conversations you have open)
@@ -60,11 +60,13 @@ Basic Usage
 
        # Once you have a message with .media (if message.media)
        # you can download it using client.download_media():
-       messages = client.get_message_history('username')
+       messages = client.get_messages('username')
        client.download_media(messages[0])
 
    **More details**: :ref:`telegram-client`
 
+   See :ref:`telethon-package` for all available friendly methods.
+
 
 Handling Updates
 ****************

readthedocs/extra/basic/installation.rst

@@ -54,12 +54,11 @@ Manual Installation
 
 3. Enter the cloned repository: ``cd Telethon``
 
-4. Run the code generator: ``python3 setup.py gen_tl``
+4. Run the code generator: ``python3 setup.py gen tl errors``
 
 5. Done!
 
-To generate the `method documentation`__, ``cd docs`` and then
-``python3 generate.py`` (if some pages render bad do it twice).
+To generate the `method documentation`__, ``python3 setup.py gen docs``.
 
 
 Optional dependencies

readthedocs/extra/basic/telegram-client.rst

@@ -10,8 +10,10 @@ Introduction
 
 .. note::
 
-    Check the :ref:`telethon-package` if you're looking for the methods
-    reference instead of this tutorial.
+    Make sure to use the friendly methods described in :ref:`telethon-package`!
+    This section is just an introduction to using the client, but all the
+    available methods are in the :ref:`telethon-package` reference, including
+    detailed descriptions to what they do.
 
 The ``TelegramClient`` is the central class of the library, the one
 you will be using most of the time. For this reason, it's important
@@ -58,11 +60,16 @@ Many other common methods for quick scripts are also available:
         # Note that you can use 'me' or 'self' to message yourself
         client.send_message('username', 'Hello World from Telethon!')
 
+        # .send_message's parse mode defaults to markdown, so you
+        # can use **bold**, __italics__, [links](https://example.com), `code`,
+        # and even [mentions](@username)/[mentions](tg://user?id=123456789)
+        client.send_message('username', '**Using** __markdown__ `too`!')
+
         client.send_file('username', '/home/myself/Pictures/holidays.jpg')
 
         # The utils package has some goodies, like .get_display_name()
         from telethon import utils
-        for message in client.get_message_history('username', limit=10):
+        for message in client.iter_messages('username', limit=10):
             print(utils.get_display_name(message.sender), message.message)
 
         # Dialogs are the conversations you have open
@@ -83,15 +90,16 @@ a single line.
 Available methods
 *****************
 
-This page lists all the "handy" methods available for you to use in the
-``TelegramClient`` class. These are simply wrappers around the "raw"
-Telegram API, making it much more manageable and easier to work with.
+The :ref:`reference <telethon-package>` lists all the "handy" methods
+available for you to use in the ``TelegramClient`` class. These are simply
+wrappers around the "raw" Telegram API, making it much more manageable and
+easier to work with.
 
 Please refer to :ref:`accessing-the-full-api` if these aren't enough,
 and don't be afraid to read the source code of the InteractiveTelegramClient_
 or even the TelegramClient_ itself to learn how it works.
 
-To see the methods available in the client, see :ref:`telethon-package`.
+See the mentioned :ref:`telethon-package` to find the available methods.
 
 .. _InteractiveTelegramClient: https://github.com/LonamiWebs/Telethon/blob/master/telethon_examples/interactive_telegram_client.py
 .. _TelegramClient: https://github.com/LonamiWebs/Telethon/blob/master/telethon/telegram_client.py

readthedocs/extra/changelog.rst

@@ -14,6 +14,242 @@ it can take advantage of new goodies!
 .. contents:: List of All Versions
 
 
+Custom Message class (v0.19.1)
+==============================
+
+*Published at 2018/06/03*
+
++-----------------------+
+| Scheme layer used: 80 |
++-----------------------+
+
+
+This update brings a new `telethon.tl.custom.message.Message` object!
+
+All the methods in the `telethon.telegram_client.TelegramClient` that
+used to return a :tl:`Message` will now return this object instead, which
+means you can do things like the following:
+
+.. code-block:: python
+
+    msg = client.send_message(chat, 'Hello!')
+    msg.edit('Hello there!')
+    msg.reply('Good day!')
+    print(msg.sender)
+
+Refer to its documentation to see all you can do, again, click
+`telethon.tl.custom.message.Message` to go to its page.
+
+
+Breaking Changes
+~~~~~~~~~~~~~~~~
+
+- The `telethon.network.connection.common.Connection` class is now an ABC,
+  and the old ``ConnectionMode`` is now gone. Use a specific connection (like
+  `telethon.network.connection.tcpabridged.ConnectionTcpAbridged`) instead.
+
+Additions
+~~~~~~~~~
+
+- You can get messages by their ID with
+  `telethon.telegram_client.TelegramClient.get_messages`'s ``ids`` parameter:
+
+  .. code-block:: python
+
+      message = client.get_messages(chats, ids=123)  # Single message
+      message_list = client.get_messages(chats, ids=[777, 778])  # Multiple
+
+- More convenience properties for `telethon.tl.custom.dialog.Dialog`.
+- New default `telethon.telegram_client.TelegramClient.parse_mode`.
+- You can edit the media of messages that already have some media.
+- New dark theme in the online ``tl`` reference, check it out at
+  https://lonamiwebs.github.io/Telethon/.
+
+Bug fixes
+~~~~~~~~~
+
+- Some IDs start with ``1000`` and these would be wrongly treated as channels.
+- Some short usernames like ``@vote`` were being ignored.
+- `telethon.telegram_client.TelegramClient.iter_messages`'s ``from_user``
+  was failing if no filter had been set.
+- `telethon.telegram_client.TelegramClient.iter_messages`'s ``min_id/max_id``
+  was being ignored by Telegram. This is now worked around.
+- `telethon.telegram_client.TelegramClient.catch_up` would fail with empty
+  states.
+- `telethon.events.newmessage.NewMessage` supports ``incoming=False``
+  to indicate ``outgoing=True``.
+
+Enhancements
+~~~~~~~~~~~~
+
+- You can now send multiple requests at once while preserving the order:
+
+  .. code-block:: python
+
+      from telethon.tl.functions.messages import SendMessageRequest
+      client([SendMessageRequest(chat, 'Hello 1!'),
+              SendMessageRequest(chat, 'Hello 2!')], ordered=True)
+
+Internal changes
+~~~~~~~~~~~~~~~~
+
+- ``without rowid`` is not used in SQLite anymore.
+- Unboxed serialization would fail.
+- Different default limit for ``iter_messages`` and ``get_messages``.
+- Some clean-up in the ``telethon_generator/`` package.
+
+
+Catching up on Updates (v0.19)
+==============================
+
+*Published at 2018/05/07*
+
++-----------------------+
+| Scheme layer used: 76 |
++-----------------------+
+
+This update prepares the library for catching up with updates with the new
+`telethon.telegram_client.TelegramClient.catch_up` method. This feature needs
+more testing, but for now it will let you "catch up" on some old updates that
+occurred while the library was offline, and brings some new features and bug
+fixes.
+
+
+Additions
+~~~~~~~~~
+
+- Add ``search``, ``filter`` and ``from_user`` parameters to
+  `telethon.telegram_client.TelegramClient.iter_messages`.
+- `telethon.telegram_client.TelegramClient.download_file` now
+  supports a ``None`` path to return the file in memory and
+  return its ``bytes``.
+- Events now have a ``.original_update`` field.
+
+Bug fixes
+~~~~~~~~~
+
+- Fixed a race condition when receiving items from the network.
+- A disconnection is made when "retries reached 0". This hasn't been
+  tested but it might fix the bug.
+- ``reply_to`` would not override :tl:`Message` object's reply value.
+- Add missing caption when sending :tl:`Message` with media.
+
+Enhancements
+~~~~~~~~~~~~
+
+- Retry automatically on ``RpcCallFailError``. This error happened a lot
+  when iterating over many messages, and retrying often fixes it.
+- Faster `telethon.telegram_client.TelegramClient.iter_messages` by
+  sleeping only as much as needed.
+- `telethon.telegram_client.TelegramClient.edit_message` now supports
+  omitting the entity if you pass a :tl:`Message`.
+- `telethon.events.raw.Raw` can now be filtered by type.
+
+Internal changes
+~~~~~~~~~~~~~~~~
+
+- The library now distinguishes between MTProto and API schemas.
+- :tl:`State` is now persisted to the session file.
+- Connection won't retry forever.
+- Fixed some errors and cleaned up the generation of code.
+- Fixed typos and enhanced some documentation in general.
+- Add auto-cast for :tl:`InputMessage` and :tl:`InputLocation`.
+
+
+Pickle-able objects (v0.18.3)
+=============================
+
+*Published at 2018/04/15*
+
+
+Now you can use Python's ``pickle`` module to serialize ``RPCError`` and
+any other ``TLObject`` thanks to **@vegeta1k95**! A fix that was fairly
+simple, but still might be useful for many people.
+
+As a side note, the documentation at https://lonamiwebs.github.io/Telethon
+now lists known ``RPCError`` for all requests, so you know what to expect.
+This required a major rewrite, but it was well worth it!
+
+Breaking changes
+~~~~~~~~~~~~~~~~
+
+- `telethon.telegram_client.TelegramClient.forward_messages` now returns
+  a single item instead of a list if the input was also a single item.
+
+Additions
+~~~~~~~~~
+
+- New `telethon.events.messageread.MessageRead` event, to find out when
+  and who read which messages as soon as it happens.
+- Now you can access ``.chat_id`` on all events and ``.sender_id`` on some.
+
+Bug fixes
+~~~~~~~~~
+
+- Possibly fix some bug regarding lost ``GzipPacked`` requests.
+- The library now uses the "real" layer 75, hopefully.
+- Fixed ``.entities`` name collision on updates by making it private.
+- ``AUTH_KEY_DUPLICATED`` is handled automatically on connection.
+- Markdown parser's offset uses ``match.start()`` to allow custom regex.
+- Some filter types (as a type) were not supported by
+  `telethon.telegram_client.TelegramClient.iter_participants`.
+- `telethon.telegram_client.TelegramClient.remove_event_handler` works.
+- `telethon.telegram_client.TelegramClient.start` works on all terminals.
+- :tl:`InputPeerSelf` case was missing from
+  `telethon.telegram_client.TelegramClient.get_input_entity`.
+
+Enhancements
+~~~~~~~~~~~~
+
+- The ``parse_mode`` for messages now accepts a callable.
+- `telethon.telegram_client.TelegramClient.download_media` accepts web previews.
+- `telethon.tl.custom.dialog.Dialog` instances can now be casted into
+  :tl:`InputPeer`.
+- Better logging when reading packages "breaks".
+- Better and more powerful ``setup.py gen`` command.
+
+Internal changes
+~~~~~~~~~~~~~~~~
+
+- The library won't call ``.get_dialogs()`` on entity not found. Instead,
+  it will ``raise ValueError()`` so you can properly ``except`` it.
+- Several new examples and updated documentation.
+- ``py:obj`` is the default Sphinx's role which simplifies ``.rst`` files.
+- ``setup.py`` now makes use of ``python_requires``.
+- Events now live in separate files.
+- Other minor changes.
+
+
+Several bug fixes (v0.18.2)
+===========================
+
+*Published at 2018/03/27*
+
+Just a few bug fixes before they become too many.
+
+Additions
+~~~~~~~~~
+
+- Getting an entity by its positive ID should be enough, regardless of their
+  type (whether it's an ``User``, a ``Chat`` or a ``Channel``). Although
+  wrapping them inside a ``Peer`` is still recommended, it's not necessary.
+- New ``client.edit_2fa`` function to change your Two Factor Authentication
+  settings.
+- ``.stringify()`` and string representation for custom ``Dialog/Draft``.
+
+Bug fixes
+~~~~~~~~~
+
+- Some bug regarding ``.get_input_peer``.
+- ``events.ChatAction`` wasn't picking up all the pins.
+- ``force_document=True`` was being ignored for albums.
+- Now you're able to send ``Photo`` and ``Document`` as files.
+- Wrong access to a member on chat forbidden error for ``.get_participants``.
+  An empty list is returned instead.
+- ``me/self`` check for ``.get[_input]_entity`` has been moved up so if
+  someone has "me" or "self" as their name they won't be retrieved.
+
+
 Iterator methods (v0.18.1)
 ==========================
 

readthedocs/extra/developing/project-structure.rst

@@ -37,7 +37,11 @@ Auto-generated code
 *******************
 
 The files under ``telethon_generator/`` are used to generate the code
-that gets placed under ``telethon/tl/``. The ``TLGenerator`` takes in a
-``.tl`` file, and spits out the generated classes which represent, as
-Python classes, the request and types defined in the ``.tl`` file. It
-also constructs an index so that they can be imported easily.
+that gets placed under ``telethon/tl/``. The parsers take in files in
+a specific format (such as ``.tl`` for objects and ``.json`` for errors)
+and spit out the generated classes which represent, as Python classes,
+the request and types defined in the ``.tl`` file. It also constructs
+an index so that they can be imported easily.
+
+Custom documentation can also be generated to easily navigate through
+the vast amount of items offered by the API.

readthedocs/extra/examples/bots.rst

@@ -13,7 +13,7 @@ Talking to Inline Bots
 
 You can query an inline bot, such as `@VoteBot`__ (note, *query*,
 not *interact* with a voting message), by making use of the
-`GetInlineBotResultsRequest`__ request:
+:tl:`GetInlineBotResultsRequest` request:
 
     .. code-block:: python
 
@@ -24,7 +24,7 @@ not *interact* with a voting message), by making use of the
         ))
 
 And you can select any of their results by using
-`SendInlineBotResultRequest`__:
+:tl:`SendInlineBotResultRequest`:
 
     .. code-block:: python
 
@@ -41,7 +41,7 @@ Talking to Bots with special reply markup
 *****************************************
 
 To interact with a message that has a special reply markup, such as
-`@VoteBot`__ polls, you would use `GetBotCallbackAnswerRequest`__:
+`@VoteBot`__ polls, you would use :tl:`GetBotCallbackAnswerRequest`:
 
     .. code-block:: python
 
@@ -58,7 +58,4 @@ show it visually (button rows, and buttons within each row, each with
 its own data).
 
 __ https://t.me/vote
-__ https://lonamiwebs.github.io/Telethon/methods/messages/get_inline_bot_results.html
-__ https://lonamiwebs.github.io/Telethon/methods/messages/send_inline_bot_result.html
-__ https://lonamiwebs.github.io/Telethon/methods/messages/get_bot_callback_answer.html
 __ https://t.me/vote

readthedocs/extra/examples/chats-and-channels.rst

@@ -11,17 +11,16 @@ Working with Chats and Channels
 Joining a chat or channel
 *************************
 
-Note that `Chat`__\ s are normal groups, and `Channel`__\ s are a
-special form of `Chat`__\ s,
-which can also be super-groups if their ``megagroup`` member is
-``True``.
+Note that :tl:`Chat` are normal groups, and :tl:`Channel` are a
+special form of ``Chat``, which can also be super-groups if
+their ``megagroup`` member is ``True``.
 
 
 Joining a public channel
 ************************
 
 Once you have the :ref:`entity <entities>` of the channel you want to join
-to, you can make use of the `JoinChannelRequest`__ to join such channel:
+to, you can make use of the :tl:`JoinChannelRequest` to join such channel:
 
     .. code-block:: python
 
@@ -36,6 +35,9 @@ to, you can make use of the `JoinChannelRequest`__ to join such channel:
 For more on channels, check the `channels namespace`__.
 
 
+__ https://lonamiwebs.github.io/Telethon/methods/channels/index.html
+
+
 Joining a private chat or channel
 *********************************
 
@@ -44,7 +46,7 @@ If all you have is a link like this one:
 enough information to join! The part after the
 ``https://t.me/joinchat/``, this is, ``AAAAAFFszQPyPEZ7wgxLtd`` on this
 example, is the ``hash`` of the chat or channel. Now you can use
-`ImportChatInviteRequest`__ as follows:
+:tl:`ImportChatInviteRequest` as follows:
 
     .. code-block:: python
 
@@ -56,21 +58,23 @@ Adding someone else to such chat or channel
 *******************************************
 
 If you don't want to add yourself, maybe because you're already in,
-you can always add someone else with the `AddChatUserRequest`__, which
-use is very straightforward, or `InviteToChannelRequest`__ for channels:
+you can always add someone else with the :tl:`AddChatUserRequest`, which
+use is very straightforward, or :tl:`InviteToChannelRequest` for channels:
 
     .. code-block:: python
 
         # For normal chats
         from telethon.tl.functions.messages import AddChatUserRequest
 
+        # Note that ``user_to_add`` is NOT the name of the parameter.
+        # It's the user you want to add (``user_id=user_to_add``).
         client(AddChatUserRequest(
             chat_id,
             user_to_add,
             fwd_limit=10  # Allow the user to see the 10 last messages
         ))
 
-        # For channels
+        # For channels (which includes megagroups)
         from telethon.tl.functions.channels import InviteToChannelRequest
 
         client(InviteToChannelRequest(
@@ -79,34 +83,31 @@ use is very straightforward, or `InviteToChannelRequest`__ for channels:
         ))
 
 
-
 Checking a link without joining
 *******************************
 
 If you don't need to join but rather check whether it's a group or a
-channel, you can use the `CheckChatInviteRequest`__, which takes in
+channel, you can use the :tl:`CheckChatInviteRequest`, which takes in
 the hash of said channel or group.
 
-__ https://lonamiwebs.github.io/Telethon/constructors/chat.html
-__ https://lonamiwebs.github.io/Telethon/constructors/channel.html
-__ https://lonamiwebs.github.io/Telethon/types/chat.html
-__ https://lonamiwebs.github.io/Telethon/methods/channels/join_channel.html
-__ https://lonamiwebs.github.io/Telethon/methods/channels/index.html
-__ https://lonamiwebs.github.io/Telethon/methods/messages/import_chat_invite.html
-__ https://lonamiwebs.github.io/Telethon/methods/messages/add_chat_user.html
-__ https://lonamiwebs.github.io/Telethon/methods/channels/invite_to_channel.html
-__ https://lonamiwebs.github.io/Telethon/methods/messages/check_chat_invite.html
-
 
 Retrieving all chat members (channels too)
 ******************************************
 
+.. note::
+
+    Use the `telethon.telegram_client.TelegramClient.iter_participants`
+    friendly method instead unless you have a better reason not to!
+
+    This method will handle different chat types for you automatically.
+
+
 In order to get all the members from a mega-group or channel, you need
-to use `GetParticipantsRequest`__. As we can see it needs an
-`InputChannel`__, (passing the mega-group or channel you're going to
-use will work), and a mandatory `ChannelParticipantsFilter`__. The
+to use :tl:`GetParticipantsRequest`. As we can see it needs an
+:tl:`InputChannel`, (passing the mega-group or channel you're going to
+use will work), and a mandatory :tl:`ChannelParticipantsFilter`. The
 closest thing to "no filter" is to simply use
-`ChannelParticipantsSearch`__ with an empty ``'q'`` string.
+:tl:`ChannelParticipantsSearch` with an empty ``'q'`` string.
 
 If we want to get *all* the members, we need to use a moving offset and
 a fixed limit:
@@ -134,20 +135,16 @@ a fixed limit:
 
 .. note::
 
-    It is **not** possible to get more than 10,000 members from a
-    group. It's a hard limit impossed by Telegram and there is
-    nothing you can do about it. Refer to `issue 573`__ for more.
+    If you need more than 10,000 members from a group you should use the
+    mentioned ``client.get_participants(..., aggressive=True)``. It will
+    do some tricks behind the scenes to get as many entities as possible.
+    Refer to `issue 573`__ for more on this.
 
 
-Note that ``GetParticipantsRequest`` returns `ChannelParticipants`__,
+Note that :tl:`GetParticipantsRequest` returns :tl:`ChannelParticipants`,
 which may have more information you need (like the role of the
 participants, total count of members, etc.)
 
-__ https://lonamiwebs.github.io/Telethon/methods/channels/get_participants.html
-__ https://lonamiwebs.github.io/Telethon/methods/channels/get_participants.html
-__ https://lonamiwebs.github.io/Telethon/types/channel_participants_filter.html
-__ https://lonamiwebs.github.io/Telethon/constructors/channel_participants_search.html
-__ https://lonamiwebs.github.io/Telethon/constructors/channels/channel_participants.html
 __ https://github.com/LonamiWebs/Telethon/issues/573
 
 
@@ -155,18 +152,17 @@ Recent Actions
 **************
 
 "Recent actions" is simply the name official applications have given to
-the "admin log". Simply use `GetAdminLogRequest`__ for that, and
+the "admin log". Simply use :tl:`GetAdminLogRequest` for that, and
 you'll get AdminLogResults.events in return which in turn has the final
 `.action`__.
 
-__ https://lonamiwebs.github.io/Telethon/methods/channels/get_admin_log.html
 __ https://lonamiwebs.github.io/Telethon/types/channel_admin_log_event_action.html
 
 
 Admin Permissions
 *****************
 
-Giving or revoking admin permissions can be done with the `EditAdminRequest`__:
+Giving or revoking admin permissions can be done with the :tl:`EditAdminRequest`:
 
     .. code-block:: python
 
@@ -201,17 +197,82 @@ Giving or revoking admin permissions can be done with the `EditAdminRequest`__:
 
         # User will now be able to change group info, delete other people's
         # messages and pin messages.
-        
-|  Thanks to `@Kyle2142`__ for `pointing out`__ that you **cannot** set all
-|  parameters to ``True`` to give a user full permissions, as not all
-|  permissions are related to both broadcast channels/megagroups.
-|
-|  E.g. trying to set ``post_messages=True`` in a megagroup will raise an
-|  error. It is recommended to always use keyword arguments, and to set only
-|  the permissions the user needs. If you don't need to change a permission,
-|  it can be omitted (full list `here`__).
 
-__ https://lonamiwebs.github.io/Telethon/methods/channels/edit_admin.html
+
+.. note::
+
+    Thanks to `@Kyle2142`__ for `pointing out`__ that you **cannot** set all
+    parameters to ``True`` to give a user full permissions, as not all
+    permissions are related to both broadcast channels/megagroups.
+
+    E.g. trying to set ``post_messages=True`` in a megagroup will raise an
+    error. It is recommended to always use keyword arguments, and to set only
+    the permissions the user needs. If you don't need to change a permission,
+    it can be omitted (full list `here`__).
+
+
+Restricting Users
+*****************
+
+Similar to how you give or revoke admin permissions, you can edit the
+banned rights of an user through :tl:`EditAdminRequest` and its parameter
+:tl:`ChannelBannedRights`:
+
+    .. code-block:: python
+
+        from telethon.tl.functions.channels import EditBannedRequest
+        from telethon.tl.types import ChannelBannedRights
+
+        from datetime import datetime, timedelta
+
+        # Restricting an user for 7 days, only allowing view/send messages.
+        #
+        # Note that it's "reversed". You must set to ``True`` the permissions
+        # you want to REMOVE, and leave as ``None`` those you want to KEEP.
+        rights = ChannelBannedRights(
+            until_date=datetime.now() + timedelta(days=7),
+            view_messages=None,
+            send_messages=None,
+            send_media=True,
+            send_stickers=True,
+            send_gifs=True,
+            send_games=True,
+            send_inline=True,
+            embed_links=True
+        )
+
+        # The above is equivalent to
+        rights = ChannelBannedRights(
+            until_date=datetime.now() + timedelta(days=7),
+            send_media=True,
+            send_stickers=True,
+            send_gifs=True,
+            send_games=True,
+            send_inline=True,
+            embed_links=True
+        )
+
+        client(EditBannedRequest(channel, user, rights))
+
+
+Kicking a member
+****************
+
+Telegram doesn't actually have a request to kick an user from a group.
+Instead, you need to restrict them so they can't see messages. Any date
+is enough:
+
+    .. code-block:: python
+
+        from telethon.tl.functions.channels import EditBannedRequest
+        from telethon.tl.types import ChannelBannedRights
+
+        client(EditBannedRequest(channel, user, ChannelBannedRights(
+            until_date=None,
+            view_messages=True
+        )))
+
+
 __ https://github.com/Kyle2142
 __ https://github.com/LonamiWebs/Telethon/issues/490
 __ https://lonamiwebs.github.io/Telethon/constructors/channel_admin_rights.html
@@ -222,13 +283,13 @@ Increasing View Count in a Channel
 
 It has been asked `quite`__ `a few`__ `times`__ (really, `many`__), and
 while I don't understand why so many people ask this, the solution is to
-use `GetMessagesViewsRequest`__, setting ``increment=True``:
+use :tl:`GetMessagesViewsRequest`, setting ``increment=True``:
 
     .. code-block:: python
 
 
         # Obtain `channel' through dialogs or through client.get_entity() or anyhow.
-        # Obtain `msg_ids' through `.get_message_history()` or anyhow. Must be a list.
+        # Obtain `msg_ids' through `.get_messages()` or anyhow. Must be a list.
 
         client(GetMessagesViewsRequest(
             peer=channel,
@@ -246,4 +307,3 @@ __ https://github.com/LonamiWebs/Telethon/issues/233
 __ https://github.com/LonamiWebs/Telethon/issues/305
 __ https://github.com/LonamiWebs/Telethon/issues/409
 __ https://github.com/LonamiWebs/Telethon/issues/447
-__ https://lonamiwebs.github.io/Telethon/methods/messages/get_messages_views.html

readthedocs/extra/examples/projects-using-telethon.rst

@@ -0,0 +1,44 @@
+=======================
+Projects using Telethon
+=======================
+
+This page lists some real world examples showcasing what can be built with
+the library.
+
+.. note::
+
+    Do you have a project that uses the library or know of any that's not
+    listed here? Feel free to leave a comment at
+    `issue 744 <https://github.com/LonamiWebs/Telethon/issues/744>`_
+    so it can be included in the next revision of the documentation!
+
+.. _projects-telegram-export:
+
+telegram-export
+***************
+
+`Link <https://github.com/expectocode/telegram-export>`_ /
+`Author's website <https://github.com/expectocode>`_
+
+A tool to download Telegram data (users, chats, messages, and media)
+into a database (and display the saved data).
+
+.. _projects-mautrix-telegram:
+
+mautrix-telegram
+****************
+
+`Link <https://github.com/tulir/mautrix-telegram>`_ /
+`Author's website <https://maunium.net/>`_
+
+A Matrix-Telegram hybrid puppeting/relaybot bridge.
+
+.. _projects-telegramtui:
+
+TelegramTUI
+***********
+
+`Link <https://github.com/bad-day/TelegramTUI>`_ /
+`Author's website <https://github.com/bad-day>`_
+
+A Telegram client on your terminal.

readthedocs/extra/examples/users.rst

@@ -0,0 +1,70 @@
+=====
+Users
+=====
+
+
+.. note::
+
+    These examples assume you have read :ref:`accessing-the-full-api`.
+
+
+Retrieving full information
+***************************
+
+If you need to retrieve the bio, biography or about information for an user
+you should use :tl:`GetFullUser`:
+
+
+    .. code-block:: python
+
+        from telethon.tl.functions.users import GetFullUserRequest
+
+        full = client(GetFullUserRequest(user))
+        # or even
+        full = client(GetFullUserRequest('username'))
+
+        bio = full.about
+
+
+See :tl:`UserFull` to know what other fields you can access.
+
+
+Updating your name and/or bio
+*****************************
+
+The first name, last name and bio (about) can all be changed with the same
+request. Omitted fields won't change after invoking :tl:`UpdateProfile`:
+
+    .. code-block:: python
+
+        from telethon.tl.functions.account import UpdateProfileRequest
+
+        client(UpdateProfileRequest(about='This is a test from Telethon'))
+
+
+Updating your username
+**********************
+
+You need to use :tl:`account.UpdateUsername`:
+
+    .. code-block:: python
+
+        from telethon.tl.functions.account import UpdateUsernameRequest
+
+        client(UpdateUsernameRequest('new_username'))
+
+
+Updating your profile photo
+***************************
+
+The easiest way is to upload a new file and use that as the profile photo
+through :tl:`UploadProfilePhoto`:
+
+
+    .. code-block:: python
+
+        from telethon.tl.functions.photos import UploadProfilePhotoRequest
+
+        client(UploadProfilePhotoRequest(
+            client.upload_file('/path/to/some/file')
+        ))

readthedocs/extra/examples/working-with-messages.rst

@@ -11,18 +11,30 @@ Working with messages
 Forwarding messages
 *******************
 
-Note that ForwardMessageRequest_ (note it's Message, singular) will *not*
-work if channels are involved. This is because channel (and megagroups) IDs
-are not unique, so you also need to know who the sender is (a parameter this
-request doesn't have).
+.. note::
 
-Either way, you are encouraged to use ForwardMessagesRequest_ (note it's
-Message*s*, plural) *always*, since it is more powerful, as follows:
+    Use the `telethon.telegram_client.TelegramClient.forward_messages`
+    friendly method instead unless you have a better reason not to!
 
-    .. code-block:: python
+    This method automatically accepts either a single message or many of them.
 
+.. code-block:: python
+
+        # If you only have the message IDs
+        client.forward_messages(
+            entity,  # to which entity you are forwarding the messages
+            message_ids,  # the IDs of the messages (or message) to forward
+            from_entity  # who sent the messages?
+        )
+
+        # If you have ``Message`` objects
+        client.forward_messages(
+            entity,  # to which entity you are forwarding the messages
+            messages  # the messages (or message) to forward
+        )
+
+        # You can also do it manually if you prefer
         from telethon.tl.functions.messages import ForwardMessagesRequest
-        #                                             note the s ^
 
         messages = foo()  # retrieve a few messages (or even one, in a list)
         from_entity = bar()
@@ -42,7 +54,15 @@ too, if that's all you have.
 Searching Messages
 *******************
 
-Messages are searched through the obvious SearchRequest_, but you may run
+.. note::
+
+    Use the `telethon.telegram_client.TelegramClient.iter_messages`
+    friendly method instead unless you have a better reason not to!
+
+    This method has ``search`` and ``filter`` parameters that will
+    suit your needs.
+
+Messages are searched through the obvious :tl:`SearchRequest`, but you may run
 into issues_. A valid example would be:
 
     .. code-block:: python
@@ -62,11 +82,12 @@ into issues_. A valid example would be:
             limit=10,       # How many results
             max_id=0,       # Maximum message ID
             min_id=0,       # Minimum message ID
-            from_id=None    # Who must have sent the message (peer)
+            from_id=None,   # Who must have sent the message (peer)
+            hash=0          # Special number to return nothing on no-change
         ))
 
 It's important to note that the optional parameter ``from_id`` could have
-been omitted (defaulting to ``None``). Changing it to InputUserEmpty_, as one
+been omitted (defaulting to ``None``). Changing it to :tl:`InputUserEmpty`, as one
 could think to specify "no user", won't work because this parameter is a flag,
 and it being unspecified has a different meaning.
 
@@ -119,8 +140,4 @@ send yourself the very first sticker you have:
         ))
 
 
-.. _ForwardMessageRequest: https://lonamiwebs.github.io/Telethon/methods/messages/forward_message.html
-.. _ForwardMessagesRequest: https://lonamiwebs.github.io/Telethon/methods/messages/forward_messages.html
-.. _SearchRequest: https://lonamiwebs.github.io/Telethon/methods/messages/search.html
 .. _issues: https://github.com/LonamiWebs/Telethon/issues/215
-.. _InputUserEmpty: https://lonamiwebs.github.io/Telethon/constructors/input_user_empty.html

readthedocs/extra/troubleshooting/deleted-limited-or-deactivated-accounts.rst

@@ -2,14 +2,15 @@
 Deleted, Limited or Deactivated Accounts
 ========================================
 
-If you're from Iran or Russian, we have bad news for you.
-Telegram is much more likely to ban these numbers,
-as they are often used to spam other accounts,
-likely through the use of libraries like this one.
-The best advice we can give you is to not abuse the API,
-like calling many requests really quickly,
+If you're from Iran or Russia, we have bad news for you. Telegram is much more
+likely to ban these numbers, as they are often used to spam other accounts,
+likely through the use of libraries like this one. The best advice we can
+give you is to not abuse the API, like calling many requests really quickly,
 and to sign up with these phones through an official application.
 
+We have also had reports from Kazakhstan and China, where connecting
+would fail. To solve these connection problems, you should use a proxy.
+
 Telegram may also ban virtual (VoIP) phone numbers,
 as again, they're likely to be used for spam.
 

readthedocs/index.rst

@@ -18,6 +18,9 @@ when you upgrade!
    If you're new here, you want to read :ref:`getting-started`. If you're
    looking for the method reference, you should check :ref:`telethon-package`.
 
+   The mentioned :ref:`telethon-package` is an important section and it
+   contains the friendly methods that **you should use** most of the time.
+
 
 What is this?
 *************
@@ -61,7 +64,9 @@ heavy job for you, so you can focus on developing an application.
 
    extra/examples/working-with-messages
    extra/examples/chats-and-channels
+   extra/examples/users
    extra/examples/bots
+   extra/examples/projects-using-telethon
 
 
 .. _Troubleshooting:

readthedocs/telethon.events.rst

@@ -3,7 +3,59 @@
 telethon\.events package
 ========================
 
+Every event (builder) subclasses `telethon.events.common.EventBuilder`,
+so all the methods in it can be used from any event builder/event instance.
+
+.. automodule:: telethon.events.common
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+.. automodule:: telethon.events.newmessage
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+.. automodule:: telethon.events.chataction
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+.. automodule:: telethon.events.userupdate
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+.. automodule:: telethon.events.messageedited
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+.. automodule:: telethon.events.messagedeleted
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+.. automodule:: telethon.events.messageread
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+.. automodule:: telethon.events.raw
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
 .. automodule:: telethon.events
     :members:
     :undoc-members:
     :show-inheritance:
+

readthedocs/telethon.tl.custom.rst

@@ -19,3 +19,21 @@ telethon\.tl\.custom\.dialog module
     :members:
     :undoc-members:
     :show-inheritance:
+
+
+telethon\.tl\.custom\.message module
+------------------------------------
+
+.. automodule:: telethon.tl.custom.message
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+telethon\.tl\.custom\.messagebutton module
+------------------------------------------
+
+.. automodule:: telethon.tl.custom.messagebutton
+    :members:
+    :undoc-members:
+    :show-inheritance:

setup.py

@@ -6,18 +6,18 @@ https://packaging.python.org/en/latest/distributing.html
 https://github.com/pypa/sampleproject
 
 Extra supported commands are:
-* gen_tl, to generate the classes required for Telethon to run
-* clean_tl, to clean these generated classes
+* gen, to generate the classes required for Telethon to run or docs
 * pypi, to generate sdist, bdist_wheel, and push to PyPi
 """
 
-# To use a consistent encoding
-from codecs import open
-from sys import argv, version_info
+import itertools
+import json
 import os
 import re
+import shutil
+from codecs import open
+from sys import argv, version_info
 
-# Always prefer setuptools over distutils
 from setuptools import find_packages, setup
 
 
@@ -37,44 +37,93 @@ class TempWorkDir:
         os.chdir(self.original)
 
 
-ERROR_LIST = 'telethon/errors/rpc_error_list.py'
-ERRORS_JSON = 'telethon_generator/errors.json'
-ERRORS_DESC = 'telethon_generator/error_descriptions'
-SCHEME_TL = 'telethon_generator/scheme.tl'
-GENERATOR_DIR = 'telethon/tl'
+GENERATOR_DIR = 'telethon_generator'
+LIBRARY_DIR = 'telethon'
+
+ERRORS_IN_JSON = os.path.join(GENERATOR_DIR, 'data', 'errors.json')
+ERRORS_IN_DESC = os.path.join(GENERATOR_DIR, 'data', 'error_descriptions')
+ERRORS_OUT = os.path.join(LIBRARY_DIR, 'errors', 'rpc_error_list.py')
+
+INVALID_BM_IN = os.path.join(GENERATOR_DIR, 'data', 'invalid_bot_methods.json')
+
+TLOBJECT_IN_CORE_TL = os.path.join(GENERATOR_DIR, 'data', 'mtproto_api.tl')
+TLOBJECT_IN_TL = os.path.join(GENERATOR_DIR, 'data', 'telegram_api.tl')
+TLOBJECT_OUT = os.path.join(LIBRARY_DIR, 'tl')
 IMPORT_DEPTH = 2
 
+DOCS_IN_RES = os.path.join(GENERATOR_DIR, 'data', 'html')
+DOCS_OUT = 'docs'
 
-def gen_tl(force=True):
-    from telethon_generator.tl_generator import TLGenerator
-    from telethon_generator.error_generator import generate_code
-    generator = TLGenerator(GENERATOR_DIR)
-    if generator.tlobjects_exist():
-        if not force:
-            return
-        print('Detected previous TLObjects. Cleaning...')
-        generator.clean_tlobjects()
 
-    print('Generating TLObjects...')
-    generator.generate_tlobjects(SCHEME_TL, import_depth=IMPORT_DEPTH)
-    print('Generating errors...')
-    generate_code(ERROR_LIST, json_file=ERRORS_JSON, errors_desc=ERRORS_DESC)
-    print('Done.')
+def generate(which):
+    from telethon_generator.parsers import parse_errors, parse_tl, find_layer
+    from telethon_generator.generators import\
+        generate_errors, generate_tlobjects, generate_docs, clean_tlobjects
+
+    with open(INVALID_BM_IN) as f:
+        ib = set(json.load(f))
+
+    layer = find_layer(TLOBJECT_IN_TL)
+    errors = list(parse_errors(ERRORS_IN_JSON, ERRORS_IN_DESC))
+    tlobjects = list(itertools.chain(
+        parse_tl(TLOBJECT_IN_CORE_TL, layer=layer, invalid_bot_methods=ib),
+        parse_tl(TLOBJECT_IN_TL, layer=layer, invalid_bot_methods=ib)))
+
+    if not which:
+        which.extend(('tl', 'errors'))
+
+    clean = 'clean' in which
+    action = 'Cleaning' if clean else 'Generating'
+    if clean:
+        which.remove('clean')
+
+    if 'all' in which:
+        which.remove('all')
+        for x in ('tl', 'errors', 'docs'):
+            if x not in which:
+                which.append(x)
+
+    if 'tl' in which:
+        which.remove('tl')
+        print(action, 'TLObjects...')
+        if clean:
+            clean_tlobjects(TLOBJECT_OUT)
+        else:
+            generate_tlobjects(tlobjects, layer, IMPORT_DEPTH, TLOBJECT_OUT)
+
+    if 'errors' in which:
+        which.remove('errors')
+        print(action, 'RPCErrors...')
+        if clean:
+            if os.path.isfile(ERRORS_OUT):
+                os.remove(ERRORS_OUT)
+        else:
+            with open(ERRORS_OUT, 'w', encoding='utf-8') as file:
+                generate_errors(errors, file)
+
+    if 'docs' in which:
+        which.remove('docs')
+        print(action, 'documentation...')
+        if clean:
+            if os.path.isdir(DOCS_OUT):
+                shutil.rmtree(DOCS_OUT)
+        else:
+            generate_docs(tlobjects, errors, layer, DOCS_IN_RES, DOCS_OUT)
+
+    if which:
+        print('The following items were not understood:', which)
+        print('  Consider using only "tl", "errors" and/or "docs".')
+        print('  Using only "clean" will clean them. "all" to act on all.')
+        print('  For instance "gen tl errors".')
 
 
 def main():
-    if len(argv) >= 2 and argv[1] == 'gen_tl':
-        gen_tl()
-
-    elif len(argv) >= 2 and argv[1] == 'clean_tl':
-        from telethon_generator.tl_generator import TLGenerator
-        print('Cleaning...')
-        TLGenerator(GENERATOR_DIR).clean_tlobjects()
-        print('Done.')
+    if len(argv) >= 2 and argv[1] == 'gen':
+        generate(argv[2:])
 
     elif len(argv) >= 2 and argv[1] == 'pypi':
         # (Re)generate the code to make sure we don't push without it
-        gen_tl()
+        generate(['tl', 'errors'])
 
         # Try importing the telethon module to assert it has no errors
         try:
@@ -96,14 +145,10 @@ def main():
         for x in ('build', 'dist', 'Telethon.egg-info'):
             rmtree(x, ignore_errors=True)
 
-    elif len(argv) >= 2 and argv[1] == 'fetch_errors':
-        from telethon_generator.error_generator import fetch_errors
-        fetch_errors(ERRORS_JSON)
-
     else:
-        # Call gen_tl() if the scheme.tl file exists, e.g. install from GitHub
-        if os.path.isfile(SCHEME_TL):
-            gen_tl(force=False)
+        # e.g. install from GitHub
+        if os.path.isdir(GENERATOR_DIR):
+            generate(['tl', 'errors'])
 
         # Get the long description from the README file
         with open('README.rst', encoding='utf-8') as f:
@@ -126,6 +171,11 @@ def main():
 
             license='MIT',
 
+            # See https://stackoverflow.com/a/40300957/4759433
+            # -> https://www.python.org/dev/peps/pep-0345/#requires-python
+            # -> http://setuptools.readthedocs.io/en/latest/setuptools.html
+            python_requires='>=3.4',
+
             # See https://pypi.python.org/pypi?%3Aaction=list_classifiers
             classifiers=[
                 #   3 - Alpha
@@ -139,7 +189,6 @@ def main():
                 'License :: OSI Approved :: MIT License',
 
                 'Programming Language :: Python :: 3',
-                'Programming Language :: Python :: 3.3',
                 'Programming Language :: Python :: 3.4',
                 'Programming Language :: Python :: 3.5',
                 'Programming Language :: Python :: 3.6'

telethon/__init__.py

@@ -1,7 +1,7 @@
 import logging
 from .telegram_bare_client import TelegramBareClient
 from .telegram_client import TelegramClient
-from .network import ConnectionMode
+from .network import connection
 from . import tl, version
 
 

telethon/crypto/rsa.py

@@ -28,7 +28,7 @@ def get_byte_array(integer):
     # Reference: https://core.telegram.org/mtproto/auth_key
     return int.to_bytes(
         integer,
-        length=(integer.bit_length() + 8 - 1) // 8,  # 8 bits per byte,
+        (integer.bit_length() + 8 - 1) // 8,  # 8 bits per byte,
         byteorder='big',
         signed=False
     )

telethon/errors/__init__.py

@@ -12,6 +12,7 @@ from .common import (
 )
 
 # This imports the base errors too, as they're imported there
+from .rpc_base_errors import *
 from .rpc_error_list import *
 
 
@@ -78,6 +79,9 @@ def rpc_message_to_error(code, message, report_method=None):
     if code == 404:
         return NotFoundError(message)
 
+    if code == 406:
+        return AuthKeyError(message)
+
     if code == 500:
         return ServerError(message)
 

telethon/errors/rpc_base_errors.py

@@ -3,6 +3,9 @@ class RPCError(Exception):
     code = None
     message = None
 
+    def __reduce__(self):
+        return type(self), ()
+
 
 class InvalidDCError(RPCError):
     """
@@ -56,6 +59,19 @@ class NotFoundError(RPCError):
         self.message = message
 
 
+class AuthKeyError(RPCError):
+    """
+    Errors related to invalid authorization key, like
+    AUTH_KEY_DUPLICATED which can cause the connection to fail.
+    """
+    code = 406
+    message = 'AUTH_KEY'
+
+    def __init__(self, message):
+        super().__init__(message)
+        self.message = message
+
+
 class FloodError(RPCError):
     """
     The maximum allowed number of attempts to invoke the given method

telethon/events/chataction.py

@@ -0,0 +1,335 @@
+from .common import EventBuilder, EventCommon, name_inner_event
+from .. import utils
+from ..tl import types, functions, custom
+
+
+@name_inner_event
+class ChatAction(EventBuilder):
+    """
+    Represents an action in a chat (such as user joined, left, or new pin).
+    """
+    def build(self, update):
+        if isinstance(update, types.UpdateChannelPinnedMessage) and update.id == 0:
+            # Telegram does not always send
+            # UpdateChannelPinnedMessage for new pins
+            # but always for unpin, with update.id = 0
+            event = ChatAction.Event(types.PeerChannel(update.channel_id),
+                                     unpin=True)
+
+        elif isinstance(update, types.UpdateChatParticipantAdd):
+            event = ChatAction.Event(types.PeerChat(update.chat_id),
+                                     added_by=update.inviter_id or True,
+                                     users=update.user_id)
+
+        elif isinstance(update, types.UpdateChatParticipantDelete):
+            event = ChatAction.Event(types.PeerChat(update.chat_id),
+                                     kicked_by=True,
+                                     users=update.user_id)
+
+        elif (isinstance(update, (
+                types.UpdateNewMessage, types.UpdateNewChannelMessage))
+              and isinstance(update.message, types.MessageService)):
+            msg = update.message
+            action = update.message.action
+            if isinstance(action, types.MessageActionChatJoinedByLink):
+                event = ChatAction.Event(msg,
+                                         added_by=True,
+                                         users=msg.from_id)
+            elif isinstance(action, types.MessageActionChatAddUser):
+                event = ChatAction.Event(msg,
+                                         added_by=msg.from_id or True,
+                                         users=action.users)
+            elif isinstance(action, types.MessageActionChatDeleteUser):
+                event = ChatAction.Event(msg,
+                                         kicked_by=msg.from_id or True,
+                                         users=action.user_id)
+            elif isinstance(action, types.MessageActionChatCreate):
+                event = ChatAction.Event(msg,
+                                         users=action.users,
+                                         created=True,
+                                         new_title=action.title)
+            elif isinstance(action, types.MessageActionChannelCreate):
+                event = ChatAction.Event(msg,
+                                         created=True,
+                                         users=msg.from_id,
+                                         new_title=action.title)
+            elif isinstance(action, types.MessageActionChatEditTitle):
+                event = ChatAction.Event(msg,
+                                         users=msg.from_id,
+                                         new_title=action.title)
+            elif isinstance(action, types.MessageActionChatEditPhoto):
+                event = ChatAction.Event(msg,
+                                         users=msg.from_id,
+                                         new_photo=action.photo)
+            elif isinstance(action, types.MessageActionChatDeletePhoto):
+                event = ChatAction.Event(msg,
+                                         users=msg.from_id,
+                                         new_photo=True)
+            elif isinstance(action, types.MessageActionPinMessage):
+                # Telegram always sends this service message for new pins
+                event = ChatAction.Event(msg,
+                                         users=msg.from_id,
+                                         new_pin=msg.reply_to_msg_id)
+            else:
+                return
+        else:
+            return
+
+        event._entities = update._entities
+        return self._filter_event(event)
+
+    class Event(EventCommon):
+        """
+        Represents the event of a new chat action.
+
+        Members:
+            action_message  (`MessageAction <https://lonamiwebs.github.io/Telethon/types/message_action.html>`_):
+                The message invoked by this Chat Action.
+
+            new_pin (`bool`):
+                ``True`` if there is a new pin.
+
+            new_photo (`bool`):
+                ``True`` if there's a new chat photo (or it was removed).
+
+            photo (:tl:`Photo`, optional):
+                The new photo (or ``None`` if it was removed).
+
+            user_added (`bool`):
+                ``True`` if the user was added by some other.
+
+            user_joined (`bool`):
+                ``True`` if the user joined on their own.
+
+            user_left (`bool`):
+                ``True`` if the user left on their own.
+
+            user_kicked (`bool`):
+                ``True`` if the user was kicked by some other.
+
+            created (`bool`, optional):
+                ``True`` if this chat was just created.
+
+            new_title (`str`, optional):
+                The new title string for the chat, if applicable.
+
+            unpin (`bool`):
+                ``True`` if the existing pin gets unpinned.
+        """
+        def __init__(self, where, new_pin=None, new_photo=None,
+                     added_by=None, kicked_by=None, created=None,
+                     users=None, new_title=None, unpin=None):
+            if isinstance(where, types.MessageService):
+                self.action_message = where
+                where = where.to_id
+            else:
+                self.action_message = None
+
+            super().__init__(chat_peer=where, msg_id=new_pin)
+
+            self.new_pin = isinstance(new_pin, int)
+            self._pinned_message = new_pin
+
+            self.new_photo = new_photo is not None
+            self.photo = \
+                new_photo if isinstance(new_photo, types.Photo) else None
+
+            self._added_by = None
+            self._kicked_by = None
+            self.user_added, self.user_joined, self.user_left,\
+                self.user_kicked, self.unpin = (False, False, False, False, False)
+
+            if added_by is True:
+                self.user_joined = True
+            elif added_by:
+                self.user_added = True
+                self._added_by = added_by
+
+            if kicked_by is True:
+                self.user_left = True
+            elif kicked_by:
+                self.user_kicked = True
+                self._kicked_by = kicked_by
+
+            self.created = bool(created)
+            self._user_peers = users if isinstance(users, list) else [users]
+            self._users = None
+            self._input_users = None
+            self.new_title = new_title
+            self.unpin = unpin
+
+        def _set_client(self, client):
+            super()._set_client(client)
+            if self.action_message:
+                self.action_message = custom.Message(
+                    client, self.action_message, self._entities, None)
+
+        def respond(self, *args, **kwargs):
+            """
+            Responds to the chat action message (not as a reply). Shorthand for
+            `telethon.telegram_client.TelegramClient.send_message` with
+            ``entity`` already set.
+            """
+            return self._client.send_message(self.input_chat, *args, **kwargs)
+
+        def reply(self, *args, **kwargs):
+            """
+            Replies to the chat action message (as a reply). Shorthand for
+            `telethon.telegram_client.TelegramClient.send_message` with
+            both ``entity`` and ``reply_to`` already set.
+
+            Has the same effect as `respond` if there is no message.
+            """
+            if not self.action_message:
+                return self.respond(*args, **kwargs)
+
+            kwargs['reply_to'] = self.action_message.id
+            return self._client.send_message(self.input_chat, *args, **kwargs)
+
+        def delete(self, *args, **kwargs):
+            """
+            Deletes the chat action message. You're responsible for checking
+            whether you have the permission to do so, or to except the error
+            otherwise. Shorthand for
+            `telethon.telegram_client.TelegramClient.delete_messages` with
+            ``entity`` and ``message_ids`` already set.
+
+            Does nothing if no message action triggered this event.
+            """
+            if self.action_message:
+                return self._client.delete_messages(self.input_chat,
+                                                    [self.action_message],
+                                                    *args, **kwargs)
+
+        @property
+        def pinned_message(self):
+            """
+            If ``new_pin`` is ``True``, this returns the
+            `telethon.tl.custom.message.Message` object that was pinned.
+            """
+            if self._pinned_message == 0:
+                return None
+
+            if isinstance(self._pinned_message, int) and self.input_chat:
+                r = self._client(functions.channels.GetMessagesRequest(
+                    self._input_chat, [self._pinned_message]
+                ))
+                try:
+                    self._pinned_message = next(
+                        x for x in r.messages
+                        if isinstance(x, types.Message)
+                        and x.id == self._pinned_message
+                    )
+                except StopIteration:
+                    pass
+
+            if isinstance(self._pinned_message, types.Message):
+                return self._pinned_message
+
+        @property
+        def added_by(self):
+            """
+            The user who added ``users``, if applicable (``None`` otherwise).
+            """
+            if self._added_by and not isinstance(self._added_by, types.User):
+                self._added_by =\
+                    self._entities.get(utils.get_peer_id(self._added_by))
+
+                if not self._added_by:
+                    self._added_by = self._client.get_entity(self._added_by)
+
+            return self._added_by
+
+        @property
+        def kicked_by(self):
+            """
+            The user who kicked ``users``, if applicable (``None`` otherwise).
+            """
+            if self._kicked_by and not isinstance(self._kicked_by, types.User):
+                self._kicked_by =\
+                    self._entities.get(utils.get_peer_id(self._kicked_by))
+
+                if not self._kicked_by:
+                    self._kicked_by = self._client.get_entity(self._kicked_by)
+
+            return self._kicked_by
+
+        @property
+        def user(self):
+            """
+            The first user that takes part in this action (e.g. joined).
+
+            Might be ``None`` if the information can't be retrieved or
+            there is no user taking part.
+            """
+            if self.users:
+                return self._users[0]
+
+        @property
+        def input_user(self):
+            """
+            Input version of the ``self.user`` property.
+            """
+            if self.input_users:
+                return self._input_users[0]
+
+        @property
+        def user_id(self):
+            """
+            Returns the marked signed ID of the first user, if any.
+            """
+            if self._user_peers:
+                return utils.get_peer_id(self._user_peers[0])
+
+        @property
+        def users(self):
+            """
+            A list of users that take part in this action (e.g. joined).
+
+            Might be empty if the information can't be retrieved or there
+            are no users taking part.
+            """
+            if not self._user_peers:
+                return []
+
+            if self._users is None:
+                have, missing = [], []
+                for peer in self._user_peers:
+                    user = self._entities.get(utils.get_peer_id(peer))
+                    if user:
+                        have.append(user)
+                    else:
+                        missing.append(peer)
+
+                try:
+                    missing = self._client.get_entity(missing)
+                except (TypeError, ValueError):
+                    missing = []
+
+                self._users = have + missing
+
+            return self._users
+
+        @property
+        def input_users(self):
+            """
+            Input version of the ``self.users`` property.
+            """
+            if self._input_users is None and self._user_peers:
+                self._input_users = []
+                for peer in self._user_peers:
+                    try:
+                        self._input_users.append(self._client.get_input_entity(
+                            peer
+                        ))
+                    except (TypeError, ValueError):
+                        pass
+            return self._input_users
+
+        @property
+        def user_ids(self):
+            """
+            Returns the marked signed ID of the users, if any.
+            """
+            if self._user_peers:
+                return [utils.get_peer_id(u) for u in self._user_peers]

telethon/events/common.py

@@ -0,0 +1,220 @@
+import abc
+import itertools
+import warnings
+
+from .. import utils
+from ..errors import RPCError
+from ..tl import TLObject, types, functions
+
+
+def _into_id_set(client, chats):
+    """Helper util to turn the input chat or chats into a set of IDs."""
+    if chats is None:
+        return None
+
+    if not utils.is_list_like(chats):
+        chats = (chats,)
+
+    result = set()
+    for chat in chats:
+        if isinstance(chat, int):
+            if chat < 0:
+                result.add(chat)  # Explicitly marked IDs are negative
+            else:
+                result.update({  # Support all valid types of peers
+                    utils.get_peer_id(types.PeerUser(chat)),
+                    utils.get_peer_id(types.PeerChat(chat)),
+                    utils.get_peer_id(types.PeerChannel(chat)),
+                })
+        elif isinstance(chat, TLObject) and chat.SUBCLASS_OF_ID == 0x2d45687:
+            # 0x2d45687 == crc32(b'Peer')
+            result.add(utils.get_peer_id(chat))
+        else:
+            chat = client.get_input_entity(chat)
+            if isinstance(chat, types.InputPeerSelf):
+                chat = client.get_me(input_peer=True)
+            result.add(utils.get_peer_id(chat))
+
+    return result
+
+
+class EventBuilder(abc.ABC):
+    """
+    The common event builder, with builtin support to filter per chat.
+
+    Args:
+        chats (`entity`, optional):
+            May be one or more entities (username/peer/etc.). By default,
+            only matching chats will be handled.
+
+        blacklist_chats (`bool`, optional):
+            Whether to treat the chats as a blacklist instead of
+            as a whitelist (default). This means that every chat
+            will be handled *except* those specified in ``chats``
+            which will be ignored if ``blacklist_chats=True``.
+    """
+    def __init__(self, chats=None, blacklist_chats=False):
+        self.chats = chats
+        self.blacklist_chats = blacklist_chats
+        self._self_id = None
+
+    @abc.abstractmethod
+    def build(self, update):
+        """Builds an event for the given update if possible, or returns None"""
+
+    def resolve(self, client):
+        """Helper method to allow event builders to be resolved before usage"""
+        self.chats = _into_id_set(client, self.chats)
+        self._self_id = client.get_me(input_peer=True).user_id
+
+    def _filter_event(self, event):
+        """
+        If the ID of ``event._chat_peer`` isn't in the chats set (or it is
+        but the set is a blacklist) returns ``None``, otherwise the event.
+        """
+        if self.chats is not None:
+            inside = utils.get_peer_id(event._chat_peer) in self.chats
+            if inside == self.blacklist_chats:
+                # If this chat matches but it's a blacklist ignore.
+                # If it doesn't match but it's a whitelist ignore.
+                return None
+        return event
+
+
+class EventCommon(abc.ABC):
+    """Intermediate class with common things to all events"""
+    _event_name = 'Event'
+
+    def __init__(self, chat_peer=None, msg_id=None, broadcast=False):
+        self._entities = {}
+        self._client = None
+        self._chat_peer = chat_peer
+        self._message_id = msg_id
+        self._input_chat = None
+        self._chat = None
+
+        self.pattern_match = None
+        self.original_update = None
+
+        self.is_private = isinstance(chat_peer, types.PeerUser)
+        self.is_group = (
+            isinstance(chat_peer, (types.PeerChat, types.PeerChannel))
+            and not broadcast
+        )
+        self.is_channel = isinstance(chat_peer, types.PeerChannel)
+
+    def _set_client(self, client):
+        """
+        Setter so subclasses can act accordingly when the client is set.
+        """
+        self._client = client
+
+    def _get_entity(self, msg_id, entity_id, chat=None):
+        """
+        Helper function to call :tl:`GetMessages` on the give msg_id and
+        return the input entity whose ID is the given entity ID.
+
+        If ``chat`` is present it must be an :tl:`InputPeer`.
+
+        Returns a tuple of ``(entity, input_peer)`` if it was found, or
+        a tuple of ``(None, None)`` if it couldn't be.
+        """
+        try:
+            if isinstance(chat, types.InputPeerChannel):
+                result = self._client(
+                    functions.channels.GetMessagesRequest(chat, [msg_id])
+                )
+            else:
+                result = self._client(
+                    functions.messages.GetMessagesRequest([msg_id])
+                )
+        except RPCError:
+            return None, None
+
+        entity = {
+            utils.get_peer_id(x): x for x in itertools.chain(
+                getattr(result, 'chats', []),
+                getattr(result, 'users', []))
+        }.get(entity_id)
+        if entity:
+            return entity, utils.get_input_peer(entity)
+        else:
+            return None, None
+
+    @property
+    def input_chat(self):
+        """
+        The (:tl:`InputPeer`) (group, megagroup or channel) on which
+        the event occurred. This doesn't have the title or anything,
+        but is useful if you don't need those to avoid further
+        requests.
+
+        Note that this might be ``None`` if the library can't find it.
+        """
+        if self._input_chat is None and self._chat_peer is not None:
+            try:
+                self._input_chat = self._client.get_input_entity(
+                    self._chat_peer
+                )
+            except (ValueError, TypeError):
+                # The library hasn't seen this chat, get the message
+                if not isinstance(self._chat_peer, types.PeerChannel):
+                    # TODO For channels, getDifference? Maybe looking
+                    # in the dialogs (which is already done) is enough.
+                    if self._message_id is not None:
+                        self._chat, self._input_chat = self._get_entity(
+                            self._message_id,
+                            utils.get_peer_id(self._chat_peer)
+                        )
+        return self._input_chat
+
+    @property
+    def client(self):
+        return self._client
+
+    @property
+    def chat(self):
+        """
+        The (:tl:`User` | :tl:`Chat` | :tl:`Channel`, optional) on which
+        the event occurred. This property may make an API call the first time
+        to get the most up to date version of the chat (mostly when the event
+        doesn't belong to a channel), so keep that in mind.
+        """
+        if not self.input_chat:
+            return None
+
+        if self._chat is None:
+            self._chat = self._entities.get(utils.get_peer_id(self._input_chat))
+
+        if self._chat is None:
+            self._chat = self._client.get_entity(self._input_chat)
+
+        return self._chat
+
+    @property
+    def chat_id(self):
+        """
+        Returns the marked integer ID of the chat, if any.
+        """
+        if self._chat_peer:
+            return utils.get_peer_id(self._chat_peer)
+
+    def __str__(self):
+        return TLObject.pretty_format(self.to_dict())
+
+    def stringify(self):
+        return TLObject.pretty_format(self.to_dict(), indent=0)
+
+    def to_dict(self):
+        d = {k: v for k, v in self.__dict__.items() if k[0] != '_'}
+        d['_'] = self._event_name
+        return d
+
+
+def name_inner_event(cls):
+    """Decorator to rename cls.Event 'Event' as 'cls.Event'"""
+    if hasattr(cls, 'Event'):
+        cls.Event._event_name = '{}.Event'.format(cls.__name__)
+    else:
+        warnings.warn('Class {} does not have a inner Event'.format(cls))
+    return cls

telethon/events/messagedeleted.py

@@ -0,0 +1,40 @@
+from .common import EventBuilder, EventCommon, name_inner_event
+from ..tl import types
+
+
+@name_inner_event
+class MessageDeleted(EventBuilder):
+    """
+    Event fired when one or more messages are deleted.
+    """
+    def build(self, update):
+        if isinstance(update, types.UpdateDeleteMessages):
+            event = MessageDeleted.Event(
+                deleted_ids=update.messages,
+                peer=None
+            )
+        elif isinstance(update, types.UpdateDeleteChannelMessages):
+            event = MessageDeleted.Event(
+                deleted_ids=update.messages,
+                peer=types.PeerChannel(update.channel_id)
+            )
+        else:
+            return
+
+        event._entities = update._entities
+        return self._filter_event(event)
+
+    class Event(EventCommon):
+        def __init__(self, deleted_ids, peer):
+            super().__init__(
+                chat_peer=peer, msg_id=(deleted_ids or [0])[0]
+            )
+            if peer is None:
+                # If it's not a channel ID, then it was private/small group.
+                # We can't know which one was exactly unless we logged all
+                # messages, but we can indicate that it was maybe either of
+                # both by setting them both to True.
+                self.is_private = self.is_group = True
+
+            self.deleted_id = None if not deleted_ids else deleted_ids[0]
+            self.deleted_ids = deleted_ids

telethon/events/messageedited.py

@@ -0,0 +1,22 @@
+from .common import name_inner_event
+from .newmessage import NewMessage
+from ..tl import types
+
+
+@name_inner_event
+class MessageEdited(NewMessage):
+    """
+    Event fired when a message has been edited.
+    """
+    def build(self, update):
+        if isinstance(update, (types.UpdateEditMessage,
+                               types.UpdateEditChannelMessage)):
+            event = MessageEdited.Event(update.message)
+        else:
+            return
+
+        event._entities = update._entities
+        return self._message_filter_event(event)
+
+    class Event(NewMessage.Event):
+        pass  # Required if we want a different name for it

telethon/events/messageread.py

@@ -0,0 +1,129 @@
+from .common import EventBuilder, EventCommon, name_inner_event
+from .. import utils
+from ..tl import types, functions
+
+
+@name_inner_event
+class MessageRead(EventBuilder):
+    """
+    Event fired when one or more messages have been read.
+
+    Args:
+        inbox (`bool`, optional):
+            If this argument is ``True``, then when you read someone else's
+            messages the event will be fired. By default (``False``) only
+            when messages you sent are read by someone else will fire it.
+    """
+    def __init__(self, inbox=False, chats=None, blacklist_chats=None):
+        super().__init__(chats, blacklist_chats)
+        self.inbox = inbox
+
+    def build(self, update):
+        if isinstance(update, types.UpdateReadHistoryInbox):
+            event = MessageRead.Event(update.peer, update.max_id, False)
+        elif isinstance(update, types.UpdateReadHistoryOutbox):
+            event = MessageRead.Event(update.peer, update.max_id, True)
+        elif isinstance(update, types.UpdateReadChannelInbox):
+            event = MessageRead.Event(types.PeerChannel(update.channel_id),
+                                      update.max_id, False)
+        elif isinstance(update, types.UpdateReadChannelOutbox):
+            event = MessageRead.Event(types.PeerChannel(update.channel_id),
+                                      update.max_id, True)
+        elif isinstance(update, types.UpdateReadMessagesContents):
+            event = MessageRead.Event(message_ids=update.messages,
+                                      contents=True)
+        elif isinstance(update, types.UpdateChannelReadMessagesContents):
+            event = MessageRead.Event(types.PeerChannel(update.channel_id),
+                                      message_ids=update.messages,
+                                      contents=True)
+        else:
+            return
+
+        if self.inbox == event.outbox:
+            return
+
+        event._entities = update._entities
+        return self._filter_event(event)
+
+    class Event(EventCommon):
+        """
+        Represents the event of one or more messages being read.
+
+        Members:
+            max_id (`int`):
+                Up to which message ID has been read. Every message
+                with an ID equal or lower to it have been read.
+
+            outbox (`bool`):
+                ``True`` if someone else has read your messages.
+
+            contents (`bool`):
+                ``True`` if what was read were the contents of a message.
+                This will be the case when e.g. you play a voice note.
+                It may only be set on ``inbox`` events.
+        """
+        def __init__(self, peer=None, max_id=None, out=False, contents=False,
+                     message_ids=None):
+            self.outbox = out
+            self.contents = contents
+            self._message_ids = message_ids or []
+            self._messages = None
+            self.max_id = max_id or max(message_ids or [], default=None)
+            super().__init__(peer, self.max_id)
+
+        @property
+        def inbox(self):
+            """
+            ``True`` if you have read someone else's messages.
+            """
+            return not self.outbox
+
+        @property
+        def message_ids(self):
+            """
+            The IDs of the messages **which contents'** were read.
+
+            Use :meth:`is_read` if you need to check whether a message
+            was read instead checking if it's in here.
+            """
+            return self._message_ids
+
+        @property
+        def messages(self):
+            """
+            The list of `telethon.tl.custom.message.Message`
+            **which contents'** were read.
+
+            Use :meth:`is_read` if you need to check whether a message
+            was read instead checking if it's in here.
+            """
+            if self._messages is None:
+                chat = self.input_chat
+                if not chat:
+                    self._messages = []
+                else:
+                    self._messages = self._client.get_messages(
+                        chat, ids=self._message_ids)
+
+            return self._messages
+
+        def is_read(self, message):
+            """
+            Returns ``True`` if the given message (or its ID) has been read.
+
+            If a list-like argument is provided, this method will return a
+            list of booleans indicating which messages have been read.
+            """
+            if utils.is_list_like(message):
+                return [(m if isinstance(m, int) else m.id) <= self.max_id
+                        for m in message]
+            else:
+                return (message if isinstance(message, int)
+                        else message.id) <= self.max_id
+
+        def __contains__(self, message):
+            """``True`` if the message(s) are read message."""
+            if utils.is_list_like(message):
+                return all(self.is_read(message))
+            else:
+                return self.is_read(message)

telethon/events/newmessage.py

@@ -0,0 +1,161 @@
+import re
+
+from .common import EventBuilder, EventCommon, name_inner_event
+from ..tl import types, custom
+
+
+@name_inner_event
+class NewMessage(EventBuilder):
+    """
+    Represents a new message event builder.
+
+    Args:
+        incoming (`bool`, optional):
+            If set to ``True``, only **incoming** messages will be handled.
+            Mutually exclusive with ``outgoing`` (can only set one of either).
+
+        outgoing (`bool`, optional):
+            If set to ``True``, only **outgoing** messages will be handled.
+            Mutually exclusive with ``incoming`` (can only set one of either).
+
+        pattern (`str`, `callable`, `Pattern`, optional):
+            If set, only messages matching this pattern will be handled.
+            You can specify a regex-like string which will be matched
+            against the message, a callable function that returns ``True``
+            if a message is acceptable, or a compiled regex pattern.
+    """
+    def __init__(self, incoming=None, outgoing=None,
+                 chats=None, blacklist_chats=False, pattern=None):
+        if incoming is not None and outgoing is None:
+            outgoing = not incoming
+        elif outgoing is not None and incoming is None:
+            incoming = not outgoing
+
+        if incoming and outgoing:
+            self.incoming = self.outgoing = None  # Same as no filter
+        elif all(x is not None and not x for x in (incoming, outgoing)):
+            raise ValueError("Don't create an event handler if you "
+                             "don't want neither incoming or outgoing!")
+
+        super().__init__(chats=chats, blacklist_chats=blacklist_chats)
+        self.incoming = incoming
+        self.outgoing = outgoing
+        if isinstance(pattern, str):
+            self.pattern = re.compile(pattern).match
+        elif not pattern or callable(pattern):
+            self.pattern = pattern
+        elif hasattr(pattern, 'match') and callable(pattern.match):
+            self.pattern = pattern.match
+        else:
+            raise TypeError('Invalid pattern type given')
+
+    def build(self, update):
+        if isinstance(update,
+                      (types.UpdateNewMessage, types.UpdateNewChannelMessage)):
+            if not isinstance(update.message, types.Message):
+                return  # We don't care about MessageService's here
+            event = NewMessage.Event(update.message)
+        elif isinstance(update, types.UpdateShortMessage):
+            event = NewMessage.Event(types.Message(
+                out=update.out,
+                mentioned=update.mentioned,
+                media_unread=update.media_unread,
+                silent=update.silent,
+                id=update.id,
+                to_id=types.PeerUser(update.user_id),
+                from_id=self._self_id if update.out else update.user_id,
+                message=update.message,
+                date=update.date,
+                fwd_from=update.fwd_from,
+                via_bot_id=update.via_bot_id,
+                reply_to_msg_id=update.reply_to_msg_id,
+                entities=update.entities
+            ))
+        elif isinstance(update, types.UpdateShortChatMessage):
+            event = NewMessage.Event(types.Message(
+                out=update.out,
+                mentioned=update.mentioned,
+                media_unread=update.media_unread,
+                silent=update.silent,
+                id=update.id,
+                from_id=update.from_id,
+                to_id=types.PeerChat(update.chat_id),
+                message=update.message,
+                date=update.date,
+                fwd_from=update.fwd_from,
+                via_bot_id=update.via_bot_id,
+                reply_to_msg_id=update.reply_to_msg_id,
+                entities=update.entities
+            ))
+        else:
+            return
+
+        event._entities = update._entities
+        return self._message_filter_event(event)
+
+    def _message_filter_event(self, event):
+        # Short-circuit if we let pass all events
+        if all(x is None for x in (self.incoming, self.outgoing, self.chats,
+                                   self.pattern)):
+            return event
+
+        if self.incoming and event.message.out:
+            return
+        if self.outgoing and not event.message.out:
+            return
+
+        if self.pattern:
+            match = self.pattern(event.message.message or '')
+            if not match:
+                return
+            event.pattern_match = match
+
+        return self._filter_event(event)
+
+    class Event(EventCommon):
+        """
+        Represents the event of a new message. This event can be treated
+        to all effects as a `telethon.tl.custom.message.Message`, so please
+        **refer to its documentation** to know what you can do with this event.
+
+        Members:
+            message (:tl:`Message`):
+                This is the only difference with the received
+                `telethon.tl.custom.message.Message`, and will
+                return the `telethon.tl.custom.message.Message` itself,
+                not the text.
+
+                See `telethon.tl.custom.message.Message` for the rest of
+                available members and methods.
+        """
+        def __init__(self, message):
+            self.__dict__['_init'] = False
+            if not message.out and isinstance(message.to_id, types.PeerUser):
+                # Incoming message (e.g. from a bot) has to_id=us, and
+                # from_id=bot (the actual "chat" from an user's perspective).
+                chat_peer = types.PeerUser(message.from_id)
+            else:
+                chat_peer = message.to_id
+
+            super().__init__(chat_peer=chat_peer,
+                             msg_id=message.id, broadcast=bool(message.post))
+
+            self.message = message
+
+        def _set_client(self, client):
+            super()._set_client(client)
+            self.message = custom.Message(
+                client, self.message, self._entities, None)
+            self.__dict__['_init'] = True  # No new attributes can be set
+
+        def __getattr__(self, item):
+            if item in self.__dict__:
+                return self.__dict__[item]
+            else:
+                return getattr(self.message, item)
+
+        def __setattr__(self, name, value):
+            if not self.__dict__['_init'] or name in self.__dict__:
+                self.__dict__[name] = value
+            else:
+                setattr(self.message, name, value)

telethon/events/raw.py

@@ -0,0 +1,30 @@
+from .common import EventBuilder
+from .. import utils
+
+
+class Raw(EventBuilder):
+    """
+    Represents a raw event. The event is the update itself.
+
+    Args:
+        types (`list` | `tuple` | `type`, optional):
+            The type or types that the :tl:`Update` instance must be.
+            Equivalent to ``if not isinstance(update, types): return``.
+    """
+    def __init__(self, types=None):
+        super().__init__()
+        if not types:
+            self.types = None
+        elif not utils.is_list_like(types):
+            assert isinstance(types, type)
+            self.types = types
+        else:
+            assert all(isinstance(x, type) for x in types)
+            self.types = tuple(types)
+
+    def resolve(self, client):
+        pass
+
+    def build(self, update):
+        if not self.types or isinstance(update, self.types):
+            return update

telethon/events/userupdate.py

@@ -0,0 +1,163 @@
+import datetime
+
+from .common import EventBuilder, EventCommon, name_inner_event
+from ..tl import types
+
+
+@name_inner_event
+class UserUpdate(EventBuilder):
+    """
+    Represents an user update (gone online, offline, joined Telegram).
+    """
+    def build(self, update):
+        if isinstance(update, types.UpdateUserStatus):
+            event = UserUpdate.Event(update.user_id,
+                                     status=update.status)
+        else:
+            return
+
+        event._entities = update._entities
+        return self._filter_event(event)
+
+    class Event(EventCommon):
+        """
+        Represents the event of an user status update (last seen, joined).
+
+        Members:
+            online (`bool`, optional):
+                ``True`` if the user is currently online, ``False`` otherwise.
+                Might be ``None`` if this information is not present.
+
+            last_seen (`datetime`, optional):
+                Exact date when the user was last seen if known.
+
+            until (`datetime`, optional):
+                Until when will the user remain online.
+
+            within_months (`bool`):
+                ``True`` if the user was seen within 30 days.
+
+            within_weeks (`bool`):
+                ``True`` if the user was seen within 7 days.
+
+            recently (`bool`):
+                ``True`` if the user was seen within a day.
+
+            action (:tl:`SendMessageAction`, optional):
+                The "typing" action if any the user is performing if any.
+
+            cancel (`bool`):
+                ``True`` if the action was cancelling other actions.
+
+            typing (`bool`):
+                ``True`` if the action is typing a message.
+
+            recording (`bool`):
+                ``True`` if the action is recording something.
+
+            uploading (`bool`):
+                ``True`` if the action is uploading something.
+
+            playing (`bool`):
+                ``True`` if the action is playing a game.
+
+            audio (`bool`):
+                ``True`` if what's being recorded/uploaded is an audio.
+
+            round (`bool`):
+                ``True`` if what's being recorded/uploaded is a round video.
+
+            video (`bool`):
+                ``True`` if what's being recorded/uploaded is an video.
+
+            document (`bool`):
+                ``True`` if what's being uploaded is document.
+
+            geo (`bool`):
+                ``True`` if what's being uploaded is a geo.
+
+            photo (`bool`):
+                ``True`` if what's being uploaded is a photo.
+
+            contact (`bool`):
+                ``True`` if what's being uploaded (selected) is a contact.
+        """
+        def __init__(self, user_id, status=None, typing=None):
+            super().__init__(types.PeerUser(user_id))
+
+            self.online = None if status is None else \
+                isinstance(status, types.UserStatusOnline)
+
+            self.last_seen = status.was_online if \
+                isinstance(status, types.UserStatusOffline) else None
+
+            self.until = status.expires if \
+                isinstance(status, types.UserStatusOnline) else None
+
+            if self.last_seen:
+                diff = datetime.datetime.now() - self.last_seen
+                if diff < datetime.timedelta(days=30):
+                    self.within_months = True
+                    if diff < datetime.timedelta(days=7):
+                        self.within_weeks = True
+                        if diff < datetime.timedelta(days=1):
+                            self.recently = True
+            else:
+                self.within_months = self.within_weeks = self.recently = False
+                if isinstance(status, (types.UserStatusOnline,
+                                       types.UserStatusRecently)):
+                    self.within_months = self.within_weeks = True
+                    self.recently = True
+                elif isinstance(status, types.UserStatusLastWeek):
+                    self.within_months = self.within_weeks = True
+                elif isinstance(status, types.UserStatusLastMonth):
+                    self.within_months = True
+
+            self.action = typing
+            if typing:
+                self.cancel = self.typing = self.recording = self.uploading = \
+                    self.playing = False
+                self.audio = self.round = self.video = self.document = \
+                    self.geo = self.photo = self.contact = False
+
+                if isinstance(typing, types.SendMessageCancelAction):
+                    self.cancel = True
+                elif isinstance(typing, types.SendMessageTypingAction):
+                    self.typing = True
+                elif isinstance(typing, types.SendMessageGamePlayAction):
+                    self.playing = True
+                elif isinstance(typing, types.SendMessageGeoLocationAction):
+                    self.geo = True
+                elif isinstance(typing, types.SendMessageRecordAudioAction):
+                    self.recording = self.audio = True
+                elif isinstance(typing, types.SendMessageRecordRoundAction):
+                    self.recording = self.round = True
+                elif isinstance(typing, types.SendMessageRecordVideoAction):
+                    self.recording = self.video = True
+                elif isinstance(typing, types.SendMessageChooseContactAction):
+                    self.uploading = self.contact = True
+                elif isinstance(typing, types.SendMessageUploadAudioAction):
+                    self.uploading = self.audio = True
+                elif isinstance(typing, types.SendMessageUploadDocumentAction):
+                    self.uploading = self.document = True
+                elif isinstance(typing, types.SendMessageUploadPhotoAction):
+                    self.uploading = self.photo = True
+                elif isinstance(typing, types.SendMessageUploadRoundAction):
+                    self.uploading = self.round = True
+                elif isinstance(typing, types.SendMessageUploadVideoAction):
+                    self.uploading = self.video = True
+
+        @property
+        def user(self):
+            """Alias around the chat (conversation)."""
+            return self.chat
+
+        @property
+        def input_user(self):
+            """Alias around the input chat."""
+            return self.input_chat
+
+        @property
+        def user_id(self):
+            """Alias around `chat_id`."""
+            return self.chat_id

telethon/extensions/html.py

@@ -121,6 +121,9 @@ def parse(html):
     :param message: the message with HTML to be parsed.
     :return: a tuple consisting of (clean message, [message entities]).
     """
+    if not html:
+        return html, []
+
     parser = HTMLToTelegramParser()
     parser.feed(_add_surrogate(html))
     return _del_surrogate(parser.text), parser.entities
@@ -135,7 +138,7 @@ def unparse(text, entities):
     :param entities: the MessageEntity's applied to the text.
     :return: a HTML representation of the combination of both inputs.
     """
-    if not entities:
+    if not text or not entities:
         return text
 
     text = _add_surrogate(text)

telethon/extensions/markdown.py

@@ -4,15 +4,16 @@ for use within the library, which attempts to handle emojies correctly,
 since they seem to count as two characters and it's a bit strange.
 """
 import re
-import struct
 
 from ..tl import TLObject
-
 from ..tl.types import (
     MessageEntityBold, MessageEntityItalic, MessageEntityCode,
     MessageEntityPre, MessageEntityTextUrl
 )
-
+from ..utils import (
+    add_surrogate as _add_surrogate,
+    del_surrogate as _del_surrogate
+)
 
 DEFAULT_DELIMITERS = {
     '**': MessageEntityBold,
@@ -25,19 +26,6 @@ DEFAULT_URL_RE = re.compile(r'\[([\S\s]+?)\]\((.+?)\)')
 DEFAULT_URL_FORMAT = '[{0}]({1})'
 
 
-def _add_surrogate(text):
-    return ''.join(
-        # SMP -> Surrogate Pairs (Telegram offsets are calculated with these).
-        # See https://en.wikipedia.org/wiki/Plane_(Unicode)#Overview for more.
-        ''.join(chr(y) for y in struct.unpack('<HH', x.encode('utf-16le')))
-        if (0x10000 <= ord(x) <= 0x10FFFF) else x for x in text
-    )
-
-
-def _del_surrogate(text):
-    return text.encode('utf-16', 'surrogatepass').decode('utf-16')
-
-
 def parse(message, delimiters=None, url_re=None):
     """
     Parses the given markdown message and returns its stripped representation
@@ -48,6 +36,9 @@ def parse(message, delimiters=None, url_re=None):
     :param url_re: the URL bytes regex to be used. Must have two groups.
     :return: a tuple consisting of (clean message, [message entities]).
     """
+    if not message:
+        return message, []
+
     if url_re is None:
         url_re = DEFAULT_URL_RE
     elif isinstance(url_re, str):
@@ -81,7 +72,7 @@ def parse(message, delimiters=None, url_re=None):
                 ))
 
                 result.append(MessageEntityTextUrl(
-                    offset=i, length=len(url_match.group(1)),
+                    offset=url_match.start(), length=len(url_match.group(1)),
                     url=_del_surrogate(url_match.group(2))
                 ))
                 i += len(url_match.group(1))
@@ -149,7 +140,7 @@ def unparse(text, entities, delimiters=None, url_fmt=None):
     :param entities: the MessageEntity's applied to the text.
     :return: a markdown-like text representing the combination of both inputs.
     """
-    if not entities:
+    if not text or not entities:
         return text
 
     if not delimiters:
@@ -181,28 +172,3 @@ def unparse(text, entities, delimiters=None, url_fmt=None):
             )
 
     return _del_surrogate(text)
-
-
-def get_inner_text(text, entity):
-    """
-    Gets the inner text that's surrounded by the given entity or entities.
-    For instance: text = 'hey!', entity = MessageEntityBold(2, 2) -> 'y!'.
-
-    :param text: the original text.
-    :param entity: the entity or entities that must be matched.
-    :return: a single result or a list of the text surrounded by the entities.
-    """
-    if isinstance(entity, TLObject):
-        entity = (entity,)
-        multiple = True
-    else:
-        multiple = False
-
-    text = _add_surrogate(text)
-    result = []
-    for e in entity:
-        start = e.offset
-        end = e.offset + e.length
-        result.append(_del_surrogate(text[start:end]))
-
-    return result if multiple else result[0]

telethon/extensions/tcp_client.py

@@ -91,7 +91,9 @@ class TcpClient:
                     # to none to recreate it on the next iteration
                     self._socket = None
                     time.sleep(timeout)
-                    timeout = min(timeout * 2, MAX_TIMEOUT)
+                    timeout *= 2
+                    if timeout > MAX_TIMEOUT:
+                        raise
                 else:
                     raise
 

telethon/network/__init__.py

@@ -5,4 +5,7 @@ with Telegram's servers and the protocol used (TCP full, abridged, etc.).
 from .mtproto_plain_sender import MtProtoPlainSender
 from .authenticator import do_authentication
 from .mtproto_sender import MtProtoSender
-from .connection import Connection, ConnectionMode
+from .connection import (
+    ConnectionTcpFull, ConnectionTcpAbridged, ConnectionTcpObfuscated,
+    ConnectionTcpIntermediate
+)

telethon/network/authenticator.py

@@ -11,8 +11,7 @@ from ..tl.types import (
     ServerDHInnerData, ClientDHInnerData, DhGenOk, DhGenRetry, DhGenFail
 )
 from .. import helpers as utils
-from ..crypto import AES, AuthKey, Factorization
-from ..crypto import rsa
+from ..crypto import AES, AuthKey, Factorization, rsa
 from ..errors import SecurityError
 from ..extensions import BinaryReader
 from ..network import MtProtoPlainSender

telethon/network/connection.py

@@ -1,332 +0,0 @@
-"""
-This module holds both the Connection class and the ConnectionMode enum,
-which specifies the protocol to be used by the Connection.
-"""
-import logging
-import os
-import struct
-from datetime import timedelta
-from zlib import crc32
-from enum import Enum
-
-import errno
-
-from ..crypto import AESModeCTR
-from ..extensions import TcpClient
-from ..errors import InvalidChecksumError
-
-__log__ = logging.getLogger(__name__)
-
-
-class ConnectionMode(Enum):
-    """Represents which mode should be used to stabilise a connection.
-
-    TCP_FULL: Default Telegram mode. Sends 12 additional bytes and
-              needs to calculate the CRC value of the packet itself.
-
-    TCP_INTERMEDIATE: Intermediate mode between TCP_FULL and TCP_ABRIDGED.
-                      Always sends 4 extra bytes for the packet length.
-
-    TCP_ABRIDGED: This is the mode with the lowest overhead, as it will
-                  only require 1 byte if the packet length is less than
-                  508 bytes (127 << 2, which is very common).
-
-    TCP_OBFUSCATED: Encodes the packet just like TCP_ABRIDGED, but encrypts
-                    every message with a randomly generated key using the
-                    AES-CTR mode so the packets are harder to discern.
-    """
-    TCP_FULL = 1
-    TCP_INTERMEDIATE = 2
-    TCP_ABRIDGED = 3
-    TCP_OBFUSCATED = 4
-
-
-class Connection:
-    """
-    Represents an abstract connection (TCP, TCP abridged...).
-    'mode' must be any of the ConnectionMode enumeration.
-
-    Note that '.send()' and '.recv()' refer to messages, which
-    will be packed accordingly, whereas '.write()' and '.read()'
-    work on plain bytes, with no further additions.
-    """
-
-    def __init__(self, mode=ConnectionMode.TCP_FULL,
-                 proxy=None, timeout=timedelta(seconds=5)):
-        """
-        Initializes a new connection.
-
-        :param mode: the ConnectionMode to be used.
-        :param proxy: whether to use a proxy or not.
-        :param timeout: timeout to be used for all operations.
-        """
-        self._mode = mode
-        self._send_counter = 0
-        self._aes_encrypt, self._aes_decrypt = None, None
-
-        # TODO Rename "TcpClient" as some sort of generic socket?
-        self.conn = TcpClient(proxy=proxy, timeout=timeout)
-
-        # Sending messages
-        if mode == ConnectionMode.TCP_FULL:
-            setattr(self, 'send', self._send_tcp_full)
-            setattr(self, 'recv', self._recv_tcp_full)
-
-        elif mode == ConnectionMode.TCP_INTERMEDIATE:
-            setattr(self, 'send', self._send_intermediate)
-            setattr(self, 'recv', self._recv_intermediate)
-
-        elif mode in (ConnectionMode.TCP_ABRIDGED,
-                      ConnectionMode.TCP_OBFUSCATED):
-            setattr(self, 'send', self._send_abridged)
-            setattr(self, 'recv', self._recv_abridged)
-
-        # Writing and reading from the socket
-        if mode == ConnectionMode.TCP_OBFUSCATED:
-            setattr(self, 'write', self._write_obfuscated)
-            setattr(self, 'read', self._read_obfuscated)
-        else:
-            setattr(self, 'write', self._write_plain)
-            setattr(self, 'read', self._read_plain)
-
-    def connect(self, ip, port):
-        """
-        Estabilishes a connection to IP:port.
-
-        :param ip: the IP to connect to.
-        :param port: the port to connect to.
-        """
-        try:
-            self.conn.connect(ip, port)
-        except OSError as e:
-            if e.errno == errno.EISCONN:
-                return  # Already connected, no need to re-set everything up
-            else:
-                raise
-
-        self._send_counter = 0
-        if self._mode == ConnectionMode.TCP_ABRIDGED:
-            self.conn.write(b'\xef')
-        elif self._mode == ConnectionMode.TCP_INTERMEDIATE:
-            self.conn.write(b'\xee\xee\xee\xee')
-        elif self._mode == ConnectionMode.TCP_OBFUSCATED:
-            self._setup_obfuscation()
-
-    def get_timeout(self):
-        """Returns the timeout used by the connection."""
-        return self.conn.timeout
-
-    def _setup_obfuscation(self):
-        """
-        Sets up the obfuscated protocol.
-        """
-        # Obfuscated messages secrets cannot start with any of these
-        keywords = (b'PVrG', b'GET ', b'POST', b'\xee' * 4)
-        while True:
-            random = os.urandom(64)
-            if (random[0] != b'\xef' and
-                    random[:4] not in keywords and
-                    random[4:4] != b'\0\0\0\0'):
-                # Invalid random generated
-                break
-
-        random = list(random)
-        random[56] = random[57] = random[58] = random[59] = 0xef
-        random_reversed = random[55:7:-1]  # Reversed (8, len=48)
-
-        # encryption has "continuous buffer" enabled
-        encrypt_key = bytes(random[8:40])
-        encrypt_iv = bytes(random[40:56])
-        decrypt_key = bytes(random_reversed[:32])
-        decrypt_iv = bytes(random_reversed[32:48])
-
-        self._aes_encrypt = AESModeCTR(encrypt_key, encrypt_iv)
-        self._aes_decrypt = AESModeCTR(decrypt_key, decrypt_iv)
-
-        random[56:64] = self._aes_encrypt.encrypt(bytes(random))[56:64]
-        self.conn.write(bytes(random))
-
-    def is_connected(self):
-        """
-        Determines whether the connection is alive or not.
-
-        :return: true if it's connected.
-        """
-        return self.conn.connected
-
-    def close(self):
-        """Closes the connection."""
-        self.conn.close()
-
-    def clone(self):
-        """Creates a copy of this Connection."""
-        return Connection(
-            mode=self._mode, proxy=self.conn.proxy, timeout=self.conn.timeout
-        )
-
-    # region Receive message implementations
-
-    def recv(self):
-        """Receives and unpacks a message"""
-        # Default implementation is just an error
-        raise ValueError('Invalid connection mode specified: ' + str(self._mode))
-
-    def _recv_tcp_full(self):
-        """
-        Receives a message from the network,
-        internally encoded using the TCP full protocol.
-
-        May raise InvalidChecksumError if the received data doesn't
-        match its valid checksum.
-
-        :return: the read message payload.
-        """
-        packet_len_seq = self.read(8)  # 4 and 4
-        packet_len, seq = struct.unpack('<ii', packet_len_seq)
-
-        # Sometimes Telegram seems to send a packet length of 0 (12)
-        # and after that, just a single byte. Not sure what this is.
-        # TODO Figure out what this is, and if there's a better fix.
-        if packet_len <= 12:
-            __log__.error('Read invalid packet length %d, '
-                          'reading data left:', packet_len)
-            while True:
-                try:
-                    __log__.error(repr(self.read(1)))
-                except TimeoutError:
-                    break
-            # Connection reset and hope it's fixed after
-            self.conn.close()
-            raise ConnectionResetError()
-
-        body = self.read(packet_len - 12)
-        checksum = struct.unpack('<I', self.read(4))[0]
-
-        valid_checksum = crc32(packet_len_seq + body)
-        if checksum != valid_checksum:
-            raise InvalidChecksumError(checksum, valid_checksum)
-
-        return body
-
-    def _recv_intermediate(self):
-        """
-        Receives a message from the network,
-        internally encoded using the TCP intermediate protocol.
-
-        :return: the read message payload.
-        """
-        return self.read(struct.unpack('<i', self.read(4))[0])
-
-    def _recv_abridged(self):
-        """
-        Receives a message from the network,
-        internally encoded using the TCP abridged protocol.
-
-        :return: the read message payload.
-        """
-        length = struct.unpack('<B', self.read(1))[0]
-        if length >= 127:
-            length = struct.unpack('<i', self.read(3) + b'\0')[0]
-
-        return self.read(length << 2)
-
-    # endregion
-
-    # region Send message implementations
-
-    def send(self, message):
-        """Encapsulates and sends the given message"""
-        # Default implementation is just an error
-        raise ValueError('Invalid connection mode specified: ' + str(self._mode))
-
-    def _send_tcp_full(self, message):
-        """
-        Encapsulates and sends the given message payload
-        using the TCP full mode (length, sequence, message, crc32).
-
-        :param message: the message to be sent.
-        """
-        # https://core.telegram.org/mtproto#tcp-transport
-        # total length, sequence number, packet and checksum (CRC32)
-        length = len(message) + 12
-        data = struct.pack('<ii', length, self._send_counter) + message
-        crc = struct.pack('<I', crc32(data))
-        self._send_counter += 1
-        self.write(data + crc)
-
-    def _send_intermediate(self, message):
-        """
-        Encapsulates and sends the given message payload
-        using the TCP intermediate mode (length, message).
-
-        :param message: the message to be sent.
-        """
-        self.write(struct.pack('<i', len(message)) + message)
-
-    def _send_abridged(self, message):
-        """
-        Encapsulates and sends the given message payload
-        using the TCP abridged mode (short length, message).
-
-        :param message: the message to be sent.
-        """
-        length = len(message) >> 2
-        if length < 127:
-            length = struct.pack('B', length)
-        else:
-            length = b'\x7f' + int.to_bytes(length, 3, 'little')
-
-        self.write(length + message)
-
-    # endregion
-
-    # region Read implementations
-
-    def read(self, length):
-        raise ValueError('Invalid connection mode specified: ' + str(self._mode))
-
-    def _read_plain(self, length):
-        """
-        Reads data from the socket connection.
-
-        :param length: how many bytes should be read.
-        :return: a byte sequence with len(data) == length
-        """
-        return self.conn.read(length)
-
-    def _read_obfuscated(self, length):
-        """
-        Reads data and decrypts from the socket connection.
-
-        :param length: how many bytes should be read.
-        :return: the decrypted byte sequence with len(data) == length
-        """
-        return self._aes_decrypt.encrypt(
-            self.conn.read(length)
-        )
-
-    # endregion
-
-    # region Write implementations
-
-    def write(self, data):
-        raise ValueError('Invalid connection mode specified: ' + str(self._mode))
-
-    def _write_plain(self, data):
-        """
-        Writes the given data through the socket connection.
-
-        :param data: the data in bytes to be written.
-        """
-        self.conn.write(data)
-
-    def _write_obfuscated(self, data):
-        """
-        Writes the given data through the socket connection,
-        using the obfuscated mode (AES encryption is applied on top).
-
-        :param data: the data in bytes to be written.
-        """
-        self.conn.write(self._aes_encrypt.encrypt(data))
-
-    # endregion

telethon/network/connection/__init__.py

@@ -0,0 +1,4 @@
+from .tcpfull import ConnectionTcpFull
+from .tcpabridged import ConnectionTcpAbridged
+from .tcpobfuscated import ConnectionTcpObfuscated
+from .tcpintermediate import ConnectionTcpIntermediate

telethon/network/connection/common.py

@@ -0,0 +1,61 @@
+"""
+This module holds the abstract `Connection` class.
+"""
+import abc
+from datetime import timedelta
+
+
+class Connection(abc.ABC):
+    """
+    Represents an abstract connection for Telegram.
+
+    Subclasses should implement the actual protocol
+    being used when encoding/decoding messages.
+    """
+    def __init__(self, proxy=None, timeout=timedelta(seconds=5)):
+        """
+        Initializes a new connection.
+
+        :param proxy: whether to use a proxy or not.
+        :param timeout: timeout to be used for all operations.
+        """
+        self._proxy = proxy
+        self._timeout = timeout
+
+    @abc.abstractmethod
+    def connect(self, ip, port):
+        raise NotImplementedError
+
+    @abc.abstractmethod
+    def get_timeout(self):
+        """Returns the timeout used by the connection."""
+        raise NotImplementedError
+
+    @abc.abstractmethod
+    def is_connected(self):
+        """
+        Determines whether the connection is alive or not.
+
+        :return: true if it's connected.
+        """
+        raise NotImplementedError
+
+    @abc.abstractmethod
+    def close(self):
+        """Closes the connection."""
+        raise NotImplementedError
+
+    @abc.abstractmethod
+    def clone(self):
+        """Creates a copy of this Connection."""
+        raise NotImplementedError
+
+    @abc.abstractmethod
+    def recv(self):
+        """Receives and unpacks a message"""
+        raise NotImplementedError
+
+    @abc.abstractmethod
+    def send(self, message):
+        """Encapsulates and sends the given message"""
+        raise NotImplementedError

telethon/network/connection/tcpabridged.py

@@ -0,0 +1,34 @@
+import struct
+
+from .tcpfull import ConnectionTcpFull
+
+
+class ConnectionTcpAbridged(ConnectionTcpFull):
+    """
+    This is the mode with the lowest overhead, as it will
+    only require 1 byte if the packet length is less than
+    508 bytes (127 << 2, which is very common).
+    """
+    def connect(self, ip, port):
+        result = super().connect(ip, port)
+        self.conn.write(b'\xef')
+        return result
+
+    def clone(self):
+        return ConnectionTcpAbridged(self._proxy, self._timeout)
+
+    def recv(self):
+        length = struct.unpack('<B', self.read(1))[0]
+        if length >= 127:
+            length = struct.unpack('<i', self.read(3) + b'\0')[0]
+
+        return self.read(length << 2)
+
+    def send(self, message):
+        length = len(message) >> 2
+        if length < 127:
+            length = struct.pack('B', length)
+        else:
+            length = b'\x7f' + int.to_bytes(length, 3, 'little')
+
+        self.write(length + message)

telethon/network/connection/tcpfull.py

@@ -0,0 +1,65 @@
+import errno
+import struct
+from datetime import timedelta
+from zlib import crc32
+
+from .common import Connection
+from ...errors import InvalidChecksumError
+from ...extensions import TcpClient
+
+
+class ConnectionTcpFull(Connection):
+    """
+    Default Telegram mode. Sends 12 additional bytes and
+    needs to calculate the CRC value of the packet itself.
+    """
+    def __init__(self, proxy=None, timeout=timedelta(seconds=5)):
+        super().__init__(proxy, timeout)
+        self._send_counter = 0
+        self.conn = TcpClient(proxy=self._proxy, timeout=self._timeout)
+        self.read = self.conn.read
+        self.write = self.conn.write
+
+    def connect(self, ip, port):
+        try:
+            self.conn.connect(ip, port)
+        except OSError as e:
+            if e.errno == errno.EISCONN:
+                return  # Already connected, no need to re-set everything up
+            else:
+                raise
+
+        self._send_counter = 0
+
+    def get_timeout(self):
+        return self.conn.timeout
+
+    def is_connected(self):
+        return self.conn.connected
+
+    def close(self):
+        self.conn.close()
+
+    def clone(self):
+        return ConnectionTcpFull(self._proxy, self._timeout)
+
+    def recv(self):
+        packet_len_seq = self.read(8)  # 4 and 4
+        packet_len, seq = struct.unpack('<ii', packet_len_seq)
+        body = self.read(packet_len - 12)
+        checksum = struct.unpack('<I', self.read(4))[0]
+
+        valid_checksum = crc32(packet_len_seq + body)
+        if checksum != valid_checksum:
+            raise InvalidChecksumError(checksum, valid_checksum)
+
+        return body
+
+    def send(self, message):
+        # https://core.telegram.org/mtproto#tcp-transport
+        # total length, sequence number, packet and checksum (CRC32)
+        length = len(message) + 12
+        data = struct.pack('<ii', length, self._send_counter) + message
+        crc = struct.pack('<I', crc32(data))
+        self._send_counter += 1
+        self.write(data + crc)

telethon/network/connection/tcpintermediate.py

@@ -0,0 +1,23 @@
+import struct
+
+from .tcpfull import ConnectionTcpFull
+
+
+class ConnectionTcpIntermediate(ConnectionTcpFull):
+    """
+    Intermediate mode between `ConnectionTcpFull` and `ConnectionTcpAbridged`.
+    Always sends 4 extra bytes for the packet length.
+    """
+    def connect(self, ip, port):
+        result = super().connect(ip, port)
+        self.conn.write(b'\xee\xee\xee\xee')
+        return result
+
+    def clone(self):
+        return ConnectionTcpIntermediate(self._proxy, self._timeout)
+
+    def recv(self):
+        return self.read(struct.unpack('<i', self.read(4))[0])
+
+    def send(self, message):
+        self.write(struct.pack('<i', len(message)) + message)

telethon/network/connection/tcpobfuscated.py

@@ -0,0 +1,50 @@
+import os
+from datetime import timedelta
+
+from .tcpfull import ConnectionTcpFull
+from .tcpabridged import ConnectionTcpAbridged
+from ...crypto import AESModeCTR
+
+
+class ConnectionTcpObfuscated(ConnectionTcpAbridged):
+    """
+    Encodes the packet just like `ConnectionTcpAbridged`, but encrypts
+    every message with a randomly generated key using the
+    AES-CTR mode so the packets are harder to discern.
+    """
+    def __init__(self, proxy=None, timeout=timedelta(seconds=5)):
+        super().__init__(proxy, timeout)
+        self._aes_encrypt, self._aes_decrypt = None, None
+        self.read = lambda s: self._aes_decrypt.encrypt(self.conn.read(s))
+        self.write = lambda d: self.conn.write(self._aes_encrypt.encrypt(d))
+
+    def connect(self, ip, port):
+        result = ConnectionTcpFull.connect(self, ip, port)
+        # Obfuscated messages secrets cannot start with any of these
+        keywords = (b'PVrG', b'GET ', b'POST', b'\xee' * 4)
+        while True:
+            random = os.urandom(64)
+            if (random[0] != b'\xef' and
+                    random[:4] not in keywords and
+                    random[4:4] != b'\0\0\0\0'):
+                break
+
+        random = list(random)
+        random[56] = random[57] = random[58] = random[59] = 0xef
+        random_reversed = random[55:7:-1]  # Reversed (8, len=48)
+
+        # encryption has "continuous buffer" enabled
+        encrypt_key = bytes(random[8:40])
+        encrypt_iv = bytes(random[40:56])
+        decrypt_key = bytes(random_reversed[:32])
+        decrypt_iv = bytes(random_reversed[32:48])
+
+        self._aes_encrypt = AESModeCTR(encrypt_key, encrypt_iv)
+        self._aes_decrypt = AESModeCTR(decrypt_key, decrypt_iv)
+
+        random[56:64] = self._aes_encrypt.encrypt(bytes(random))[56:64]
+        self.conn.write(bytes(random))
+        return result
+
+    def clone(self):
+        return ConnectionTcpObfuscated(self._proxy, self._timeout)

telethon/network/mtproto_sender.py

@@ -2,11 +2,10 @@
 This module contains the class used to communicate with Telegram's servers
 encrypting every packet, and relies on a valid AuthKey in the used Session.
 """
-import gzip
 import logging
 from threading import Lock
 
-from .. import helpers as utils
+from .. import helpers, utils
 from ..errors import (
     BadMessageError, InvalidChecksumError, BrokenAuthKeyError,
     rpc_message_to_error
@@ -14,6 +13,7 @@ from ..errors import (
 from ..extensions import BinaryReader
 from ..tl import TLMessage, MessageContainer, GzipPacked
 from ..tl.all_tlobjects import tlobjects
+from ..tl.functions import InvokeAfterMsgRequest
 from ..tl.functions.auth import LogOutRequest
 from ..tl.types import (
     MsgsAck, Pong, BadServerSalt, BadMsgNotification, FutureSalts,
@@ -57,6 +57,12 @@ class MtProtoSender:
         # Multithreading
         self._send_lock = Lock()
 
+        # If  we're  invoking something from an  update thread  but we're also
+        # receiving other request from the main thread (e.g. an update arrives
+        # and we need to process it)  we must  ensure that only one is calling
+        # receive at a given moment, since the receive step is fragile.
+        self._recv_lock = Lock()
+
     def connect(self):
         """Connects to the server."""
         self.connection.connect(self.session.server_address, self.session.port)
@@ -73,20 +79,32 @@ class MtProtoSender:
         """Disconnects from the server."""
         __log__.info('Disconnecting MtProtoSender...')
         self.connection.close()
-        self._need_confirmation.clear()
         self._clear_all_pending()
 
     # region Send and receive
 
-    def send(self, *requests):
+    def send(self, requests, ordered=False):
         """
         Sends the specified TLObject(s) (which must be requests),
         and acknowledging any message which needed confirmation.
 
         :param requests: the requests to be sent.
+        :param ordered: whether the requests should be invoked in the
+                        order in which they appear or they can be executed
+                        in arbitrary order in the server.
         """
-        # Finally send our packed request(s)
-        messages = [TLMessage(self.session, r) for r in requests]
+        if not utils.is_list_like(requests):
+            requests = (requests,)
+
+        if ordered:
+            requests = iter(requests)
+            messages = [TLMessage(self.session, next(requests))]
+            for r in requests:
+                messages.append(TLMessage(self.session, r,
+                                          after_id=messages[-1].msg_id))
+        else:
+            messages = [TLMessage(self.session, r) for r in requests]
+
         self._pending_receive.update({m.msg_id: m for m in messages})
 
         __log__.debug('Sending requests with IDs: %s', ', '.join(
@@ -132,8 +150,17 @@ class MtProtoSender:
             the UpdateState that will process all the received
             Update and Updates objects.
         """
+        if self._recv_lock.locked():
+            with self._recv_lock:
+                # Don't busy wait, acquire it but return because there's
+                # already a receive running and we don't want another one.
+                # It would lock until Telegram sent another update even if
+                # the current receive already received the expected response.
+                return
+
         try:
-            body = self.connection.recv()
+            with self._recv_lock:
+                body = self.connection.recv()
         except (BufferError, InvalidChecksumError):
             # TODO BufferError, we should spot the cause...
             # "No more bytes left"; something wrong happened, clear
@@ -163,7 +190,7 @@ class MtProtoSender:
         :param message: the TLMessage to be sent.
         """
         with self._send_lock:
-            self.connection.send(utils.pack_message(self.session, message))
+            self.connection.send(helpers.pack_message(self.session, message))
 
     def _decode_msg(self, body):
         """
@@ -179,7 +206,7 @@ class MtProtoSender:
                 raise BufferError("Can't decode packet ({})".format(body))
 
         with BinaryReader(body) as reader:
-            return utils.unpack_message(self.session, reader)
+            return helpers.unpack_message(self.session, reader)
 
     def _process_msg(self, msg_id, sequence, reader, state):
         """
@@ -476,11 +503,13 @@ class MtProtoSender:
         reader.read_int(signed=False)  # code
         request_id = reader.read_long()
         inner_code = reader.read_int(signed=False)
+        reader.seek(-4)
 
         __log__.debug('Received response for request with ID %d', request_id)
         request = self._pop_request(request_id)
 
         if inner_code == 0x2144ca19:  # RPC Error
+            reader.seek(4)
             if self.session.report_errors and request:
                 error = rpc_message_to_error(
                     reader.read_int(), reader.tgread_string(),
@@ -505,12 +534,10 @@ class MtProtoSender:
             return True  # All contents were read okay
 
         elif request:
-            if inner_code == 0x3072cfa1:  # GZip packed
-                unpacked_data = gzip.decompress(reader.tgread_bytes())
-                with BinaryReader(unpacked_data) as compressed_reader:
+            if inner_code == GzipPacked.CONSTRUCTOR_ID:
+                with BinaryReader(GzipPacked.read(reader)) as compressed_reader:
                     request.on_response(compressed_reader)
             else:
-                reader.seek(-4)
                 request.on_response(reader)
 
             self.session.process_entities(request.result)
@@ -525,10 +552,17 @@ class MtProtoSender:
         # session, it will be skipped by the handle_container().
         # For some reason this also seems to happen when downloading
         # photos, where the server responds with FileJpeg().
-        try:
-            obj = reader.tgread_object()
-        except Exception as e:
-            obj = '(failed to read: %s)' % e
+        def _try_read(r):
+            try:
+                return r.tgread_object()
+            except Exception as e:
+                return '(failed to read: {})'.format(e)
+
+        if inner_code == GzipPacked.CONSTRUCTOR_ID:
+            with BinaryReader(GzipPacked.read(reader)) as compressed_reader:
+                obj = _try_read(compressed_reader)
+        else:
+            obj = _try_read(reader)
 
         __log__.warning(
             'Lost request (ID %d) with code %s will be skipped, contents: %s',

telethon/sessions/abstract.py

@@ -67,6 +67,25 @@ class Session(ABC):
         """
         raise NotImplementedError
 
+    @abstractmethod
+    def get_update_state(self, entity_id):
+        """
+        Returns the ``UpdateState`` associated with the given `entity_id`.
+        If the `entity_id` is 0, it should return the ``UpdateState`` for
+        no specific channel (the "general" state). If no state is known
+        it should ``return None``.
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    def set_update_state(self, entity_id, state):
+        """
+        Sets the given ``UpdateState`` for the specified `entity_id`, which
+        should be 0 if the ``UpdateState`` is the "general" state (and not
+        for any specific channel).
+        """
+        raise NotImplementedError
+
     @abstractmethod
     def close(self):
         """

telethon/sessions/memory.py

@@ -35,6 +35,7 @@ class MemorySession(Session):
 
         self._files = {}
         self._entities = set()
+        self._update_states = {}
 
     def set_dc(self, dc_id, server_address, port):
         self._dc_id = dc_id or 0
@@ -57,6 +58,12 @@ class MemorySession(Session):
     def auth_key(self, value):
         self._auth_key = value
 
+    def get_update_state(self, entity_id):
+        return self._update_states.get(entity_id, None)
+
+    def set_update_state(self, entity_id, state):
+        self._update_states[entity_id] = state
+
     def close(self):
         pass
 
@@ -196,14 +203,15 @@ class MemorySession(Session):
             result = self.get_entity_rows_by_name(key)
 
         if result:
-            i, h = result  # unpack resulting tuple
-            i, k = utils.resolve_id(i)  # removes the mark and returns kind
-            if k == PeerUser:
-                return InputPeerUser(i, h)
-            elif k == PeerChat:
-                return InputPeerChat(i)
-            elif k == PeerChannel:
-                return InputPeerChannel(i, h)
+            entity_id, entity_hash = result  # unpack resulting tuple
+            entity_id, kind = utils.resolve_id(entity_id)
+            # removes the mark and returns type of entity
+            if kind == PeerUser:
+                return InputPeerUser(entity_id, entity_hash)
+            elif kind == PeerChat:
+                return InputPeerChat(entity_id)
+            elif kind == PeerChannel:
+                return InputPeerChannel(entity_id, entity_hash)
         else:
             raise ValueError('Could not find input entity with key ', key)
 

telethon/sessions/sqlite.py

@@ -1,3 +1,4 @@
+import datetime
 import json
 import os
 import sqlite3
@@ -5,6 +6,7 @@ from base64 import b64decode
 from os.path import isfile as file_exists
 from threading import Lock, RLock
 
+from telethon.tl import types
 from .memory import MemorySession, _SentFileType
 from .. import utils
 from ..crypto import AuthKey
@@ -13,12 +15,12 @@ from ..tl.types import (
 )
 
 EXTENSION = '.session'
-CURRENT_VERSION = 3  # database version
+CURRENT_VERSION = 4  # database version
 
 
 class SQLiteSession(MemorySession):
     """This session contains the required information to login into your
-       Telegram account. NEVER give the saved JSON file to anyone, since
+       Telegram account. NEVER give the saved session file to anyone, since
        they would gain instant access to all your messages and contacts.
 
        If you think the session has been compromised, close all the sessions
@@ -99,6 +101,14 @@ class SQLiteSession(MemorySession):
                     hash integer,
                     primary key(md5_digest, file_size, type)
                 )"""
+                ,
+                """update_state (
+                    id integer primary key,
+                    pts integer,
+                    qts integer,
+                    date integer,
+                    seq integer
+                )"""
             )
             c.execute("insert into version values (?)", (CURRENT_VERSION,))
             # Migrating from JSON -> new table and may have entities
@@ -141,32 +151,36 @@ class SQLiteSession(MemorySession):
 
     def _upgrade_database(self, old):
         c = self._cursor()
-        # old == 1 doesn't have the old sent_files so no need to drop
+        if old == 1:
+            old += 1
+            # old == 1 doesn't have the old sent_files so no need to drop
         if old == 2:
+            old += 1
             # Old cache from old sent_files lasts then a day anyway, drop
             c.execute('drop table sent_files')
-        self._create_table(c, """sent_files (
-            md5_digest blob,
-            file_size integer,
-            type integer,
-            id integer,
-            hash integer,
-            primary key(md5_digest, file_size, type)
-        )""")
+            self._create_table(c, """sent_files (
+                md5_digest blob,
+                file_size integer,
+                type integer,
+                id integer,
+                hash integer,
+                primary key(md5_digest, file_size, type)
+            )""")
+        if old == 3:
+            old += 1
+            self._create_table(c, """update_state (
+                id integer primary key,
+                pts integer,
+                qts integer,
+                date integer,
+                seq integer
+            )""")
         c.close()
 
     @staticmethod
     def _create_table(c, *definitions):
-        """
-        Creates a table given its definition 'name (columns).
-        If the sqlite version is >= 3.8.2, it will use "without rowid".
-        See http://www.sqlite.org/releaselog/3_8_2.html.
-        """
-        required = (3, 8, 2)
-        sqlite_v = tuple(int(x) for x in sqlite3.sqlite_version.split('.'))
-        extra = ' without rowid' if sqlite_v >= required else ''
         for definition in definitions:
-            c.execute('create table {}{}'.format(definition, extra))
+            c.execute('create table {}'.format(definition))
 
     # Data from sessions should be kept as properties
     # not to fetch the database every time we need it
@@ -206,6 +220,25 @@ class SQLiteSession(MemorySession):
             ))
             c.close()
 
+    def get_update_state(self, entity_id):
+        c = self._cursor()
+        row = c.execute('select pts, qts, date, seq from update_state '
+                        'where id = ?', (entity_id,)).fetchone()
+        c.close()
+        if row:
+            pts, qts, date, seq = row
+            date = datetime.datetime.utcfromtimestamp(date)
+            return types.updates.State(pts, qts, date, seq, unread_count=0)
+
+    def set_update_state(self, entity_id, state):
+        with self._db_lock:
+            c = self._cursor()
+            c.execute('insert or replace into update_state values (?,?,?,?,?)',
+                      (entity_id, state.pts, state.qts,
+                       state.date.timestamp(), state.seq))
+            c.close()
+            self.save()
+
     def save(self):
         """Saves the current session object as session_user_id.session"""
         with self._db_lock:

telethon/telegram_bare_client.py

@@ -11,9 +11,10 @@ from .crypto import rsa
 from .errors import (
     RPCError, BrokenAuthKeyError, ServerError, FloodWaitError,
     FloodTestPhoneWaitError, TypeNotFoundError, UnauthorizedError,
-    PhoneMigrateError, NetworkMigrateError, UserMigrateError
+    PhoneMigrateError, NetworkMigrateError, UserMigrateError, AuthKeyError,
+    RpcCallFailError
 )
-from .network import authenticator, MtProtoSender, Connection, ConnectionMode
+from .network import authenticator, MtProtoSender, ConnectionTcpFull
 from .sessions import Session, SQLiteSession
 from .tl import TLObject
 from .tl.all_tlobjects import LAYER
@@ -67,7 +68,8 @@ class TelegramBareClient:
     # region Initialization
 
     def __init__(self, session, api_id, api_hash,
-                 connection_mode=ConnectionMode.TCP_FULL,
+                 *,
+                 connection=ConnectionTcpFull,
                  use_ipv6=False,
                  proxy=None,
                  update_workers=None,
@@ -113,9 +115,10 @@ class TelegramBareClient:
         # that calls .connect(). Every other thread will spawn a new
         # temporary connection. The connection on this one is always
         # kept open so Telegram can send us updates.
-        self._sender = MtProtoSender(self.session, Connection(
-            mode=connection_mode, proxy=proxy, timeout=timeout
-        ))
+        if isinstance(connection, type):
+            connection = connection(proxy=proxy, timeout=timeout)
+
+        self._sender = MtProtoSender(self.session, connection)
 
         # Two threads may be calling reconnect() when the connection is lost,
         # we only want one to actually perform the reconnection.
@@ -227,6 +230,15 @@ class TelegramBareClient:
             self.disconnect()
             return self.connect(_sync_updates=_sync_updates)
 
+        except AuthKeyError as e:
+            # As of late March 2018 there were two AUTH_KEY_DUPLICATED
+            # reports. Retrying with a clean auth_key should fix this.
+            __log__.warning('Auth key error %s. Clearing it and retrying.', e)
+            self.disconnect()
+            self.session.auth_key = None
+            self.session.save()
+            return self.connect(_sync_updates=_sync_updates)
+
         except (RPCError, ConnectionError) as e:
             # Probably errors from the previous session, ignore them
             __log__.error('Connection failed due to %s', e)
@@ -265,6 +277,7 @@ class TelegramBareClient:
 
         # TODO Shall we clear the _exported_sessions, or may be reused?
         self._first_request = True  # On reconnect it will be first again
+        self.session.set_update_state(0, self.updates.get_update_state(0))
         self.session.close()
 
     def _reconnect(self, new_dc=None):
@@ -418,28 +431,51 @@ class TelegramBareClient:
 
     # region Invoking Telegram requests
 
-    def __call__(self, *requests, retries=5):
-        """Invokes (sends) a MTProtoRequest and returns (receives) its result.
-
-           The invoke will be retried up to 'retries' times before raising
-           RuntimeError().
+    def __call__(self, request, retries=5, ordered=False):
         """
+        Invokes (sends) one or more MTProtoRequests and returns (receives)
+        their result.
+
+        Args:
+            request (`TLObject` | `list`):
+                The request or requests to be invoked.
+
+            retries (`bool`, optional):
+                How many times the request should be retried automatically
+                in case it fails with a non-RPC error.
+
+               The invoke will be retried up to 'retries' times before raising
+               ``RuntimeError``.
+
+            ordered (`bool`, optional):
+                Whether the requests (if more than one was given) should be
+                executed sequentially on the server. They run in arbitrary
+                order by default.
+
+        Returns:
+            The result of the request (often a `TLObject`) or a list of
+            results if more than one request was given.
+        """
+        single = not utils.is_list_like(request)
+        if single:
+            request = (request,)
+
         if not all(isinstance(x, TLObject) and
-                   x.content_related for x in requests):
+                   x.content_related for x in request):
             raise TypeError('You can only invoke requests, not types!')
 
         if self._background_error:
             raise self._background_error
 
-        for request in requests:
-            request.resolve(self, utils)
+        for r in request:
+            r.resolve(self, utils)
 
         # For logging purposes
-        if len(requests) == 1:
-            which = type(requests[0]).__name__
+        if single:
+            which = type(request[0]).__name__
         else:
             which = '{} requests ({})'.format(
-                len(requests), [type(x).__name__ for x in requests])
+                len(request), [type(x).__name__ for x in request])
 
         # Determine the sender to be used (main or a new connection)
         __log__.debug('Invoking %s', which)
@@ -447,13 +483,13 @@ class TelegramBareClient:
             not self._idling.is_set() or self._reconnect_lock.locked()
 
         for retry in range(retries):
-            result = self._invoke(call_receive, *requests)
+            result = self._invoke(call_receive, request, ordered=ordered)
             if result is not None:
-                return result
+                return result[0] if single else result
 
             log = __log__.info if retry == 0 else __log__.warning
             log('Invoking %s failed %d times, connecting again and retrying',
-                [str(x) for x in requests], retry + 1)
+                which, retry + 1)
 
             sleep(1)
             # The ReadThread has priority when attempting reconnection,
@@ -464,13 +500,13 @@ class TelegramBareClient:
                     self._reconnect()
 
         raise RuntimeError('Number of retries reached 0 for {}.'.format(
-            [type(x).__name__ for x in requests]
+            which
         ))
 
     # Let people use client.invoke(SomeRequest()) instead client(...)
     invoke = __call__
 
-    def _invoke(self, call_receive, *requests):
+    def _invoke(self, call_receive, requests, ordered=False):
         try:
             # Ensure that we start with no previous errors (i.e. resending)
             for x in requests:
@@ -495,7 +531,7 @@ class TelegramBareClient:
                         self._wrap_init_connection(GetConfigRequest())
                     )
 
-            self._sender.send(*requests)
+            self._sender.send(requests, ordered=ordered)
 
             if not call_receive:
                 # TODO This will be slightly troublesome if we allow
@@ -530,6 +566,11 @@ class TelegramBareClient:
             __log__.warning('Connection was reset while invoking')
             if self._user_connected:
                 # Server disconnected us, __call__ will try reconnecting.
+                try:
+                    self._sender.disconnect()
+                except:
+                    pass
+
                 return None
             else:
                 # User never called .connect(), so raise this error.
@@ -546,10 +587,7 @@ class TelegramBareClient:
                 # rejected by the other party as a whole."
                 return None
 
-            if len(requests) == 1:
-                return requests[0].result
-            else:
-                return [x.result for x in requests]
+            return [x.result for x in requests]
 
         except (PhoneMigrateError, NetworkMigrateError,
                 UserMigrateError) as e:
@@ -559,11 +597,11 @@ class TelegramBareClient:
             # be on the very first connection (not authorized, not running),
             # but may be an issue for people who actually travel?
             self._reconnect(new_dc=e.new_dc)
-            return self._invoke(call_receive, *requests)
+            return self._invoke(call_receive, requests)
 
-        except ServerError as e:
+        except (ServerError, RpcCallFailError) as e:
             # Telegram is having some issues, just retry
-            __log__.error('Telegram servers are having internal errors %s', e)
+            __log__.warning('Telegram is having internal issues: %s', e)
 
         except (FloodWaitError, FloodTestPhoneWaitError) as e:
             __log__.warning('Request invoked too often, wait %ds', e.seconds)

telethon/tl/custom/__init__.py

@@ -1,3 +1,5 @@
 from .draft import Draft
 from .dialog import Dialog
 from .input_sized_file import InputSizedFile
+from .messagebutton import MessageButton
+from .message import Message

telethon/tl/custom/dialog.py

@@ -1,4 +1,5 @@
 from . import Draft
+from .. import TLObject, types
 from ... import utils
 
 
@@ -13,7 +14,7 @@ class Dialog:
         dialog (:tl:`Dialog`):
             The original ``Dialog`` instance.
 
-        pinned (:obj:`bool`):
+        pinned (`bool`):
             Whether this dialog is pinned to the top or not.
 
         message (:tl:`Message`):
@@ -21,33 +22,46 @@ class Dialog:
             will not be updated when new messages arrive, it's only set
             on creation of the instance.
 
-        date (:obj:`datetime`):
+        date (`datetime`):
             The date of the last message sent on this dialog.
 
-        entity (:obj:`entity`):
+        entity (`entity`):
             The entity that belongs to this dialog (user, chat or channel).
 
         input_entity (:tl:`InputPeer`):
             Input version of the entity.
 
-        id (:obj:`int`):
+        id (`int`):
             The marked ID of the entity, which is guaranteed to be unique.
 
-        name (:obj:`str`):
+        name (`str`):
             Display name for this dialog. For chats and channels this is
             their title, and for users it's "First-Name Last-Name".
 
-        unread_count (:obj:`int`):
+        title (`str`):
+            Alias for `name`.
+
+        unread_count (`int`):
             How many messages are currently unread in this dialog. Note that
             this value won't update when new messages arrive.
 
-        unread_mentions_count (:obj:`int`):
+        unread_mentions_count (`int`):
             How many mentions are currently unread in this dialog. Note that
             this value won't update when new messages arrive.
 
-        draft (:obj:`telethon.tl.custom.draft.Draft`):
+        draft (`telethon.tl.custom.draft.Draft`):
             The draft object in this dialog. It will not be ``None``,
             so you can call ``draft.set_message(...)``.
+
+        is_user (`bool`):
+            ``True`` if the `entity` is a :tl:`User`.
+
+        is_group (`bool`):
+            ``True`` if the `entity` is a :tl:`Chat`
+            or a :tl:`Channel` megagroup.
+
+        is_channel (`bool`):
+            ``True`` if the `entity` is a :tl:`Channel`.
     """
     def __init__(self, client, dialog, entities, messages):
         # Both entities and messages being dicts {ID: item}
@@ -60,16 +74,39 @@ class Dialog:
         self.entity = entities[utils.get_peer_id(dialog.peer)]
         self.input_entity = utils.get_input_peer(self.entity)
         self.id = utils.get_peer_id(self.entity)  # ^ May be InputPeerSelf()
-        self.name = utils.get_display_name(self.entity)
+        self.name = self.title = utils.get_display_name(self.entity)
 
         self.unread_count = dialog.unread_count
         self.unread_mentions_count = dialog.unread_mentions_count
 
         self.draft = Draft(client, dialog.peer, dialog.draft)
 
+        self.is_user = isinstance(self.entity, types.User)
+        self.is_group = (
+            isinstance(self.entity, types.Chat) or
+            (isinstance(self.entity, types.Channel) and self.entity.megagroup)
+        )
+        self.is_channel = isinstance(self.entity, types.Channel)
+
     def send_message(self, *args, **kwargs):
         """
         Sends a message to this dialog. This is just a wrapper around
         ``client.send_message(dialog.input_entity, *args, **kwargs)``.
         """
         return self._client.send_message(self.input_entity, *args, **kwargs)
+
+    def to_dict(self):
+        return {
+            '_': 'Dialog',
+            'name': self.name,
+            'date': self.date,
+            'draft': self.draft,
+            'message': self.message,
+            'entity': self.entity,
+        }
+
+    def __str__(self):
+        return TLObject.pretty_format(self.to_dict())
+
+    def stringify(self):
+        return TLObject.pretty_format(self.to_dict(), indent=0)

telethon/tl/custom/draft.py

@@ -1,8 +1,11 @@
 import datetime
 
+from .. import TLObject
 from ..functions.messages import SaveDraftRequest
 from ..types import UpdateDraftMessage, DraftMessage
+from ...errors import RPCError
 from ...extensions import markdown
+from ...utils import Default
 
 
 class Draft:
@@ -12,13 +15,13 @@ class Draft:
     instances of this class when calling :meth:`get_drafts()`.
 
     Args:
-        date (:obj:`datetime`):
+        date (`datetime`):
             The date of the draft.
 
-        link_preview (:obj:`bool`):
+        link_preview (`bool`):
             Whether the link preview is enabled or not.
 
-        reply_to_msg_id (:obj:`int`):
+        reply_to_msg_id (`int`):
             The message ID that the draft will reply to.
     """
     def __init__(self, client, peer, draft):
@@ -80,7 +83,7 @@ class Draft:
         """
         return not self._text
 
-    def set_message(self, text=None, reply_to=0, parse_mode='md',
+    def set_message(self, text=None, reply_to=0, parse_mode=Default,
                     link_preview=None):
         """
         Changes the draft message on the Telegram servers. The changes are
@@ -125,7 +128,7 @@ class Draft:
 
         return result
 
-    def send(self, clear=True, parse_mode='md'):
+    def send(self, clear=True, parse_mode=Default):
         """
         Sends the contents of this draft to the dialog. This is just a
         wrapper around ``send_message(dialog.input_entity, *args, **kwargs)``.
@@ -141,3 +144,24 @@ class Draft:
         Deletes this draft, and returns ``True`` on success.
         """
         return self.set_message(text='')
+
+    def to_dict(self):
+        try:
+            entity = self.entity
+        except RPCError as e:
+            entity = e
+
+        return {
+            '_': 'Draft',
+            'text': self.text,
+            'entity': entity,
+            'date': self.date,
+            'link_preview': self.link_preview,
+            'reply_to_msg_id': self.reply_to_msg_id
+        }
+
+    def __str__(self):
+        return TLObject.pretty_format(self.to_dict())
+
+    def stringify(self):
+        return TLObject.pretty_format(self.to_dict(), indent=0)

telethon/tl/custom/message.py

@@ -0,0 +1,604 @@
+from .. import types
+from ...utils import get_input_peer, get_peer_id, get_inner_text
+from .messagebutton import MessageButton
+
+
+class Message:
+    """
+    Custom class that encapsulates a message providing an abstraction to
+    easily access some commonly needed features (such as the markdown text
+    or the text for a given message entity).
+
+    Attributes:
+
+        original_message (:tl:`Message`):
+            The original :tl:`Message` object.
+
+        Any other attribute:
+            Attributes not described here are the same as those available
+            in the original :tl:`Message`.
+    """
+    def __init__(self, client, original, entities, input_chat):
+        # Share the original dictionary. Modifications to this
+        # object should also be reflected in the original one.
+        # This way there's no need to worry about get/setattr.
+        self.__dict__ = original.__dict__
+        self.original_message = original
+        self.stringify = self.original_message.stringify
+        self.to_dict = self.original_message.to_dict
+        self._client = client
+        self._text = None
+        self._reply_message = None
+        self._buttons = None
+        self._buttons_flat = None
+        self._sender = entities.get(self.original_message.from_id)
+        self._chat = entities.get(get_peer_id(self.original_message.to_id))
+        if self._sender:
+            self._input_sender = get_input_peer(self._sender)
+        else:
+            self._input_sender = None
+        self._input_chat = input_chat
+        self._fwd_from_entity = None
+        if getattr(self.original_message, 'fwd_from', None):
+            fwd = self.original_message.fwd_from
+            if fwd.from_id:
+                self._fwd_from_entity = entities.get(fwd.from_id)
+            elif fwd.channel_id:
+                self._fwd_from_entity = entities.get(get_peer_id(
+                    types.PeerChannel(fwd.channel_id)))
+
+    def __new__(cls, client, original, entities, input_chat):
+        if isinstance(original, types.Message):
+            return super().__new__(_CustomMessage)
+        elif isinstance(original, types.MessageService):
+            return super().__new__(_CustomMessageService)
+        else:
+            return cls
+
+    def __str__(self):
+        return str(self.original_message)
+
+    def __repr__(self):
+        return repr(self.original_message)
+
+    def __bytes__(self):
+        return bytes(self.original_message)
+
+    @property
+    def client(self):
+        """
+        Returns the `telethon.telegram_client.TelegramClient` instance that
+        created this instance.
+        """
+        return self._client
+
+    @property
+    def text(self):
+        """
+        The message text, formatted using the client's default parse mode.
+        Will be ``None`` for :tl:`MessageService`.
+        """
+        if self._text is None\
+                and isinstance(self.original_message, types.Message):
+            if not self._client.parse_mode:
+                return self.original_message.message
+            self._text = self._client.parse_mode.unparse(
+                self.original_message.message, self.original_message.entities)
+        return self._text
+
+    @text.setter
+    def text(self, value):
+        if isinstance(self.original_message, types.Message):
+            if self._client.parse_mode:
+                msg, ent = self._client.parse_mode.parse(value)
+            else:
+                msg, ent = value, []
+            self.original_message.message = msg
+            self.original_message.entities = ent
+            self._text = value
+
+    @property
+    def raw_text(self):
+        """
+        The raw message text, ignoring any formatting.
+        Will be ``None`` for :tl:`MessageService`.
+        """
+        if isinstance(self.original_message, types.Message):
+            return self.original_message.message
+
+    @raw_text.setter
+    def raw_text(self, value):
+        if isinstance(self.original_message, types.Message):
+            self.original_message.message = value
+            self.original_message.entities = []
+            self._text = None
+
+    @property
+    def message(self):
+        """
+        The raw message text, ignoring any formatting.
+        Will be ``None`` for :tl:`MessageService`.
+        """
+        return self.raw_text
+
+    @message.setter
+    def message(self, value):
+        self.raw_text = value
+
+    @property
+    def action(self):
+        """
+        The :tl:`MessageAction` for the :tl:`MessageService`.
+        Will be ``None`` for :tl:`Message`.
+        """
+        if isinstance(self.original_message, types.MessageService):
+            return self.original_message.action
+
+    def _reload_message(self):
+        """
+        Re-fetches this message to reload the sender and chat entities,
+        along with their input versions.
+        """
+        try:
+            chat = self.input_chat if self.is_channel else None
+            msg = self._client.get_messages(chat, ids=self.original_message.id)
+        except ValueError:
+            return  # We may not have the input chat/get message failed
+        if not msg:
+            return  # The message may be deleted and it will be None
+
+        self._sender = msg._sender
+        self._input_sender = msg._input_sender
+        self._chat = msg._chat
+        self._input_chat = msg._input_chat
+
+    @property
+    def sender(self):
+        """
+        This (:tl:`User`) may make an API call the first time to get
+        the most up to date version of the sender (mostly when the event
+        doesn't belong to a channel), so keep that in mind.
+
+        `input_sender` needs to be available (often the case).
+        """
+        if self._sender is None:
+            try:
+                self._sender = self._client.get_entity(self.input_sender)
+            except ValueError:
+                self._reload_message()
+        return self._sender
+
+    @property
+    def chat(self):
+        if self._chat is None:
+            try:
+                self._chat = self._client.get_entity(self.input_chat)
+            except ValueError:
+                self._reload_message()
+        return self._chat
+
+    @property
+    def input_sender(self):
+        """
+        This (:tl:`InputPeer`) is the input version of the user who
+        sent the message. Similarly to `input_chat`, this doesn't have
+        things like username or similar, but still useful in some cases.
+
+        Note that this might not be available if the library can't
+        find the input chat, or if the message a broadcast on a channel.
+        """
+        if self._input_sender is None:
+            if self.is_channel and not self.is_group:
+                return None
+            if self._sender is not None:
+                self._input_sender = get_input_peer(self._sender)
+            else:
+                try:
+                    self._input_sender = self._client.get_input_entity(
+                        self.original_message.from_id)
+                except ValueError:
+                    self._reload_message()
+        return self._input_sender
+
+    @property
+    def input_chat(self):
+        """
+        This (:tl:`InputPeer`) is the input version of the chat where the
+        message was sent. Similarly to `input_sender`, this doesn't have
+        things like username or similar, but still useful in some cases.
+
+        Note that this might not be available if the library doesn't know
+        where the message came from, and it may fetch the dialogs to try
+        to find it in the worst case.
+        """
+        if self._input_chat is None:
+            if self._chat is None:
+                try:
+                    self._chat = self._client.get_input_entity(
+                        self.original_message.to_id)
+                except ValueError:
+                    # There's a chance that the chat is a recent new dialog.
+                    # The input chat cannot rely on ._reload_message() because
+                    # said method may need the input chat.
+                    target = self.chat_id
+                    for d in self._client.iter_dialogs(100):
+                        if d.id == target:
+                            self._chat = d.entity
+                            break
+            if self._chat is not None:
+                self._input_chat = get_input_peer(self._chat)
+
+        return self._input_chat
+
+    @property
+    def sender_id(self):
+        """
+        Returns the marked sender integer ID, if present.
+        """
+        return self.original_message.from_id
+
+    @property
+    def chat_id(self):
+        """
+        Returns the marked chat integer ID.
+        """
+        return get_peer_id(self.original_message.to_id)
+
+    @property
+    def is_private(self):
+        """True if the message was sent as a private message."""
+        return isinstance(self.original_message.to_id, types.PeerUser)
+
+    @property
+    def is_group(self):
+        """True if the message was sent on a group or megagroup."""
+        return not self.original_message.broadcast and isinstance(
+            self.original_message.to_id, (types.PeerChat, types.PeerChannel))
+
+    @property
+    def is_channel(self):
+        """True if the message was sent on a megagroup or channel."""
+        return isinstance(self.original_message.to_id, types.PeerChannel)
+
+    @property
+    def is_reply(self):
+        """True if the message is a reply to some other or not."""
+        return bool(self.original_message.reply_to_msg_id)
+
+    @property
+    def buttons(self):
+        """
+        Returns a matrix (list of lists) containing all buttons of the message
+        as `telethon.tl.custom.messagebutton.MessageButton` instances.
+        """
+        if self._buttons is None and self.original_message.reply_markup:
+            if isinstance(self.original_message.reply_markup, (
+                    types.ReplyInlineMarkup, types.ReplyKeyboardMarkup)):
+                self._buttons = [[
+                    MessageButton(self._client, button, self.input_sender,
+                                  self.input_chat, self.original_message.id)
+                    for button in row.buttons
+                ] for row in self.original_message.reply_markup.rows]
+                self._buttons_flat = [x for row in self._buttons for x in row]
+        return self._buttons
+
+    @property
+    def button_count(self):
+        """
+        Returns the total button count.
+        """
+        return len(self._buttons_flat) if self.buttons else 0
+
+    @property
+    def photo(self):
+        """
+        If the message media is a photo,
+        this returns the :tl:`Photo` object.
+        """
+        if isinstance(self.original_message.media, types.MessageMediaPhoto):
+            photo = self.original_message.media.photo
+            if isinstance(photo, types.Photo):
+                return photo
+
+    @property
+    def document(self):
+        """
+        If the message media is a document,
+        this returns the :tl:`Document` object.
+        """
+        if isinstance(self.original_message.media, types.MessageMediaDocument):
+            doc = self.original_message.media.document
+            if isinstance(doc, types.Document):
+                return doc
+
+    def _document_by_attribute(self, kind, condition=None):
+        """
+        Helper method to return the document only if it has an attribute
+        that's an instance of the given kind, and passes the condition.
+        """
+        doc = self.document
+        if doc:
+            for attr in doc.attributes:
+                if isinstance(attr, kind):
+                    if not condition or condition(doc):
+                        return doc
+
+    @property
+    def audio(self):
+        """
+        If the message media is a document with an Audio attribute,
+        this returns the :tl:`Document` object.
+        """
+        return self._document_by_attribute(types.DocumentAttributeAudio,
+                                           lambda attr: not attr.voice)
+
+    @property
+    def voice(self):
+        """
+        If the message media is a document with a Voice attribute,
+        this returns the :tl:`Document` object.
+        """
+        return self._document_by_attribute(types.DocumentAttributeAudio,
+                                           lambda attr: attr.voice)
+
+    @property
+    def video(self):
+        """
+        If the message media is a document with a Video attribute,
+        this returns the :tl:`Document` object.
+        """
+        return self._document_by_attribute(types.DocumentAttributeVideo)
+
+    @property
+    def video_note(self):
+        """
+        If the message media is a document with a Video attribute,
+        this returns the :tl:`Document` object.
+        """
+        return self._document_by_attribute(types.DocumentAttributeVideo,
+                                           lambda attr: attr.round_message)
+
+    @property
+    def gif(self):
+        """
+        If the message media is a document with an Animated attribute,
+        this returns the :tl:`Document` object.
+        """
+        return self._document_by_attribute(types.DocumentAttributeAnimated)
+
+    @property
+    def sticker(self):
+        """
+        If the message media is a document with a Sticker attribute,
+        this returns the :tl:`Document` object.
+        """
+        return self._document_by_attribute(types.DocumentAttributeSticker)
+
+    @property
+    def out(self):
+        """
+        Whether the message is outgoing (i.e. you sent it from
+        another session) or incoming (i.e. someone else sent it).
+        """
+        return self.original_message.out
+
+    @property
+    def reply_message(self):
+        """
+        The `telethon.tl.custom.message.Message` that this message is replying
+        to, or ``None``.
+
+        Note that this will make a network call to fetch the message and
+        will later be cached.
+        """
+        if self._reply_message is None:
+            if not self.original_message.reply_to_msg_id:
+                return None
+            self._reply_message = self._client.get_messages(
+                self.input_chat if self.is_channel else None,
+                ids=self.original_message.reply_to_msg_id
+            )
+
+        return self._reply_message
+
+    @property
+    def fwd_from_entity(self):
+        """
+        If the :tl:`Message` is a forwarded message, returns the :tl:`User`
+        or :tl:`Channel` who originally sent the message, or ``None``.
+        """
+        if self._fwd_from_entity is None:
+            if getattr(self.original_message, 'fwd_from', None):
+                fwd = self.original_message.fwd_from
+                if fwd.from_id:
+                    self._fwd_from_entity = self._client.get_entity(
+                        fwd.from_id)
+                elif fwd.channel_id:
+                    self._fwd_from_entity = self._client.get_entity(
+                        get_peer_id(types.PeerChannel(fwd.channel_id)))
+        return self._fwd_from_entity
+
+    def respond(self, *args, **kwargs):
+        """
+        Responds to the message (not as a reply). Shorthand for
+        `telethon.telegram_client.TelegramClient.send_message` with
+        ``entity`` already set.
+        """
+        return self._client.send_message(self.input_chat, *args, **kwargs)
+
+    def reply(self, *args, **kwargs):
+        """
+        Replies to the message (as a reply). Shorthand for
+        `telethon.telegram_client.TelegramClient.send_message` with
+        both ``entity`` and ``reply_to`` already set.
+        """
+        kwargs['reply_to'] = self.original_message.id
+        return self._client.send_message(self.original_message.to_id,
+                                         *args, **kwargs)
+
+    def forward_to(self, *args, **kwargs):
+        """
+        Forwards the message. Shorthand for
+        `telethon.telegram_client.TelegramClient.forward_messages` with
+        both ``messages`` and ``from_peer`` already set.
+
+        If you need to forward more than one message at once, don't use
+        this `forward_to` method. Use a
+        `telethon.telegram_client.TelegramClient` instance directly.
+        """
+        kwargs['messages'] = self.original_message.id
+        kwargs['from_peer'] = self.input_chat
+        return self._client.forward_messages(*args, **kwargs)
+
+    def edit(self, *args, **kwargs):
+        """
+        Edits the message iff it's outgoing. Shorthand for
+        `telethon.telegram_client.TelegramClient.edit_message` with
+        both ``entity`` and ``message`` already set.
+
+        Returns ``None`` if the message was incoming, or the edited
+        :tl:`Message` otherwise.
+        """
+        if self.original_message.fwd_from:
+            return None
+        if not self.original_message.out:
+            if not isinstance(self.original_message.to_id, types.PeerUser):
+                return None
+            me = self._client.get_me(input_peer=True)
+            if self.original_message.to_id.user_id != me.user_id:
+                return None
+
+        return self._client.edit_message(
+            self.input_chat, self.original_message, *args, **kwargs)
+
+    def delete(self, *args, **kwargs):
+        """
+        Deletes the message. You're responsible for checking whether you
+        have the permission to do so, or to except the error otherwise.
+        Shorthand for
+        `telethon.telegram_client.TelegramClient.delete_messages` with
+        ``entity`` and ``message_ids`` already set.
+
+        If you need to delete more than one message at once, don't use
+        this `delete` method. Use a
+        `telethon.telegram_client.TelegramClient` instance directly.
+        """
+        return self._client.delete_messages(
+            self.input_chat, [self.original_message], *args, **kwargs)
+
+    def download_media(self, *args, **kwargs):
+        """
+        Downloads the media contained in the message, if any.
+        `telethon.telegram_client.TelegramClient.download_media` with
+        the ``message`` already set.
+        """
+        return self._client.download_media(self.original_message,
+                                           *args, **kwargs)
+
+    def get_entities_text(self, cls=None):
+        """
+        Returns a list of tuples [(:tl:`MessageEntity`, `str`)], the string
+        being the inner text of the message entity (like bold, italics, etc).
+
+        Args:
+            cls (`type`):
+                Returns entities matching this type only. For example,
+                the following will print the text for all ``code`` entities:
+
+                >>> from telethon.tl.types import MessageEntityCode
+                >>>
+                >>> m = Message(...)
+                >>> for _, inner_text in m.get_entities_text(MessageEntityCode):
+                >>>     print(inner_text)
+        """
+        if cls and self.original_message.entities:
+            texts = get_inner_text(
+                self.original_message.message,
+                [c for c in self.original_message.entities
+                 if isinstance(c, cls)]
+            )
+        else:
+            texts = get_inner_text(self.original_message.message,
+                                   self.original_message.entities)
+        return list(zip(self.original_message.entities, texts))
+
+    def click(self, i=None, j=None, *, text=None, filter=None):
+        """
+        Clicks the inline keyboard button of the message, if any.
+
+        If the message has a non-inline keyboard, clicking it will
+        send the message, switch to inline, or open its URL.
+
+        Does nothing if the message has no buttons.
+
+        Args:
+            i (`int`):
+                Clicks the i'th button (starting from the index 0).
+                Will ``raise IndexError`` if out of bounds. Example:
+
+                >>> message = Message(...)
+                >>> # Clicking the 3rd button
+                >>> # [button1] [button2]
+                >>> # [     button3     ]
+                >>> # [button4] [button5]
+                >>> message.click(2)  # index
+
+            j (`int`):
+                Clicks the button at position (i, j), these being the
+                indices for the (row, column) respectively. Example:
+
+                >>> # Clicking the 2nd button on the 1st row.
+                >>> # [button1] [button2]
+                >>> # [     button3     ]
+                >>> # [button4] [button5]
+                >>> message.click(0, 1)  # (row, column)
+
+                This is equivalent to ``message.buttons[0][1].click()``.
+
+            text (`str` | `callable`):
+                Clicks the first button with the text "text". This may
+                also be a callable, like a ``re.compile(...).match``,
+                and the text will be passed to it.
+
+            filter (`callable`):
+                Clicks the first button for which the callable
+                returns ``True``. The callable should accept a single
+                `telethon.tl.custom.messagebutton.MessageButton` argument.
+        """
+        if sum(int(x is not None) for x in (i, text, filter)) >= 2:
+            raise ValueError('You can only set either of i, text or filter')
+
+        if not self.buttons:
+            return  # Accessing the property sets self._buttons[_flat]
+
+        if text is not None:
+            if callable(text):
+                for button in self._buttons_flat:
+                    if text(button.text):
+                        return button.click()
+            else:
+                for button in self._buttons_flat:
+                    if button.text == text:
+                        return button.click()
+            return
+
+        if filter is not None:
+            for button in self._buttons_flat:
+                if filter(button):
+                    return button.click()
+            return
+
+        if i is None:
+            i = 0
+        if j is None:
+            return self._buttons_flat[i].click()
+        else:
+            return self._buttons[i][j].click()
+
+
+class _CustomMessage(Message, types.Message):
+    pass
+
+
+class _CustomMessageService(Message, types.MessageService):
+    pass

telethon/tl/custom/messagebutton.py

@@ -0,0 +1,73 @@
+from .. import types, functions
+import webbrowser
+
+
+class MessageButton:
+    """
+    Custom class that encapsulates a message providing an abstraction to
+    easily access some commonly needed features (such as the markdown text
+    or the text for a given message entity).
+
+    Attributes:
+
+        button (:tl:`KeyboardButton`):
+            The original :tl:`KeyboardButton` object.
+    """
+    def __init__(self, client, original, from_user, chat, msg_id):
+        self.button = original
+        self._from = from_user
+        self._chat = chat
+        self._msg_id = msg_id
+        self._client = client
+
+    @property
+    def client(self):
+        """
+        Returns the `telethon.telegram_client.TelegramClient` instance that
+        created this instance.
+        """
+        return self._client
+
+    @property
+    def text(self):
+        """The text string of the button."""
+        return self.button.text
+
+    @property
+    def data(self):
+        """The ``bytes`` data for :tl:`KeyboardButtonCallback` objects."""
+        if isinstance(self.button, types.KeyboardButtonCallback):
+            return self.button.data
+
+    @property
+    def inline_query(self):
+        """The query ``str`` for :tl:`KeyboardButtonSwitchInline` objects."""
+        if isinstance(self.button, types.KeyboardButtonSwitchInline):
+            return self.button.query
+
+    @property
+    def url(self):
+        """The url ``str`` for :tl:`KeyboardButtonUrl` objects."""
+        if isinstance(self.button, types.KeyboardButtonUrl):
+            return self.button.url
+
+    def click(self):
+        """
+        Clicks the inline keyboard button of the message, if any.
+
+        If the message has a non-inline keyboard, clicking it will
+        send the message, switch to inline, or open its URL.
+        """
+        if isinstance(self.button, types.KeyboardButton):
+            return self._client.send_message(
+                self._chat, self.button.text, reply_to=self._msg_id)
+        elif isinstance(self.button, types.KeyboardButtonCallback):
+            return self._client(functions.messages.GetBotCallbackAnswerRequest(
+                peer=self._chat, msg_id=self._msg_id, data=self.button.data
+            ), retries=1)
+        elif isinstance(self.button, types.KeyboardButtonSwitchInline):
+            return self._client(functions.messages.StartBotRequest(
+                bot=self._from, peer=self._chat, start_param=self.button.query
+            ), retries=1)
+        elif isinstance(self.button, types.KeyboardButtonUrl):
+            return webbrowser.open(self.button.url)

telethon/tl/tl_message.py

@@ -1,11 +1,12 @@
 import struct
 
 from . import TLObject, GzipPacked
+from ..tl.functions import InvokeAfterMsgRequest
 
 
 class TLMessage(TLObject):
     """https://core.telegram.org/mtproto/service_messages#simple-container"""
-    def __init__(self, session, request):
+    def __init__(self, session, request, after_id=None):
         super().__init__()
         del self.content_related
         self.msg_id = session.get_new_msg_id()
@@ -13,16 +14,27 @@ class TLMessage(TLObject):
         self.request = request
         self.container_msg_id = None
 
+        # After which message ID this one should run. We do this so
+        # InvokeAfterMsgRequest is transparent to the user and we can
+        # easily invoke after while confirming the original request.
+        self.after_id = after_id
+
     def to_dict(self, recursive=True):
         return {
             'msg_id': self.msg_id,
             'seq_no': self.seq_no,
             'request': self.request,
             'container_msg_id': self.container_msg_id,
+            'after_id': self.after_id
         }
 
     def __bytes__(self):
-        body = GzipPacked.gzip_if_smaller(self.request)
+        if self.after_id is None:
+            body = GzipPacked.gzip_if_smaller(self.request)
+        else:
+            body = GzipPacked.gzip_if_smaller(
+                InvokeAfterMsgRequest(self.after_id, self.request))
+
         return struct.pack('<qii', self.msg_id, self.seq_no, len(body)) + body
 
     def __str__(self):

telethon/tl/tlobject.py

@@ -5,12 +5,35 @@ from threading import Event
 
 class TLObject:
     def __init__(self):
-        self.confirm_received = Event()
         self.rpc_error = None
         self.result = None
 
         # These should be overrode
         self.content_related = False  # Only requests/functions/queries are
+        
+        # Internal parameter to tell pickler in which state Event object was
+        self._event_is_set = False 
+        self._set_event()
+
+    def _set_event(self):
+        self.confirm_received = Event()
+        
+        # Set Event state to 'set' if needed
+        if self._event_is_set:
+            self.confirm_received.set()
+
+    def __getstate__(self):
+        # Save state of the Event object 
+        self._event_is_set = self.confirm_received.is_set()
+        
+        # Exclude Event object from dict and return new state
+        new_dct = dict(self.__dict__)
+        del new_dct["confirm_received"]
+        return new_dct
+
+    def __setstate__(self, state):
+        self.__dict__ = state
+        self._set_event()
 
     # These should not be overrode
     @staticmethod
@@ -166,6 +189,6 @@ class TLObject:
     def __bytes__(self):
         return b''
 
-    @staticmethod
-    def from_reader(reader):
+    @classmethod
+    def from_reader(cls, reader):
         return TLObject()

telethon/update_state.py

@@ -110,6 +110,10 @@ class UpdateState:
                 # We don't want to crash a worker thread due to any reason
                 __log__.exception('Unhandled exception on worker %d', wid)
 
+    def get_update_state(self, entity_id):
+        """Gets the updates.State corresponding to the given entity or 0."""
+        return self._state
+
     def process(self, update):
         """Processes an update object. This method is normally called by
            the library itself.
@@ -129,12 +133,9 @@ class UpdateState:
             # After running the script for over an hour and receiving over
             # 1000 updates, the only duplicates received were users going
             # online or offline. We can trust the server until new reports.
-            #
-            # TODO Note somewhere that all updates are modified to include
-            # .entities, which is a dictionary you can access but may be empty.
             # This should only be used as read-only.
             if isinstance(update, tl.UpdateShort):
-                update.update.entities = {}
+                update.update._entities = {}
                 self._updates.put(update.update)
             # Expand "Updates" into "Update", and pass these to callbacks.
             # Since .users and .chats have already been processed, we
@@ -143,9 +144,9 @@ class UpdateState:
                 entities = {utils.get_peer_id(x): x for x in
                             itertools.chain(update.users, update.chats)}
                 for u in update.updates:
-                    u.entities = entities
+                    u._entities = entities
                     self._updates.put(u)
             # TODO Handle "tl.UpdatesTooLong"
             else:
-                update.entities = {}
+                update._entities = {}
                 self._updates.put(update)

telethon/utils.py

@@ -2,14 +2,17 @@
 Utilities for working with the Telegram API itself (such as handy methods
 to convert between an entity like an User, Chat, etc. into its Input version)
 """
+import itertools
 import math
 import mimetypes
 import os
 import re
+import struct
 import types
 from collections import UserList
 from mimetypes import guess_extension
 
+from .tl import TLObject
 from .tl.types import (
     Channel, ChannelForbidden, Chat, ChatEmpty, ChatForbidden, ChatFull,
     ChatPhoto, InputPeerChannel, InputPeerChat, InputPeerUser, InputPeerEmpty,
@@ -25,7 +28,8 @@ from .tl.types import (
     InputPhotoEmpty, FileLocation, ChatPhotoEmpty, UserProfilePhotoEmpty,
     FileLocationUnavailable, InputMediaUploadedDocument, ChannelFull,
     InputMediaUploadedPhoto, DocumentAttributeFilename, photos,
-    TopPeer, InputNotifyPeer
+    TopPeer, InputNotifyPeer, InputMessageID, InputFileLocation,
+    InputDocumentFileLocation, PhotoSizeEmpty, InputDialogPeer
 )
 from .tl.types.contacts import ResolvedPeer
 
@@ -33,7 +37,35 @@ USERNAME_RE = re.compile(
     r'@|(?:https?://)?(?:www\.)?(?:telegram\.(?:me|dog)|t\.me)/(joinchat/)?'
 )
 
-VALID_USERNAME_RE = re.compile(r'^[a-zA-Z][\w\d]{3,30}[a-zA-Z\d]$')
+# The only shorter-than-five-characters usernames are those used for some
+# special, very well known bots. This list may be incomplete though:
+#    "[...] @gif, @vid, @pic, @bing, @wiki, @imdb and @bold [...]"
+#
+# See https://telegram.org/blog/inline-bots#how-does-it-work
+VALID_USERNAME_RE = re.compile(
+    r'^([a-z][\w\d]{3,30}[a-z\d]'
+    r'|gif|vid|pic|bing|wiki|imdb|bold|vote|like|coub|ya)$',
+    re.IGNORECASE
+)
+
+
+class Default:
+    """
+    Sentinel value to indicate that the default value should be used.
+    Currently used for the ``parse_mode``, where a ``None`` mode should
+    be considered different from using the default.
+    """
+
+
+def chunks(iterable, size=100):
+    """
+    Turns the given iterable into chunks of the specified size,
+    which is 100 by default since that's what Telegram uses the most.
+    """
+    it = iter(iterable)
+    size -= 1
+    for head in it:
+        yield itertools.chain([head], itertools.islice(it, size))
 
 
 def get_display_name(entity):
@@ -91,7 +123,13 @@ def get_input_peer(entity, allow_self=True):
         if entity.SUBCLASS_OF_ID == 0xc91c90b6:  # crc32(b'InputPeer')
             return entity
     except AttributeError:
-        _raise_cast_fail(entity, 'InputPeer')
+        # e.g. custom.Dialog (can't cyclic import).
+        if allow_self and hasattr(entity, 'input_entity'):
+            return entity.input_entity
+        elif hasattr(entity, 'entity'):
+            return get_input_peer(entity.entity)
+        else:
+            _raise_cast_fail(entity, 'InputPeer')
 
     if isinstance(entity, User):
         if entity.is_self and allow_self:
@@ -105,7 +143,6 @@ def get_input_peer(entity, allow_self=True):
     if isinstance(entity, (Channel, ChannelForbidden)):
         return InputPeerChannel(entity.id, entity.access_hash or 0)
 
-    # Less common cases
     if isinstance(entity, InputUser):
         return InputPeerUser(entity.user_id, entity.access_hash)
 
@@ -176,6 +213,24 @@ def get_input_user(entity):
     _raise_cast_fail(entity, 'InputUser')
 
 
+def get_input_dialog(dialog):
+    """Similar to :meth:`get_input_peer`, but for dialogs"""
+    try:
+        if dialog.SUBCLASS_OF_ID == 0xa21c9795:  # crc32(b'InputDialogPeer')
+            return dialog
+        if dialog.SUBCLASS_OF_ID == 0xc91c90b6:  # crc32(b'InputPeer')
+            return InputDialogPeer(dialog)
+    except AttributeError:
+        _raise_cast_fail(dialog, 'InputDialogPeer')
+
+    try:
+        return InputDialogPeer(get_input_peer(dialog))
+    except TypeError:
+        pass
+
+    _raise_cast_fail(dialog, 'InputDialogPeer')
+
+
 def get_input_document(document):
     """Similar to :meth:`get_input_peer`, but for documents"""
     try:
@@ -250,8 +305,12 @@ def get_input_media(media, is_photo=False):
     it will be treated as an :tl:`InputMediaUploadedPhoto`.
     """
     try:
-        if media.SUBCLASS_OF_ID == 0xfaf846f4:  # crc32(b'InputMedia'):
+        if media.SUBCLASS_OF_ID == 0xfaf846f4:  # crc32(b'InputMedia')
             return media
+        elif media.SUBCLASS_OF_ID == 0x846363e0:  # crc32(b'InputPhoto')
+            return InputMediaPhoto(media)
+        elif media.SUBCLASS_OF_ID == 0xf33fdb68:  # crc32(b'InputDocument')
+            return InputMediaDocument(media)
     except AttributeError:
         _raise_cast_fail(media, 'InputMedia')
 
@@ -261,12 +320,22 @@ def get_input_media(media, is_photo=False):
             ttl_seconds=media.ttl_seconds
         )
 
+    if isinstance(media, (Photo, photos.Photo, PhotoEmpty)):
+        return InputMediaPhoto(
+            id=get_input_photo(media)
+        )
+
     if isinstance(media, MessageMediaDocument):
         return InputMediaDocument(
             id=get_input_document(media.document),
             ttl_seconds=media.ttl_seconds
         )
 
+    if isinstance(media, (Document, DocumentEmpty)):
+        return InputMediaDocument(
+            id=get_input_document(media)
+        )
+
     if isinstance(media, FileLocation):
         if is_photo:
             return InputMediaUploadedPhoto(file=media)
@@ -318,6 +387,54 @@ def get_input_media(media, is_photo=False):
     _raise_cast_fail(media, 'InputMedia')
 
 
+def get_input_message(message):
+    """Similar to :meth:`get_input_peer`, but for input messages."""
+    try:
+        if isinstance(message, int):  # This case is really common too
+            return InputMessageID(message)
+        elif message.SUBCLASS_OF_ID == 0x54b6bcc5:  # crc32(b'InputMessage'):
+            return message
+        elif message.SUBCLASS_OF_ID == 0x790009e3:  # crc32(b'Message'):
+            return InputMessageID(message.id)
+    except AttributeError:
+        pass
+
+    _raise_cast_fail(message, 'InputMedia')
+
+
+def get_input_location(location):
+    """Similar to :meth:`get_input_peer`, but for input messages."""
+    try:
+        if location.SUBCLASS_OF_ID == 0x1523d462:
+            return location  # crc32(b'InputFileLocation'):
+    except AttributeError:
+        _raise_cast_fail(location, 'InputFileLocation')
+
+    if isinstance(location, Message):
+        location = location.media
+
+    if isinstance(location, MessageMediaDocument):
+        location = location.document
+    elif isinstance(location, MessageMediaPhoto):
+        location = location.photo
+
+    if isinstance(location, Document):
+        return InputDocumentFileLocation(
+            location.id, location.access_hash, location.version)
+    elif isinstance(location, Photo):
+        try:
+            location = next(x for x in reversed(location.sizes)
+                            if not isinstance(x, PhotoSizeEmpty)).location
+        except StopIteration:
+            pass
+
+    if isinstance(location, (FileLocation, FileLocationUnavailable)):
+        return InputFileLocation(
+            location.volume_id, location.local_id, location.secret)
+
+    _raise_cast_fail(location, 'InputFileLocation')
+
+
 def is_image(file):
     """
     Returns ``True`` if the file extension looks like an image file to Telegram.
@@ -397,6 +514,38 @@ def _fix_peer_id(peer_id):
     return int(peer_id)
 
 
+def add_surrogate(text):
+    return ''.join(
+        # SMP -> Surrogate Pairs (Telegram offsets are calculated with these).
+        # See https://en.wikipedia.org/wiki/Plane_(Unicode)#Overview for more.
+        ''.join(chr(y) for y in struct.unpack('<HH', x.encode('utf-16le')))
+        if (0x10000 <= ord(x) <= 0x10FFFF) else x for x in text
+    )
+
+
+def del_surrogate(text):
+    return text.encode('utf-16', 'surrogatepass').decode('utf-16')
+
+
+def get_inner_text(text, entities):
+    """
+    Gets the inner text that's surrounded by the given entities.
+    For instance: text = 'hey!', entity = MessageEntityBold(2, 2) -> 'y!'.
+
+    :param text:     the original text.
+    :param entities: the entity or entities that must be matched.
+    :return: a single result or a list of the text surrounded by the entities.
+    """
+    text = add_surrogate(text)
+    result = []
+    for e in entities:
+        start = e.offset
+        end = e.offset + e.length
+        result.append(del_surrogate(text[start:end]))
+
+    return result
+
+
 def get_peer_id(peer):
     """
     Finds the ID of the given peer, and converts it to the "bot api" format
@@ -457,8 +606,13 @@ def resolve_id(marked_id):
     if marked_id >= 0:
         return marked_id, PeerUser
 
-    if str(marked_id).startswith('-100'):
-        return int(str(marked_id)[4:]), PeerChannel
+    # There have been report of chat IDs being 10000xyz, which means their
+    # marked version is -10000xyz, which in turn looks like a channel but
+    # it becomes 00xyz (= xyz). Hence, we must assert that there are only
+    # two zeroes.
+    m = re.match(r'-100([^0]\d*)', str(marked_id))
+    if m:
+        return int(m.group(1)), PeerChannel
 
     return -marked_id, PeerChat
 

telethon/version.py

@@ -1,3 +1,3 @@
 # Versions should comply with PEP440.
 # This line is parsed in setup.py:
-__version__ = '0.18.1'
+__version__ = '0.19.1'

telethon_examples/interactive_telegram_client.py

@@ -3,7 +3,8 @@ from getpass import getpass
 
 from telethon.utils import get_display_name
 
-from telethon import ConnectionMode, TelegramClient
+from telethon import TelegramClient, events
+from telethon.network import ConnectionTcpAbridged
 from telethon.errors import SessionPasswordNeededError
 from telethon.tl.types import (
     PeerChat, UpdateShortChatMessage, UpdateShortMessage
@@ -70,11 +71,11 @@ class InteractiveTelegramClient(TelegramClient):
             # These parameters should be passed always, session name and API
             session_user_id, api_id, api_hash,
 
-            # You can optionally change the connection mode by using this enum.
-            # This changes how much data will be sent over the network with
-            # every request, and how it will be formatted. Default is
-            # ConnectionMode.TCP_FULL, and smallest is TCP_TCP_ABRIDGED.
-            connection_mode=ConnectionMode.TCP_ABRIDGED,
+            # You can optionally change the connection mode by passing a
+            # type or an instance of it. This changes how the sent packets
+            # look (low-level concept you normally shouldn't worry about).
+            # Default is ConnectionTcpFull, smallest is ConnectionTcpAbridged.
+            connection=ConnectionTcpAbridged,
 
             # If you're using a proxy, set it here.
             proxy=proxy,
@@ -126,10 +127,11 @@ class InteractiveTelegramClient(TelegramClient):
     def run(self):
         """Main loop of the TelegramClient, will wait for user action"""
 
-        # Once everything is ready, we can add an update handler. Every
-        # update object will be passed to the self.update_handler method,
-        # where we can process it as we need.
-        self.add_update_handler(self.update_handler)
+        # Once everything is ready, we can add an event handler.
+        #
+        # Events are an abstraction over Telegram's "Updates" and
+        # are much easier to use.
+        self.add_event_handler(self.message_handler, events.NewMessage)
 
         # Enter a while loop to chat as long as the user wants
         while True:
@@ -207,7 +209,7 @@ class InteractiveTelegramClient(TelegramClient):
                 # History
                 elif msg == '!h':
                     # First retrieve the messages and some information
-                    messages = self.get_message_history(entity, limit=10)
+                    messages = self.get_messages(entity, limit=10)
 
                     # Iterate over all (in reverse order so the latest appear
                     # the last in the console) and print them with format:
@@ -216,7 +218,7 @@ class InteractiveTelegramClient(TelegramClient):
                         # Note that the .sender attribute is only there for
                         # convenience, the API returns it differently. But
                         # this shouldn't concern us. See the documentation
-                        # for .get_message_history() for more information.
+                        # for .iter_messages() for more information.
                         name = get_display_name(msg.sender)
 
                         # Format the message content
@@ -334,31 +336,29 @@ class InteractiveTelegramClient(TelegramClient):
             bytes_to_string(total_bytes), downloaded_bytes / total_bytes)
         )
 
-    def update_handler(self, update):
-        """Callback method for received Updates"""
+    def message_handler(self, event):
+        """Callback method for received events.NewMessage"""
 
-        # We have full control over what we want to do with the updates.
-        # In our case we only want to react to chat messages, so we use
-        # isinstance() to behave accordingly on these cases.
-        if isinstance(update, UpdateShortMessage):
-            who = self.get_entity(update.user_id)
-            if update.out:
+        # Note that accessing ``.sender`` and ``.chat`` may be slow since
+        # these are not cached and must be queried always! However it lets
+        # us access the chat title and user name.
+        if event.is_group:
+            if event.out:
+                sprint('>> sent "{}" to chat {}'.format(
+                    event.text, get_display_name(event.chat)
+                ))
+            else:
+                sprint('<< {} @ {} sent "{}"'.format(
+                    get_display_name(event.sender),
+                    get_display_name(event.chat),
+                    event.text
+                ))
+        else:
+            if event.out:
                 sprint('>> "{}" to user {}'.format(
-                    update.message, get_display_name(who)
+                    event.text, get_display_name(event.chat)
                 ))
             else:
                 sprint('<< {} sent "{}"'.format(
-                    get_display_name(who), update.message
-                ))
-
-        elif isinstance(update, UpdateShortChatMessage):
-            which = self.get_entity(PeerChat(update.chat_id))
-            if update.out:
-                sprint('>> sent "{}" to chat {}'.format(
-                    update.message, get_display_name(which)
-                ))
-            else:
-                who = self.get_entity(update.from_id)
-                sprint('<< {} @ {} sent "{}"'.format(
-                    get_display_name(which), get_display_name(who), update.message
+                    get_display_name(event.chat), event.text
                 ))

telethon_examples/print_updates.py

@@ -1,5 +1,9 @@
 #!/usr/bin/env python3
 # A simple script to print all updates received
+#
+# NOTE: To run this script you MUST have 'TG_API_ID' and 'TG_API_HASH' in
+#       your environment variables. This is a good way to use these private
+#       values. See https://superuser.com/q/284342.
 
 from os import environ
 
@@ -23,7 +27,7 @@ def main():
     else:
         client.start()
 
-    client.add_update_handler(update_handler)
+    client.add_event_handler(update_handler)
     print('(Press Ctrl+C to stop this)')
     client.idle()
 

telethon_examples/replier.py

@@ -2,9 +2,9 @@
 """
 A example script to automatically send messages based on certain triggers.
 
-The script makes uses of environment variables to determine the API ID,
-hash, phone and such to be used. You may want to add these to your .bashrc
-file, including TG_API_ID, TG_API_HASH, TG_PHONE and optionally TG_SESSION.
+NOTE: To run this script you MUST have 'TG_API_ID' and 'TG_API_HASH' in
+      your environment variables. This is a good way to use these private
+      values. See https://superuser.com/q/284342.
 
 This script assumes that you have certain files on the working directory,
 such as "xfiles.m4a" or "anytime.png" for some of the automated replies.

telethon_generator/data/error_descriptions

@@ -64,3 +64,4 @@ USER_ALREADY_PARTICIPANT=The authenticated user is already a participant of the
 USER_DEACTIVATED=The user has been deleted/deactivated
 FLOOD_WAIT_X=A wait of {} seconds is required
 FLOOD_TEST_PHONE_WAIT_X=A wait of {} seconds is required in the test servers
+CHAT_NOT_MODIFIED=The chat or channel wasn't modified (title, invites, username, admins, etc. are the same).ç

telethon_generator/data/html/core.html

@@ -4,7 +4,36 @@
     <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
     <title>Telethon API</title>
     <meta name="viewport" content="width=device-width, initial-scale=1.0">
-    <link href="css/docs.css" rel="stylesheet">
+    <link id="style" href="css/docs.light.css" rel="stylesheet">
+    <script>
+        (function () {
+            var style = document.getElementById('style');
+
+            // setTheme(<link />, 'light' / 'dark')
+            function setTheme(theme) {
+                document.cookie = 'css=' + theme + '; path=/';
+                return style.href = 'css/docs.' + theme + '.css';
+            }
+
+            // setThemeOnClick(<link />, 'light' / 'dark', <a />)
+            function setThemeOnClick(theme, button) {
+                return button.addEventListener('click', function (e) {
+                    setTheme(theme);
+                    e.preventDefault();
+                    return false;
+                });
+            }
+
+            setTheme(document.cookie
+                .split(';')[0]
+                .split('=')[1] || 'light');
+
+            document.addEventListener('DOMContentLoaded', function () {
+                setThemeOnClick('light', document.getElementById('themeLight'));
+                setThemeOnClick('dark', document.getElementById('themeDark'));
+            });
+        })();
+    </script>
     <link href="https://fonts.googleapis.com/css?family=Nunito|Source+Code+Pro" rel="stylesheet">
     <style>
         body {
@@ -21,7 +50,10 @@
     on what the methods, constructors and types mean. Nevertheless, this
     page aims to provide easy access to all the available methods, their
     definition and parameters.</p>
-
+    <p id="themeSelect">
+        <a href="#" id="themeLight">light</a> /
+        <a href="#" id="themeDark">dark</a> theme.
+    </p>
     <p>Please note that when you see this:</p>
     <pre>---functions---
 users.getUsers#0d91a548 id:Vector&lt;InputUser&gt; = Vector&lt;User&gt;</pre>

telethon_generator/data/html/css/docs.dark.css

@@ -0,0 +1,185 @@
+body {
+    font-family: 'Nunito', sans-serif;
+    color: #bbb;
+    background-color:#000;
+    font-size: 16px;
+}
+
+a {
+    color: #42aaed;
+    text-decoration: none;
+}
+
+pre {
+    font-family: 'Source Code Pro', monospace;
+    padding: 8px;
+    color: #567;
+    background: #080a0c;
+    border-radius: 0;
+    overflow-x: auto;
+}
+
+a:hover {
+    color: #64bbdd;
+    text-decoration: underline;
+}
+
+table {
+    width: 100%;
+    max-width: 100%;
+}
+
+table td {
+    border-top: 1px solid #111;
+    padding: 8px;
+}
+
+.horizontal {
+    margin-bottom: 16px;
+    list-style: none;
+    background: #080a0c;
+    border-radius: 4px;
+    padding: 8px 16px;
+}
+
+.horizontal li {
+    display: inline-block;
+    margin: 0 8px 0 0;
+}
+
+.horizontal img {
+    display: inline-block;
+    margin: 0 8px -2px 0;
+}
+
+h1, summary.title {
+    font-size: 24px;
+}
+
+h3 {
+    font-size: 20px;
+}
+
+#main_div {
+  padding: 20px 0;
+  max-width: 800px;
+  margin: 0 auto;
+}
+
+pre::-webkit-scrollbar {
+    visibility: visible;
+    display: block;
+    height: 12px;
+}
+
+pre::-webkit-scrollbar-track:horizontal {
+    background: #222;
+    border-radius: 0;
+    height: 12px;
+}
+
+pre::-webkit-scrollbar-thumb:horizontal {
+    background: #444;
+    border-radius: 0;
+    height: 12px;
+}
+
+:target {
+    border: 2px solid #149;
+    background: #246;
+    padding: 4px;
+}
+
+/* 'sh' stands for Syntax Highlight */
+span.sh1 {
+    color: #f93;
+}
+
+span.tooltip {
+    border-bottom: 1px dashed #ddd;
+}
+
+#searchBox {
+    width: 100%;
+    border: none;
+    height: 20px;
+    padding: 8px;
+    font-size: 16px;
+    border-radius: 2px;
+    border: 2px solid #222;
+    background: #000;
+    color: #eee;
+}
+
+#searchBox:placeholder-shown {
+    color: #bbb;
+    font-style: italic;
+}
+
+button {
+    border-radius: 2px;
+    font-size: 16px;
+    padding: 8px;
+    color: #bbb;
+    background-color: #111;
+    border: 2px solid #146;
+    transition-duration: 300ms;
+}
+
+button:hover {
+    background-color: #146;
+    color: #fff;
+}
+
+/* https://www.w3schools.com/css/css_navbar.asp */
+ul.together {
+    list-style-type: none;
+    margin: 0;
+    padding: 0;
+    overflow: hidden;
+}
+
+ul.together li {
+    float: left;
+}
+
+ul.together li a {
+    display: block;
+    border-radius: 8px;
+    background: #111;
+    padding: 4px 8px;
+    margin: 8px;
+}
+
+/* https://stackoverflow.com/a/30810322 */
+.invisible {
+    left: 0;
+    top: -99px;
+    padding: 0;
+    width: 2em;
+    height: 2em;
+    border: none;
+    outline: none;
+    position: fixed;
+    box-shadow: none;
+    color: transparent;
+    background: transparent;
+}
+
+@media (max-width: 640px) {
+    h1, summary.title {
+        font-size: 18px;
+    }
+    h3 {
+        font-size: 16px;
+    }
+
+    #dev_page_content_wrap {
+        padding-top: 12px;
+    }
+
+    #dev_page_title {
+        margin-top: 10px;
+        margin-bottom: 20px;
+    }
+}

telethon_generator/data/html/css/docs.h4x0r.css

@@ -0,0 +1,229 @@
+/* Begin of https://cdn.jsdelivr.net/npm/hack-font@3/build/web/hack.css
+ *
+ *  Hack typeface https://github.com/source-foundry/Hack
+ *  License: https://github.com/source-foundry/Hack/blob/master/LICENSE.md
+ */
+@font-face {
+  font-family: 'Hack';
+  src: url('fonts/hack-regular.woff2?sha=3114f1256') format('woff2'), url('fonts/hack-regular.woff?sha=3114f1256') format('woff');
+  font-weight: 400;
+  font-style: normal;
+}
+
+@font-face {
+  font-family: 'Hack';
+  src: url('fonts/hack-bold.woff2?sha=3114f1256') format('woff2'), url('fonts/hack-bold.woff?sha=3114f1256') format('woff');
+  font-weight: 700;
+  font-style: normal;
+}
+
+@font-face {
+  font-family: 'Hack';
+  src: url('fonts/hack-italic.woff2?sha=3114f1256') format('woff2'), url('fonts/hack-italic.woff?sha=3114f1256') format('woff');
+  font-weight: 400;
+  font-style: italic;
+}
+
+@font-face {
+  font-family: 'Hack';
+  src: url('fonts/hack-bolditalic.woff2?sha=3114f1256') format('woff2'), url('fonts/hack-bolditalic.woff?sha=3114f1256') format('woff');
+  font-weight: 700;
+  font-style: italic;
+}
+
+/* End of https://cdn.jsdelivr.net/npm/hack-font@3/build/web/hack.css */
+
+body {
+    font-family: 'Hack', monospace;
+    color: #0a0;
+    background-color: #000;
+    font-size: 16px;
+}
+
+::-moz-selection {
+    color: #000;
+    background: #0a0;
+}
+
+::selection {
+    color: #000;
+    background: #0a0;
+}
+
+a {
+    color: #0a0;
+}
+
+pre {
+    padding: 8px;
+    color: #0c0;
+    background: #010;
+    border-radius: 0;
+    overflow-x: auto;
+}
+
+a:hover {
+    color: #0f0;
+    text-decoration: underline;
+}
+
+table {
+    width: 100%;
+    max-width: 100%;
+}
+
+table td {
+    border-top: 1px solid #111;
+    padding: 8px;
+}
+
+.horizontal {
+    margin-bottom: 16px;
+    list-style: none;
+    background: #010;
+    border-radius: 4px;
+    padding: 8px 16px;
+}
+
+.horizontal li {
+    display: inline-block;
+    margin: 0 8px 0 0;
+}
+
+.horizontal img {
+    opacity: 0;
+    display: inline-block;
+    margin: 0 8px -2px 0;
+}
+
+h1, summary.title {
+    font-size: 24px;
+}
+
+h3 {
+    font-size: 20px;
+}
+
+#main_div {
+  padding: 20px 0;
+  max-width: 800px;
+  margin: 0 auto;
+}
+
+pre::-webkit-scrollbar {
+    visibility: visible;
+    display: block;
+    height: 12px;
+}
+
+pre::-webkit-scrollbar-track:horizontal {
+    background: #222;
+    border-radius: 0;
+    height: 12px;
+}
+
+pre::-webkit-scrollbar-thumb:horizontal {
+    background: #444;
+    border-radius: 0;
+    height: 12px;
+}
+
+:target {
+    border: 2px solid #0f0;
+    background: #010;
+    padding: 4px;
+}
+
+/* 'sh' stands for Syntax Highlight */
+span.sh1 {
+    color: #0f0;
+}
+
+span.tooltip {
+    border-bottom: 1px dashed #ddd;
+}
+
+#searchBox {
+    width: 100%;
+    border: none;
+    height: 20px;
+    padding: 8px;
+    font-size: 16px;
+    border-radius: 2px;
+    border: 2px solid #222;
+    background: #000;
+    color: #0e0;
+    font-family: 'Hack', monospace;
+}
+
+#searchBox:placeholder-shown {
+    color: #0b0;
+    font-style: italic;
+}
+
+button {
+    font-size: 16px;
+    padding: 8px;
+    color: #0f0;
+    background-color: #071007;
+    border: 2px solid #131;
+    transition-duration: 300ms;
+    font-family: 'Hack', monospace;
+}
+
+button:hover {
+    background-color: #131;
+}
+
+/* https://www.w3schools.com/css/css_navbar.asp */
+ul.together {
+    list-style-type: none;
+    margin: 0;
+    padding: 0;
+    overflow: hidden;
+}
+
+ul.together li {
+    float: left;
+}
+
+ul.together li a {
+    display: block;
+    border-radius: 8px;
+    background: #121;
+    padding: 4px 8px;
+    margin: 8px;
+}
+
+/* https://stackoverflow.com/a/30810322 */
+.invisible {
+    left: 0;
+    top: -99px;
+    padding: 0;
+    width: 2em;
+    height: 2em;
+    border: none;
+    outline: none;
+    position: fixed;
+    box-shadow: none;
+    color: transparent;
+    background: transparent;
+}
+
+@media (max-width: 640px) {
+    h1, summary.title {
+        font-size: 18px;
+    }
+    h3 {
+        font-size: 16px;
+    }
+
+    #dev_page_content_wrap {
+        padding-top: 12px;
+    }
+
+    #dev_page_title {
+        margin-top: 10px;
+        margin-bottom: 20px;
+    }
+}

telethon_generator/data/html/css/docs.light.css

@@ -95,19 +95,6 @@ span.sh1 {
     color: #f70;
 }
 
-span.sh2 {
-    color: #0c7;
-}
-
-span.sh3 {
-    color: #aaa;
-    font-style: italic;
-}
-
-span.sh4 {
-    color: #06c;
-}
-
 span.tooltip {
     border-bottom: 1px dashed #444;
 }

telethon_generator/data/invalid_bot_methods.json

@@ -0,0 +1 @@
+["account.changePhone", "account.checkUsername", "account.confirmPhone", "account.deleteAccount", "account.getAccountTTL", "account.getAuthorizations", "account.getNotifySettings", "account.getPassword", "account.getPasswordSettings", "account.getPrivacy", "account.getTmpPassword", "account.getWallPapers", "account.registerDevice", "account.reportPeer", "account.resetAuthorization", "account.resetNotifySettings", "account.sendChangePhoneCode", "account.sendConfirmPhoneCode", "account.setAccountTTL", "account.setPrivacy", "account.unregisterDevice", "account.updateDeviceLocked", "account.updateNotifySettings", "account.updatePasswordSettings", "account.updateProfile", "account.updateStatus", "account.updateUsername", "auth.cancelCode", "auth.checkPassword", "auth.checkPhone", "auth.recoverPassword", "auth.requestPasswordRecovery", "auth.resendCode", "auth.resetAuthorizations", "auth.sendCode", "auth.sendInvites", "auth.signIn", "auth.signUp", "channels.checkUsername", "channels.createChannel", "channels.deleteChannel", "channels.deleteUserHistory", "channels.exportMessageLink", "channels.getAdminedPublicChannels", "channels.getAdminLog", "channels.inviteToChannel", "channels.joinChannel", "channels.readHistory", "channels.readMessageContents", "channels.reportSpam", "channels.toggleInvites", "channels.toggleSignatures", "channels.updateUsername", "contacts.block", "contacts.deleteContact", "contacts.deleteContacts", "contacts.exportCard", "contacts.getBlocked", "contacts.getContacts", "contacts.getStatuses", "contacts.getTopPeers", "contacts.importCard", "contacts.importContacts", "contacts.resetTopPeerRating", "contacts.search", "contacts.unblock", "help.getAppChangelog", "help.getAppUpdate", "help.getInviteText", "help.getNearestDc", "help.getSupport", "help.getTermsOfService", "help.saveAppLog", "langpack.getDifference", "langpack.getLangPack", "langpack.getLanguages", "langpack.getStrings", "messages.acceptEncryption", "messages.addChatUser", "messages.checkChatInvite", "messages.clearRecentStickers", "messages.createChat", "messages.deleteHistory", "messages.discardEncryption", "messages.editChatAdmin", "messages.exportChatInvite", "messages.getAllChats", "messages.getAllDrafts", "messages.getAllStickers", "messages.getArchivedStickers", "messages.getAttachedStickers", "messages.getBotCallbackAnswer", "messages.getCommonChats", "messages.getDhConfig", "messages.getDialogs", "messages.getFeaturedStickers", "messages.getHistory", "messages.getInlineBotResults", "messages.getMaskStickers", "messages.getMessageEditData", "messages.getMessagesViews", "messages.getPeerDialogs", "messages.getPeerSettings", "messages.getPinnedDialogs", "messages.getRecentStickers", "messages.getSavedGifs", "messages.getUnreadMentions", "messages.getWebPage", "messages.getWebPagePreview", "messages.hideReportSpam", "messages.importChatInvite", "messages.installStickerSet", "messages.migrateChat", "messages.readFeaturedStickers", "messages.readHistory", "messages.readMessageContents", "messages.receivedMessages", "messages.reorderPinnedDialogs", "messages.reorderStickerSets", "messages.reportEncryptedSpam", "messages.reportSpam", "messages.requestEncryption", "messages.saveDraft", "messages.saveGif", "messages.saveRecentSticker", "messages.search", "messages.searchGifs", "messages.searchGlobal", "messages.sendInlineBotResult", "messages.sendScreenshotNotification", "messages.setEncryptedTyping", "messages.startBot", "messages.toggleChatAdmins", "messages.toggleDialogPin", "messages.uninstallStickerSet", "payments.clearSavedInfo", "payments.getPaymentForm", "payments.getPaymentReceipt", "payments.getSavedInfo", "payments.sendPaymentForm", "payments.validateRequestedInfo", "phone.acceptCall", "phone.confirmCall", "phone.discardCall", "phone.getCallConfig", "phone.receivedCall", "phone.requestCall", "phone.saveCallDebug", "phone.setCallRating", "photos.deletePhotos", "photos.updateProfilePhoto", "photos.uploadProfilePhoto", "upload.getCdnFile", "upload.getWebFile"]

telethon_generator/data/mtproto_api.tl

@@ -0,0 +1,90 @@
+//int ? = Int;
+//long ? = Long;
+//double ? = Double;
+//string ? = String;
+
+dummyHttpWait = HttpWait;
+
+//vector {t:Type} # [ t ] = Vector t;
+
+//int128 4*[ int ] = Int128;
+//int256 8*[ int ] = Int256;
+
+resPQ#05162463 nonce:int128 server_nonce:int128 pq:bytes server_public_key_fingerprints:Vector<long> = ResPQ;
+
+p_q_inner_data#83c95aec pq:bytes p:bytes q:bytes nonce:int128 server_nonce:int128 new_nonce:int256 = P_Q_inner_data;
+p_q_inner_data_temp#3c6a84d4 pq:bytes p:bytes q:bytes nonce:int128 server_nonce:int128 new_nonce:int256 expires_in:int = P_Q_inner_data;
+
+server_DH_params_fail#79cb045d nonce:int128 server_nonce:int128 new_nonce_hash:int128 = Server_DH_Params;
+server_DH_params_ok#d0e8075c nonce:int128 server_nonce:int128 encrypted_answer:bytes = Server_DH_Params;
+
+server_DH_inner_data#b5890dba nonce:int128 server_nonce:int128 g:int dh_prime:bytes g_a:bytes server_time:int = Server_DH_inner_data;
+
+client_DH_inner_data#6643b654 nonce:int128 server_nonce:int128 retry_id:long g_b:bytes = Client_DH_Inner_Data;
+
+dh_gen_ok#3bcbf734 nonce:int128 server_nonce:int128 new_nonce_hash1:int128 = Set_client_DH_params_answer;
+dh_gen_retry#46dc1fb9 nonce:int128 server_nonce:int128 new_nonce_hash2:int128 = Set_client_DH_params_answer;
+dh_gen_fail#a69dae02 nonce:int128 server_nonce:int128 new_nonce_hash3:int128 = Set_client_DH_params_answer;
+
+bind_auth_key_inner#75a3f765 nonce:long temp_auth_key_id:long perm_auth_key_id:long temp_session_id:long expires_at:int = BindAuthKeyInner;
+
+//rpc_result#f35c6d01 req_msg_id:long result:bytes = RpcResult;
+rpc_error#2144ca19 error_code:int error_message:string = RpcError;
+
+rpc_answer_unknown#5e2ad36e = RpcDropAnswer;
+rpc_answer_dropped_running#cd78e586 = RpcDropAnswer;
+rpc_answer_dropped#a43ad8b7 msg_id:long seq_no:int bytes:int = RpcDropAnswer;
+
+future_salt#0949d9dc valid_since:int valid_until:int salt:long = FutureSalt;
+future_salts#ae500895 req_msg_id:long now:int salts:vector<future_salt> = FutureSalts;
+
+pong#347773c5 msg_id:long ping_id:long = Pong;
+
+destroy_session_ok#e22045fc session_id:long = DestroySessionRes;
+destroy_session_none#62d350c9 session_id:long = DestroySessionRes;
+
+new_session_created#9ec20908 first_msg_id:long unique_id:long server_salt:long = NewSession;
+
+//msg_container#73f1f8dc messages:vector<%Message> = MessageContainer;
+//message msg_id:long seqno:int bytes:int body:bytes = Message;
+//msg_copy#e06046b2 orig_message:Message = MessageCopy;
+
+gzip_packed#3072cfa1 packed_data:bytes = Object;
+
+msgs_ack#62d6b459 msg_ids:Vector<long> = MsgsAck;
+
+bad_msg_notification#a7eff811 bad_msg_id:long bad_msg_seqno:int error_code:int = BadMsgNotification;
+bad_server_salt#edab447b bad_msg_id:long bad_msg_seqno:int error_code:int new_server_salt:long = BadMsgNotification;
+
+msg_resend_req#7d861a08 msg_ids:Vector<long> = MsgResendReq;
+msgs_state_req#da69fb52 msg_ids:Vector<long> = MsgsStateReq;
+msgs_state_info#04deb57d req_msg_id:long info:string = MsgsStateInfo;
+msgs_all_info#8cc0d131 msg_ids:Vector<long> info:string = MsgsAllInfo;
+msg_detailed_info#276d3ec6 msg_id:long answer_msg_id:long bytes:int status:int = MsgDetailedInfo;
+msg_new_detailed_info#809db6df answer_msg_id:long bytes:int status:int = MsgDetailedInfo;
+
+rsa_public_key n:string e:bytes = RSAPublicKey;
+
+---functions---
+
+req_pq_multi#be7e8ef1 nonce:int128 = ResPQ;
+
+req_DH_params#d712e4be nonce:int128 server_nonce:int128 p:bytes q:bytes public_key_fingerprint:long encrypted_data:string = Server_DH_Params;
+
+set_client_DH_params#f5045f1f nonce:int128 server_nonce:int128 encrypted_data:bytes = Set_client_DH_params_answer;
+
+rpc_drop_answer#58e4a740 req_msg_id:long = RpcDropAnswer;
+get_future_salts#b921bd04 num:int = FutureSalts;
+ping#7abe77ec ping_id:long = Pong;
+ping_delay_disconnect#f3427b8c ping_id:long disconnect_delay:int = Pong;
+destroy_session#e7512126 session_id:long = DestroySessionRes;
+
+http_wait#9299359f max_delay:int wait_after:int max_wait:int = HttpWait;
+
+//test.useGzipPacked = GzipPacked;
+//test.useServerDhInnerData = Server_DH_inner_data;
+//test.useNewSessionCreated = NewSession;
+//test.useMsgsAck = MsgsAck;
+//test.useBadMsgNotification = BadMsgNotification;
+
+//test.useOther key:rsa_public_key p_q_data:P_Q_inner_data dh_data:client_DH_inner_data = RpcError;

telethon_generator/data/telegram_api.tl

@@ -1,147 +1,27 @@
-// Core types (no need to gen)
+//int ? = Int;
+//long ? = Long;
+//double ? = Double;
+//string ? = String;
+
+//bytes = Bytes;
+
+//true#3fedd339 = True;
+
+//boolFalse#bc799737 = Bool;
+//boolTrue#997275b5 = Bool;
 
 //vector#1cb5c415 {t:Type} # [ t ] = Vector t;
 
-///////////////////////////////
-/////////////////// Layer cons
-///////////////////////////////
+//ipPort ipv4:int port:int = IpPort;
+//help.configSimple#d997c3c5 date:int expires:int dc_id:int ip_port_list:Vector<ipPort> = help.ConfigSimple;
 
-//invokeAfterMsg#cb9f372d msg_id:long query:!X = X;
-//invokeAfterMsgs#3dc4b4f0 msg_ids:Vector<long> query:!X = X;
-//invokeWithLayer1#53835315 query:!X = X;
-//invokeWithLayer2#289dd1f6 query:!X = X;
-//invokeWithLayer3#b7475268 query:!X = X;
-//invokeWithLayer4#dea0d430 query:!X = X;
-//invokeWithLayer5#417a57ae query:!X = X;
-//invokeWithLayer6#3a64d54d query:!X = X;
-//invokeWithLayer7#a5be56d3 query:!X = X;
-//invokeWithLayer8#e9abd9fd query:!X = X;
-//invokeWithLayer9#76715a63 query:!X = X;
-//invokeWithLayer10#39620c41 query:!X = X;
-//invokeWithLayer11#a6b88fdf query:!X = X;
-//invokeWithLayer12#dda60d3c query:!X = X;
-//invokeWithLayer13#427c8ea2 query:!X = X;
-//invokeWithLayer14#2b9b08fa query:!X = X;
-//invokeWithLayer15#b4418b64 query:!X = X;
-//invokeWithLayer16#cf5f0987 query:!X = X;
-//invokeWithLayer17#50858a19 query:!X = X;
-//invokeWithLayer18#1c900537 query:!X = X;
-//invokeWithLayer#da9b0d0d layer:int query:!X = X; // after 18 layer
-
-///////////////////////////////
-/// Authorization key creation
-///////////////////////////////
-
-resPQ#05162463 nonce:int128 server_nonce:int128 pq:bytes server_public_key_fingerprints:Vector<long> = ResPQ;
-
-p_q_inner_data#83c95aec pq:bytes p:bytes q:bytes nonce:int128 server_nonce:int128 new_nonce:int256 = P_Q_inner_data;
-
-server_DH_params_fail#79cb045d nonce:int128 server_nonce:int128 new_nonce_hash:int128 = Server_DH_Params;
-server_DH_params_ok#d0e8075c nonce:int128 server_nonce:int128 encrypted_answer:bytes = Server_DH_Params;
-
-server_DH_inner_data#b5890dba nonce:int128 server_nonce:int128 g:int dh_prime:bytes g_a:bytes server_time:int = Server_DH_inner_data;
-
-client_DH_inner_data#6643b654 nonce:int128 server_nonce:int128 retry_id:long g_b:bytes = Client_DH_Inner_Data;
-
-dh_gen_ok#3bcbf734 nonce:int128 server_nonce:int128 new_nonce_hash1:int128 = Set_client_DH_params_answer;
-dh_gen_retry#46dc1fb9 nonce:int128 server_nonce:int128 new_nonce_hash2:int128 = Set_client_DH_params_answer;
-dh_gen_fail#a69dae02 nonce:int128 server_nonce:int128 new_nonce_hash3:int128 = Set_client_DH_params_answer;
-
-destroy_auth_key_ok#f660e1d4 = DestroyAuthKeyRes;
-destroy_auth_key_none#0a9f2259 = DestroyAuthKeyRes;
-destroy_auth_key_fail#ea109b13 = DestroyAuthKeyRes;
-
----functions---
-
-// Deprecated since somewhere around February of 2018
-// See https://core.telegram.org/mtproto/auth_key
-req_pq#60469778 nonce:int128 = ResPQ;
-req_pq_multi#be7e8ef1 nonce:int128 = ResPQ;
-
-req_DH_params#d712e4be nonce:int128 server_nonce:int128 p:bytes q:bytes public_key_fingerprint:long encrypted_data:bytes = Server_DH_Params;
-
-set_client_DH_params#f5045f1f nonce:int128 server_nonce:int128 encrypted_data:bytes = Set_client_DH_params_answer;
-
-destroy_auth_key#d1435160 = DestroyAuthKeyRes;
-
-///////////////////////////////
-////////////// System messages
-///////////////////////////////
-
----types---
-
-msgs_ack#62d6b459 msg_ids:Vector<long> = MsgsAck;
-
-bad_msg_notification#a7eff811 bad_msg_id:long bad_msg_seqno:int error_code:int = BadMsgNotification;
-bad_server_salt#edab447b bad_msg_id:long bad_msg_seqno:int error_code:int new_server_salt:long = BadMsgNotification;
-
-msgs_state_req#da69fb52 msg_ids:Vector<long> = MsgsStateReq;
-msgs_state_info#04deb57d req_msg_id:long info:string = MsgsStateInfo;
-msgs_all_info#8cc0d131 msg_ids:Vector<long> info:string = MsgsAllInfo;
-
-msg_detailed_info#276d3ec6 msg_id:long answer_msg_id:long bytes:int status:int = MsgDetailedInfo;
-msg_new_detailed_info#809db6df answer_msg_id:long bytes:int status:int = MsgDetailedInfo;
-
-msg_resend_req#7d861a08 msg_ids:Vector<long> = MsgResendReq;
-
-//rpc_result#f35c6d01 req_msg_id:long result:Object = RpcResult; // parsed manually
-
-rpc_error#2144ca19 error_code:int error_message:string = RpcError;
-
-rpc_answer_unknown#5e2ad36e = RpcDropAnswer;
-rpc_answer_dropped_running#cd78e586 = RpcDropAnswer;
-rpc_answer_dropped#a43ad8b7 msg_id:long seq_no:int bytes:int = RpcDropAnswer;
-
-future_salt#0949d9dc valid_since:int valid_until:int salt:long = FutureSalt;
-future_salts#ae500895 req_msg_id:long now:int salts:vector<future_salt> = FutureSalts;
-
-pong#347773c5 msg_id:long ping_id:long = Pong;
-
-destroy_session_ok#e22045fc session_id:long = DestroySessionRes;
-destroy_session_none#62d350c9 session_id:long = DestroySessionRes;
-
-new_session_created#9ec20908 first_msg_id:long unique_id:long server_salt:long = NewSession;
-
-//message msg_id:long seqno:int bytes:int body:Object = Message; // parsed manually
-//msg_container#73f1f8dc messages:vector<message> = MessageContainer; // parsed manually
-//msg_copy#e06046b2 orig_message:Message = MessageCopy; // parsed manually, not used - use msg_container
-//gzip_packed#3072cfa1 packed_data:string = Object; // parsed manually
-
-http_wait#9299359f max_delay:int wait_after:int max_wait:int = HttpWait;
-
-ipPort ipv4:int port:int = IpPort;
-help.configSimple#d997c3c5 date:int expires:int dc_id:int ip_port_list:Vector<ipPort> = help.ConfigSimple;
-
----functions---
-
-rpc_drop_answer#58e4a740 req_msg_id:long = RpcDropAnswer;
-
-get_future_salts#b921bd04 num:int = FutureSalts;
-
-ping#7abe77ec ping_id:long = Pong;
-ping_delay_disconnect#f3427b8c ping_id:long disconnect_delay:int = Pong;
-
-destroy_session#e7512126 session_id:long = DestroySessionRes;
-
-contest.saveDeveloperInfo#9a5f6e95 vk_id:int name:string phone_number:string age:int city:string = Bool;
-
-///////////////////////////////
-///////// Main application API
-///////////////////////////////
-
----types---
-
-boolFalse#bc799737 = Bool;
-boolTrue#997275b5 = Bool;
-
-true#3fedd339 = True;
-
-vector#1cb5c415 {t:Type} # [ t ] = Vector t;
+ipPort#d433ad73 ipv4:int port:int = IpPort;
+ipPortSecret#37982646 ipv4:int port:int secret:bytes = IpPort;
+accessPointRule#4679b65f phone_prefix_rules:string dc_id:int ips:vector<IpPort> = AccessPointRule;
+help.configSimple#5a592a6c date:int expires:int rules:vector<AccessPointRule> = help.ConfigSimple;
 
 error#c4b9f9bb code:int text:string = Error;
 
-null#56730bcc = Null;
-
 inputPeerEmpty#7f3b18ea = InputPeer;
 inputPeerSelf#7da07ec9 = InputPeer;
 inputPeerChat#179be863 chat_id:int = InputPeer;
@@ -222,7 +102,7 @@ userStatusLastMonth#77ebc742 = UserStatus;
 chatEmpty#9ba2d800 id:int = Chat;
 chat#d91cdd54 flags:# creator:flags.0?true kicked:flags.1?true left:flags.2?true admins_enabled:flags.3?true admin:flags.4?true deactivated:flags.5?true id:int title:string photo:ChatPhoto participants_count:int date:int version:int migrated_to:flags.6?InputChannel = Chat;
 chatForbidden#7328bdb id:int title:string = Chat;
-channel#450b7115 flags:# creator:flags.0?true left:flags.2?true editor:flags.3?true broadcast:flags.5?true verified:flags.7?true megagroup:flags.8?true restricted:flags.9?true democracy:flags.10?true signatures:flags.11?true min:flags.12?true id:int access_hash:flags.13?long title:string username:flags.6?string photo:ChatPhoto date:int version:int restriction_reason:flags.9?string admin_rights:flags.14?ChannelAdminRights banned_rights:flags.15?ChannelBannedRights participants_count:flags.17?int = Chat;
+channel#c88974ac flags:# creator:flags.0?true left:flags.2?true editor:flags.3?true broadcast:flags.5?true verified:flags.7?true megagroup:flags.8?true restricted:flags.9?true democracy:flags.10?true signatures:flags.11?true min:flags.12?true id:int access_hash:flags.13?long title:string username:flags.6?string photo:ChatPhoto date:int version:int restriction_reason:flags.9?string admin_rights:flags.14?ChannelAdminRights banned_rights:flags.15?ChannelBannedRights participants_count:flags.17?int = Chat;
 channelForbidden#289da732 flags:# broadcast:flags.5?true megagroup:flags.8?true id:int access_hash:long title:string until_date:flags.16?int = Chat;
 
 chatFull#2e02a614 id:int participants:ChatParticipants chat_photo:Photo notify_settings:PeerNotifySettings exported_invite:ExportedChatInvite bot_info:Vector<BotInfo> = ChatFull;
@@ -273,6 +153,7 @@ messageActionPaymentSent#40699cd0 currency:string total_amount:long = MessageAct
 messageActionPhoneCall#80e11a7f flags:# call_id:long reason:flags.0?PhoneCallDiscardReason duration:flags.1?int = MessageAction;
 messageActionScreenshotTaken#4792929b = MessageAction;
 messageActionCustomAction#fae69f56 message:string = MessageAction;
+messageActionBotAllowed#abe9affe domain:string = MessageAction;
 
 dialog#e4def5db flags:# pinned:flags.2?true peer:Peer top_message:int read_inbox_max_id:int read_outbox_max_id:int unread_count:int unread_mentions_count:int notify_settings:PeerNotifySettings pts:flags.0?int draft:flags.1?DraftMessage = Dialog;
 
@@ -288,7 +169,7 @@ geoPoint#2049d70c long:double lat:double = GeoPoint;
 
 auth.checkedPhone#811ea28e phone_registered:Bool = auth.CheckedPhone;
 
-auth.sentCode#5e002502 flags:# phone_registered:flags.0?true type:auth.SentCodeType phone_code_hash:string next_type:flags.1?auth.CodeType timeout:flags.2?int = auth.SentCode;
+auth.sentCode#38faab5f flags:# phone_registered:flags.0?true type:auth.SentCodeType phone_code_hash:string next_type:flags.1?auth.CodeType timeout:flags.2?int terms_of_service:flags.3?help.TermsOfService = auth.SentCode;
 
 auth.authorization#cd050916 flags:# tmp_sessions:flags.0?int user:User = auth.Authorization;
 
@@ -297,18 +178,10 @@ auth.exportedAuthorization#df969c2d id:int bytes:bytes = auth.ExportedAuthorizat
 inputNotifyPeer#b8bc5b0c peer:InputPeer = InputNotifyPeer;
 inputNotifyUsers#193b4417 = InputNotifyPeer;
 inputNotifyChats#4a95e84e = InputNotifyPeer;
-inputNotifyAll#a429b886 = InputNotifyPeer;
 
-inputPeerNotifyEventsEmpty#f03064d8 = InputPeerNotifyEvents;
-inputPeerNotifyEventsAll#e86a2c74 = InputPeerNotifyEvents;
+inputPeerNotifySettings#9c3d198e flags:# show_previews:flags.0?Bool silent:flags.1?Bool mute_until:flags.2?int sound:flags.3?string = InputPeerNotifySettings;
 
-inputPeerNotifySettings#38935eb2 flags:# show_previews:flags.0?true silent:flags.1?true mute_until:int sound:string = InputPeerNotifySettings;
-
-peerNotifyEventsEmpty#add53cb3 = PeerNotifyEvents;
-peerNotifyEventsAll#6d1ded88 = PeerNotifyEvents;
-
-peerNotifySettingsEmpty#70a68512 = PeerNotifySettings;
-peerNotifySettings#9acda4c0 flags:# show_previews:flags.0?true silent:flags.1?true mute_until:int sound:string = PeerNotifySettings;
+peerNotifySettings#af509d20 flags:# show_previews:flags.0?Bool silent:flags.1?Bool mute_until:flags.2?int sound:flags.3?string = PeerNotifySettings;
 
 peerSettings#818426cd flags:# report_spam:flags.0?true = PeerSettings;
 
@@ -425,8 +298,8 @@ updateRecentStickers#9a422c20 = Update;
 updateConfig#a229dd06 = Update;
 updatePtsChanged#3354678f = Update;
 updateChannelWebPage#40771900 channel_id:int webpage:WebPage pts:int pts_count:int = Update;
-updateDialogPinned#d711a2cc flags:# pinned:flags.0?true peer:Peer = Update;
-updatePinnedDialogs#d8caf68d flags:# order:flags.0?Vector<Peer> = Update;
+updateDialogPinned#19d27f3c flags:# pinned:flags.0?true peer:DialogPeer = Update;
+updatePinnedDialogs#ea4cb65b flags:# order:flags.0?Vector<DialogPeer> = Update;
 updateBotWebhookJSON#8317c0c3 data:DataJSON = Update;
 updateBotWebhookJSONQuery#9b9240a6 query_id:long data:DataJSON timeout:int = Update;
 updateBotShippingQuery#e0cdc940 query_id:long user_id:int payload:bytes shipping_address:PostAddress = Update;
@@ -460,11 +333,11 @@ photos.photosSlice#15051f54 count:int photos:Vector<Photo> users:Vector<User> =
 photos.photo#20212ca8 photo:Photo users:Vector<User> = photos.Photo;
 
 upload.file#96a18d5 type:storage.FileType mtime:int bytes:bytes = upload.File;
-upload.fileCdnRedirect#ea52fe5a dc_id:int file_token:bytes encryption_key:bytes encryption_iv:bytes cdn_file_hashes:Vector<CdnFileHash> = upload.File;
+upload.fileCdnRedirect#f18cda44 dc_id:int file_token:bytes encryption_key:bytes encryption_iv:bytes file_hashes:Vector<FileHash> = upload.File;
 
-dcOption#5d8c6cc flags:# ipv6:flags.0?true media_only:flags.1?true tcpo_only:flags.2?true cdn:flags.3?true static:flags.4?true id:int ip_address:string port:int = DcOption;
+dcOption#18b7a10d flags:# ipv6:flags.0?true media_only:flags.1?true tcpo_only:flags.2?true cdn:flags.3?true static:flags.4?true id:int ip_address:string port:int secret:flags.10?bytes = DcOption;
 
-config#9c840964 flags:# phonecalls_enabled:flags.1?true default_p2p_contacts:flags.3?true date:int expires:int test_mode:Bool this_dc:int dc_options:Vector<DcOption> chat_size_max:int megagroup_size_max:int forwarded_count_max:int online_update_period_ms:int offline_blur_timeout_ms:int offline_idle_timeout_ms:int online_cloud_timeout_ms:int notify_cloud_delay_ms:int notify_default_delay_ms:int chat_big_size:int push_chat_period_ms:int push_chat_limit:int saved_gifs_limit:int edit_time_limit:int rating_e_decay:int stickers_recent_limit:int stickers_faved_limit:int channels_read_media_period:int tmp_sessions:flags.0?int pinned_dialogs_count_max:int call_receive_timeout_ms:int call_ring_timeout_ms:int call_connect_timeout_ms:int call_packet_timeout_ms:int me_url_prefix:string suggested_lang_code:flags.2?string lang_pack_version:flags.2?int disabled_features:Vector<DisabledFeature> = Config;
+config#eb7bb160 flags:# phonecalls_enabled:flags.1?true default_p2p_contacts:flags.3?true preload_featured_stickers:flags.4?true ignore_phone_entities:flags.5?true revoke_pm_inbox:flags.6?true blocked_mode:flags.8?true date:int expires:int test_mode:Bool this_dc:int dc_options:Vector<DcOption> chat_size_max:int megagroup_size_max:int forwarded_count_max:int online_update_period_ms:int offline_blur_timeout_ms:int offline_idle_timeout_ms:int online_cloud_timeout_ms:int notify_cloud_delay_ms:int notify_default_delay_ms:int push_chat_period_ms:int push_chat_limit:int saved_gifs_limit:int edit_time_limit:int revoke_time_limit:int revoke_pm_time_limit:int rating_e_decay:int stickers_recent_limit:int stickers_faved_limit:int channels_read_media_period:int tmp_sessions:flags.0?int pinned_dialogs_count_max:int call_receive_timeout_ms:int call_ring_timeout_ms:int call_connect_timeout_ms:int call_packet_timeout_ms:int me_url_prefix:string autoupdate_url_prefix:flags.7?string suggested_lang_code:flags.2?string lang_pack_version:flags.2?int = Config;
 
 nearestDc#8e1a1775 country:string this_dc:int nearest_dc:int = NearestDc;
 
@@ -509,7 +382,6 @@ help.support#17c6b5f6 phone_number:string user:User = help.Support;
 notifyPeer#9fd40bd8 peer:Peer = NotifyPeer;
 notifyUsers#b4c83b4c = NotifyPeer;
 notifyChats#c007cec3 = NotifyPeer;
-notifyAll#74d07c60 = NotifyPeer;
 
 sendMessageTypingAction#16bf744e = SendMessageAction;
 sendMessageCancelAction#fd5ec8f5 = SendMessageAction;
@@ -556,21 +428,19 @@ accountDaysTTL#b8d0afdf days:int = AccountDaysTTL;
 documentAttributeImageSize#6c37c15c w:int h:int = DocumentAttribute;
 documentAttributeAnimated#11b58939 = DocumentAttribute;
 documentAttributeSticker#6319d612 flags:# mask:flags.1?true alt:string stickerset:InputStickerSet mask_coords:flags.0?MaskCoords = DocumentAttribute;
-documentAttributeVideo#ef02ce6 flags:# round_message:flags.0?true duration:int w:int h:int = DocumentAttribute;
+documentAttributeVideo#ef02ce6 flags:# round_message:flags.0?true supports_streaming:flags.1?true duration:int w:int h:int = DocumentAttribute;
 documentAttributeAudio#9852f9c6 flags:# voice:flags.10?true duration:int title:flags.0?string performer:flags.1?string waveform:flags.2?bytes = DocumentAttribute;
 documentAttributeFilename#15590068 file_name:string = DocumentAttribute;
 documentAttributeHasStickers#9801d2f7 = DocumentAttribute;
 
 messages.stickersNotModified#f1749a22 = messages.Stickers;
-messages.stickers#8a8ecd32 hash:string stickers:Vector<Document> = messages.Stickers;
+messages.stickers#e4599bbd hash:int stickers:Vector<Document> = messages.Stickers;
 
 stickerPack#12b299d4 emoticon:string documents:Vector<long> = StickerPack;
 
 messages.allStickersNotModified#e86602c3 = messages.AllStickers;
 messages.allStickers#edfd405f hash:int sets:Vector<StickerSet> = messages.AllStickers;
 
-disabledFeature#ae636f24 feature:string description:string = DisabledFeature;
-
 messages.affectedMessages#84d19185 pts:int pts_count:int = messages.AffectedMessages;
 
 contactLinkUnknown#5f4f9247 = ContactLink;
@@ -608,7 +478,7 @@ inputStickerSetEmpty#ffb62b95 = InputStickerSet;
 inputStickerSetID#9de7a269 id:long access_hash:long = InputStickerSet;
 inputStickerSetShortName#861cc8a0 short_name:string = InputStickerSet;
 
-stickerSet#cd303b41 flags:# installed:flags.0?true archived:flags.1?true official:flags.2?true masks:flags.3?true id:long access_hash:long title:string short_name:string count:int hash:int = StickerSet;
+stickerSet#5585a139 flags:# archived:flags.1?true official:flags.2?true masks:flags.3?true installed_date:flags.0?int id:long access_hash:long title:string short_name:string count:int hash:int = StickerSet;
 
 messages.stickerSet#b60a24a6 set:StickerSet packs:Vector<StickerPack> documents:Vector<Document> = messages.StickerSet;
 
@@ -645,6 +515,8 @@ messageEntityPre#73924be0 offset:int length:int language:string = MessageEntity;
 messageEntityTextUrl#76a6d327 offset:int length:int url:string = MessageEntity;
 messageEntityMentionName#352dca58 offset:int length:int user_id:int = MessageEntity;
 inputMessageEntityMentionName#208e68c9 offset:int length:int user_id:InputUser = MessageEntity;
+messageEntityPhone#9b69e34b offset:int length:int = MessageEntity;
+messageEntityCashtag#4c4e743f offset:int length:int = MessageEntity;
 
 inputChannelEmpty#ee8c1e86 = InputChannel;
 inputChannel#afeb712e channel_id:int access_hash:long = InputChannel;
@@ -678,7 +550,7 @@ channels.channelParticipantsNotModified#f0173fe9 = channels.ChannelParticipants;
 
 channels.channelParticipant#d0d9b163 participant:ChannelParticipant users:Vector<User> = channels.ChannelParticipant;
 
-help.termsOfService#f1ee3e90 text:string = help.TermsOfService;
+help.termsOfService#780a0310 flags:# popup:flags.0?true id:DataJSON text:string entities:Vector<MessageEntity> min_age_confirm:flags.1?int = help.TermsOfService;
 
 foundGif#162ecc1f url:string thumb_url:string content_url:string content_type:string w:int h:int = FoundGif;
 foundGifCached#9c750409 url:string photo:Photo document:Document = FoundGif;
@@ -691,11 +563,11 @@ messages.savedGifs#2e0709a5 hash:int gifs:Vector<Document> = messages.SavedGifs;
 inputBotInlineMessageMediaAuto#3380c786 flags:# message:string entities:flags.1?Vector<MessageEntity> reply_markup:flags.2?ReplyMarkup = InputBotInlineMessage;
 inputBotInlineMessageText#3dcd7a87 flags:# no_webpage:flags.0?true message:string entities:flags.1?Vector<MessageEntity> reply_markup:flags.2?ReplyMarkup = InputBotInlineMessage;
 inputBotInlineMessageMediaGeo#c1b15d65 flags:# geo_point:InputGeoPoint period:int reply_markup:flags.2?ReplyMarkup = InputBotInlineMessage;
-inputBotInlineMessageMediaVenue#aaafadc8 flags:# geo_point:InputGeoPoint title:string address:string provider:string venue_id:string reply_markup:flags.2?ReplyMarkup = InputBotInlineMessage;
+inputBotInlineMessageMediaVenue#417bbf11 flags:# geo_point:InputGeoPoint title:string address:string provider:string venue_id:string venue_type:string reply_markup:flags.2?ReplyMarkup = InputBotInlineMessage;
 inputBotInlineMessageMediaContact#2daf01a7 flags:# phone_number:string first_name:string last_name:string reply_markup:flags.2?ReplyMarkup = InputBotInlineMessage;
 inputBotInlineMessageGame#4b425864 flags:# reply_markup:flags.2?ReplyMarkup = InputBotInlineMessage;
 
-inputBotInlineResult#2cbbe15a flags:# id:string type:string title:flags.1?string description:flags.2?string url:flags.3?string thumb_url:flags.4?string content_url:flags.5?string content_type:flags.5?string w:flags.6?int h:flags.6?int duration:flags.7?int send_message:InputBotInlineMessage = InputBotInlineResult;
+inputBotInlineResult#88bf9319 flags:# id:string type:string title:flags.1?string description:flags.2?string url:flags.3?string thumb:flags.4?InputWebDocument content:flags.5?InputWebDocument send_message:InputBotInlineMessage = InputBotInlineResult;
 inputBotInlineResultPhoto#a8d864a7 id:string type:string photo:InputPhoto send_message:InputBotInlineMessage = InputBotInlineResult;
 inputBotInlineResultDocument#fff8fdc4 flags:# id:string type:string title:flags.1?string description:flags.2?string document:InputDocument send_message:InputBotInlineMessage = InputBotInlineResult;
 inputBotInlineResultGame#4fa417f2 id:string short_name:string send_message:InputBotInlineMessage = InputBotInlineResult;
@@ -703,10 +575,10 @@ inputBotInlineResultGame#4fa417f2 id:string short_name:string send_message:Input
 botInlineMessageMediaAuto#764cf810 flags:# message:string entities:flags.1?Vector<MessageEntity> reply_markup:flags.2?ReplyMarkup = BotInlineMessage;
 botInlineMessageText#8c7f65e2 flags:# no_webpage:flags.0?true message:string entities:flags.1?Vector<MessageEntity> reply_markup:flags.2?ReplyMarkup = BotInlineMessage;
 botInlineMessageMediaGeo#b722de65 flags:# geo:GeoPoint period:int reply_markup:flags.2?ReplyMarkup = BotInlineMessage;
-botInlineMessageMediaVenue#4366232e flags:# geo:GeoPoint title:string address:string provider:string venue_id:string reply_markup:flags.2?ReplyMarkup = BotInlineMessage;
+botInlineMessageMediaVenue#8a86659c flags:# geo:GeoPoint title:string address:string provider:string venue_id:string venue_type:string reply_markup:flags.2?ReplyMarkup = BotInlineMessage;
 botInlineMessageMediaContact#35edb4d4 flags:# phone_number:string first_name:string last_name:string reply_markup:flags.2?ReplyMarkup = BotInlineMessage;
 
-botInlineResult#9bebaeb9 flags:# id:string type:string title:flags.1?string description:flags.2?string url:flags.3?string thumb_url:flags.4?string content_url:flags.5?string content_type:flags.5?string w:flags.6?int h:flags.6?int duration:flags.7?int send_message:BotInlineMessage = BotInlineResult;
+botInlineResult#11965f3a flags:# id:string type:string title:flags.1?string description:flags.2?string url:flags.3?string thumb:flags.4?WebDocument content:flags.5?WebDocument send_message:BotInlineMessage = BotInlineResult;
 botInlineMediaResult#17db940b flags:# id:string type:string photo:flags.0?Photo document:flags.1?Document title:flags.2?string description:flags.3?string send_message:BotInlineMessage = BotInlineResult;
 
 messages.botResults#947ca848 flags:# gallery:flags.0?true query_id:long next_offset:flags.1?string switch_pm:flags.2?InlineBotSwitchPM results:Vector<BotInlineResult> cache_time:int users:Vector<User> = messages.BotResults;
@@ -755,7 +627,7 @@ messages.featuredStickersNotModified#4ede3cf = messages.FeaturedStickers;
 messages.featuredStickers#f89d88e5 hash:int sets:Vector<StickerSetCovered> unread:Vector<long> = messages.FeaturedStickers;
 
 messages.recentStickersNotModified#b17f890 = messages.RecentStickers;
-messages.recentStickers#5ce20970 hash:int stickers:Vector<Document> = messages.RecentStickers;
+messages.recentStickers#22f3afb3 hash:int packs:Vector<StickerPack> stickers:Vector<Document> dates:Vector<int> = messages.RecentStickers;
 
 messages.archivedStickers#4fcba9c8 count:int sets:Vector<StickerSetCovered> = messages.ArchivedStickers;
 
@@ -837,10 +709,13 @@ paymentRequestedInfo#909c3f94 flags:# name:flags.0?string phone:flags.1?string e
 paymentSavedCredentialsCard#cdc27a1f id:string title:string = PaymentSavedCredentials;
 
 webDocument#c61acbd8 url:string access_hash:long size:int mime_type:string attributes:Vector<DocumentAttribute> dc_id:int = WebDocument;
+webDocumentNoProxy#f9c8bcc6 url:string size:int mime_type:string attributes:Vector<DocumentAttribute> = WebDocument;
 
 inputWebDocument#9bed434d url:string size:int mime_type:string attributes:Vector<DocumentAttribute> = InputWebDocument;
 
 inputWebFileLocation#c239d686 url:string access_hash:long = InputWebFileLocation;
+inputWebFileGeoPointLocation#66275a62 geo_point:InputGeoPoint w:int h:int zoom:int scale:int = InputWebFileLocation;
+inputWebFileGeoMessageLocation#553f32eb peer:InputPeer msg_id:int w:int h:int zoom:int scale:int = InputWebFileLocation;
 
 upload.webFile#21e753bc size:int mime_type:string file_type:storage.FileType mtime:int bytes:bytes = upload.WebFile;
 
@@ -896,7 +771,7 @@ langPackDifference#f385c1f6 lang_code:string from_version:int version:int string
 
 langPackLanguage#117698f1 name:string native_name:string lang_code:string = LangPackLanguage;
 
-channelAdminRights#5d7ceba5 flags:# change_info:flags.0?true post_messages:flags.1?true edit_messages:flags.2?true delete_messages:flags.3?true ban_users:flags.4?true invite_users:flags.5?true invite_link:flags.6?true pin_messages:flags.7?true add_admins:flags.9?true = ChannelAdminRights;
+channelAdminRights#5d7ceba5 flags:# change_info:flags.0?true post_messages:flags.1?true edit_messages:flags.2?true delete_messages:flags.3?true ban_users:flags.4?true invite_users:flags.5?true invite_link:flags.6?true pin_messages:flags.7?true add_admins:flags.9?true manage_call:flags.10?true = ChannelAdminRights;
 
 channelBannedRights#58cf4249 flags:# view_messages:flags.0?true send_messages:flags.1?true send_media:flags.2?true send_stickers:flags.3?true send_gifs:flags.4?true send_games:flags.5?true send_inline:flags.6?true embed_links:flags.7?true until_date:int = ChannelBannedRights;
 
@@ -925,8 +800,6 @@ channelAdminLogEventsFilter#ea107ae4 flags:# join:flags.0?true leave:flags.1?tru
 
 popularContact#5ce14175 client_id:long importers:int = PopularContact;
 
-cdnFileHash#77eec38f offset:int limit:int hash:bytes = CdnFileHash;
-
 messages.favedStickersNotModified#9e8fa6d3 = messages.FavedStickers;
 messages.favedStickers#f37f2f16 hash:int packs:Vector<StickerPack> stickers:Vector<Document> = messages.FavedStickers;
 
@@ -938,17 +811,41 @@ recentMeUrlStickerSet#bc0a57dc url:string set:StickerSetCovered = RecentMeUrl;
 
 help.recentMeUrls#e0310d7 urls:Vector<RecentMeUrl> chats:Vector<Chat> users:Vector<User> = help.RecentMeUrls;
 
-inputSingleMedia#31bc3d25 media:InputMedia flags:# random_id:long message:string entities:flags.0?Vector<MessageEntity> = InputSingleMedia;
+inputSingleMedia#1cc6e91f flags:# media:InputMedia random_id:long message:string entities:flags.0?Vector<MessageEntity> = InputSingleMedia;
+
+webAuthorization#cac943f2 hash:long bot_id:int domain:string browser:string platform:string date_created:int date_active:int ip:string region:string = WebAuthorization;
+
+account.webAuthorizations#ed56c9fc authorizations:Vector<WebAuthorization> users:Vector<User> = account.WebAuthorizations;
+
+inputMessageID#a676a322 id:int = InputMessage;
+inputMessageReplyTo#bad88395 id:int = InputMessage;
+inputMessagePinned#86872538 = InputMessage;
+
+inputDialogPeer#fcaafeb7 peer:InputPeer = InputDialogPeer;
+
+dialogPeer#e56dbf05 peer:Peer = DialogPeer;
+
+messages.foundStickerSetsNotModified#d54b65d = messages.FoundStickerSets;
+messages.foundStickerSets#5108d648 hash:int sets:Vector<StickerSetCovered> = messages.FoundStickerSets;
+
+fileHash#6242c773 offset:int limit:int hash:bytes = FileHash;
+
+inputClientProxy#75588b3f address:string port:int = InputClientProxy;
+
+help.proxyDataEmpty#e09e1fb8 expires:int = help.ProxyData;
+help.proxyDataPromo#2bf7ee23 expires:int peer:Peer chats:Vector<Chat> users:Vector<User> = help.ProxyData;
+
+help.termsOfServiceUpdateEmpty#e3309f7f expires:int = help.TermsOfServiceUpdate;
+help.termsOfServiceUpdate#28ecf961 expires:int terms_of_service:help.TermsOfService = help.TermsOfServiceUpdate;
 
 ---functions---
 
 invokeAfterMsg#cb9f372d {X:Type} msg_id:long query:!X = X;
 invokeAfterMsgs#3dc4b4f0 {X:Type} msg_ids:Vector<long> query:!X = X;
-initConnection#c7481da6 {X:Type} api_id:int device_model:string system_version:string app_version:string system_lang_code:string lang_pack:string lang_code:string query:!X = X;
+initConnection#785188b8 {X:Type} flags:# api_id:int device_model:string system_version:string app_version:string system_lang_code:string lang_pack:string lang_code:string proxy:flags.0?InputClientProxy query:!X = X;
 invokeWithLayer#da9b0d0d {X:Type} layer:int query:!X = X;
 invokeWithoutUpdates#bf9459b7 {X:Type} query:!X = X;
 
-auth.checkPhone#6fe51dfb phone_number:string = auth.CheckedPhone;
 auth.sendCode#86aef0ec flags:# allow_flashcall:flags.0?true phone_number:string current_number:flags.0?Bool api_id:int api_hash:string = auth.SentCode;
 auth.signUp#1b067634 phone_number:string phone_code_hash:string phone_code:string first_name:string last_name:string = auth.Authorization;
 auth.signIn#bcd51581 phone_number:string phone_code_hash:string phone_code:string = auth.Authorization;
@@ -966,7 +863,7 @@ auth.resendCode#3ef1a9bf phone_number:string phone_code_hash:string = auth.SentC
 auth.cancelCode#1f040578 phone_number:string phone_code_hash:string = Bool;
 auth.dropTempAuthKeys#8e48a188 except_auth_keys:Vector<long> = Bool;
 
-account.registerDevice#1389cc token_type:int token:string app_sandbox:Bool other_uids:Vector<int> = Bool;
+account.registerDevice#5cbea590 token_type:int token:string app_sandbox:Bool secret:bytes other_uids:Vector<int> = Bool;
 account.unregisterDevice#3076c4bf token_type:int token:string other_uids:Vector<int> = Bool;
 account.updateNotifySettings#84be5b93 peer:InputNotifyPeer settings:InputPeerNotifySettings = Bool;
 account.getNotifySettings#12b3ad31 peer:InputNotifyPeer = PeerNotifySettings;
@@ -993,6 +890,9 @@ account.updatePasswordSettings#fa7c4b86 current_password_hash:bytes new_settings
 account.sendConfirmPhoneCode#1516d7bd flags:# allow_flashcall:flags.0?true hash:string current_number:flags.0?Bool = auth.SentCode;
 account.confirmPhone#5f2178c3 phone_code_hash:string phone_code:string = Bool;
 account.getTmpPassword#4a82327e password_hash:bytes period:int = account.TmpPassword;
+account.getWebAuthorizations#182e6d6f = account.WebAuthorizations;
+account.resetWebAuthorization#2d01b9ef hash:long = Bool;
+account.resetWebAuthorizations#682d2594 = Bool;
 
 users.getUsers#d91a548 id:Vector<InputUser> = Vector<User>;
 users.getFullUser#ca30a5b1 id:InputUser = UserFull;
@@ -1013,10 +913,10 @@ contacts.getTopPeers#d4982db5 flags:# correspondents:flags.0?true bots_pm:flags.
 contacts.resetTopPeerRating#1ae373ac category:TopPeerCategory peer:InputPeer = Bool;
 contacts.resetSaved#879537f1 = Bool;
 
-messages.getMessages#4222fa74 id:Vector<int> = messages.Messages;
+messages.getMessages#63c66506 id:Vector<InputMessage> = messages.Messages;
 messages.getDialogs#191ba9c5 flags:# exclude_pinned:flags.0?true offset_date:int offset_id:int offset_peer:InputPeer limit:int = messages.Dialogs;
 messages.getHistory#dcbb8260 peer:InputPeer offset_id:int offset_date:int add_offset:int limit:int max_id:int min_id:int hash:int = messages.Messages;
-messages.search#39e9ea0 flags:# peer:InputPeer q:string from_id:flags.0?InputUser filter:MessagesFilter min_date:int max_date:int offset_id:int add_offset:int limit:int max_id:int min_id:int = messages.Messages;
+messages.search#8614ef68 flags:# peer:InputPeer q:string from_id:flags.0?InputUser filter:MessagesFilter min_date:int max_date:int offset_id:int add_offset:int limit:int max_id:int min_id:int hash:int = messages.Messages;
 messages.readHistory#e306d3a peer:InputPeer max_id:int = messages.AffectedMessages;
 messages.deleteHistory#1c015b09 flags:# just_clear:flags.0?true peer:InputPeer max_id:int = messages.AffectedHistory;
 messages.deleteMessages#e58e95d2 flags:# revoke:flags.0?true id:Vector<int> = messages.AffectedMessages;
@@ -1028,6 +928,7 @@ messages.forwardMessages#708e0195 flags:# silent:flags.5?true background:flags.6
 messages.reportSpam#cf1592db peer:InputPeer = Bool;
 messages.hideReportSpam#a8f1709b peer:InputPeer = Bool;
 messages.getPeerSettings#3672e09c peer:InputPeer = PeerSettings;
+messages.report#bd82b658 peer:InputPeer id:Vector<int> reason:ReportReason = Bool;
 messages.getChats#3c6aa187 id:Vector<int> = messages.Chats;
 messages.getFullChat#3b831c66 chat_id:int = messages.ChatFull;
 messages.editChatTitle#dc452855 chat_id:int title:string = Updates;
@@ -1047,7 +948,7 @@ messages.sendEncryptedService#32d439a4 peer:InputEncryptedChat random_id:long da
 messages.receivedQueue#55a5bb66 max_qts:int = Vector<long>;
 messages.reportEncryptedSpam#4b0c8c0f peer:InputEncryptedChat = Bool;
 messages.readMessageContents#36a73f77 id:Vector<int> = messages.AffectedMessages;
-messages.getStickers#ae22e045 emoticon:string hash:string = messages.Stickers;
+messages.getStickers#43d4f2c emoticon:string hash:int = messages.Stickers;
 messages.getAllStickers#1c9618b1 hash:int = messages.AllStickers;
 messages.getWebPagePreview#8b68b0cc flags:# message:string entities:flags.3?Vector<MessageEntity> = MessageMedia;
 messages.exportChatInvite#7d885289 chat_id:int = ExportedChatInvite;
@@ -1071,11 +972,11 @@ messages.getInlineBotResults#514e999d flags:# bot:InputUser peer:InputPeer geo_p
 messages.setInlineBotResults#eb5ea206 flags:# gallery:flags.0?true private:flags.1?true query_id:long results:Vector<InputBotInlineResult> cache_time:int next_offset:flags.2?string switch_pm:flags.3?InlineBotSwitchPM = Bool;
 messages.sendInlineBotResult#b16e06fe flags:# silent:flags.5?true background:flags.6?true clear_draft:flags.7?true peer:InputPeer reply_to_msg_id:flags.0?int random_id:long query_id:long id:string = Updates;
 messages.getMessageEditData#fda68d36 peer:InputPeer id:int = messages.MessageEditData;
-messages.editMessage#5d1b8dd flags:# no_webpage:flags.1?true stop_geo_live:flags.12?true peer:InputPeer id:int message:flags.11?string reply_markup:flags.2?ReplyMarkup entities:flags.3?Vector<MessageEntity> geo_point:flags.13?InputGeoPoint = Updates;
-messages.editInlineBotMessage#b0e08243 flags:# no_webpage:flags.1?true stop_geo_live:flags.12?true id:InputBotInlineMessageID message:flags.11?string reply_markup:flags.2?ReplyMarkup entities:flags.3?Vector<MessageEntity> geo_point:flags.13?InputGeoPoint = Bool;
+messages.editMessage#c000e4c8 flags:# no_webpage:flags.1?true stop_geo_live:flags.12?true peer:InputPeer id:int message:flags.11?string media:flags.14?InputMedia reply_markup:flags.2?ReplyMarkup entities:flags.3?Vector<MessageEntity> geo_point:flags.13?InputGeoPoint = Updates;
+messages.editInlineBotMessage#adc3e828 flags:# no_webpage:flags.1?true stop_geo_live:flags.12?true id:InputBotInlineMessageID message:flags.11?string media:flags.14?InputMedia reply_markup:flags.2?ReplyMarkup entities:flags.3?Vector<MessageEntity> geo_point:flags.13?InputGeoPoint = Bool;
 messages.getBotCallbackAnswer#810a9fec flags:# game:flags.1?true peer:InputPeer msg_id:int data:flags.0?bytes = messages.BotCallbackAnswer;
 messages.setBotCallbackAnswer#d58f130a flags:# alert:flags.1?true query_id:long message:flags.0?string url:flags.2?string cache_time:int = Bool;
-messages.getPeerDialogs#2d9776b9 peers:Vector<InputPeer> = messages.PeerDialogs;
+messages.getPeerDialogs#e470bcfd peers:Vector<InputDialogPeer> = messages.PeerDialogs;
 messages.saveDraft#bc39e14b flags:# no_webpage:flags.1?true reply_to_msg_id:flags.0?int peer:InputPeer message:string entities:flags.3?Vector<MessageEntity> = Bool;
 messages.getAllDrafts#6a3f8d65 = Updates;
 messages.getFeaturedStickers#2dacca4f hash:int = messages.FeaturedStickers;
@@ -1093,8 +994,8 @@ messages.getInlineGameHighScores#f635e1b id:InputBotInlineMessageID user_id:Inpu
 messages.getCommonChats#d0a48c4 user_id:InputUser max_id:int limit:int = messages.Chats;
 messages.getAllChats#eba80ff0 except_ids:Vector<int> = messages.Chats;
 messages.getWebPage#32ca8f91 url:string hash:int = WebPage;
-messages.toggleDialogPin#3289be6a flags:# pinned:flags.0?true peer:InputPeer = Bool;
-messages.reorderPinnedDialogs#959ff644 flags:# force:flags.0?true order:Vector<InputPeer> = Bool;
+messages.toggleDialogPin#a731e257 flags:# pinned:flags.0?true peer:InputDialogPeer = Bool;
+messages.reorderPinnedDialogs#5b51d63f flags:# force:flags.0?true order:Vector<InputDialogPeer> = Bool;
 messages.getPinnedDialogs#e254d64e = messages.PeerDialogs;
 messages.setBotShippingResults#e5f672fa flags:# query_id:long error:flags.0?string shipping_options:flags.1?Vector<ShippingOption> = Bool;
 messages.setBotPrecheckoutResults#9c2dd95 flags:# success:flags.1?true query_id:long error:flags.0?string = Bool;
@@ -1104,9 +1005,10 @@ messages.getFavedStickers#21ce0b0e hash:int = messages.FavedStickers;
 messages.faveSticker#b9ffc55b id:InputDocument unfave:Bool = Bool;
 messages.getUnreadMentions#46578472 peer:InputPeer offset_id:int add_offset:int limit:int max_id:int min_id:int = messages.Messages;
 messages.readMentions#f0189d3 peer:InputPeer = messages.AffectedHistory;
-messages.getRecentLocations#249431e2 peer:InputPeer limit:int = messages.Messages;
+messages.getRecentLocations#bbc45b09 peer:InputPeer limit:int hash:int = messages.Messages;
 messages.sendMultiMedia#2095512f flags:# silent:flags.5?true background:flags.6?true clear_draft:flags.7?true peer:InputPeer reply_to_msg_id:flags.0?int multi_media:Vector<InputSingleMedia> = Updates;
 messages.uploadEncryptedFile#5057c497 peer:InputEncryptedChat file:InputEncryptedFile = EncryptedFile;
+messages.searchStickerSets#c2b7d08b flags:# exclude_featured:flags.0?true q:string hash:int = messages.FoundStickerSets;
 
 updates.getState#edd4882a = updates.State;
 updates.getDifference#25939651 flags:# pts:int pts_total_limit:flags.0?int date:int qts:int = updates.Difference;
@@ -1122,8 +1024,9 @@ upload.getFile#e3a6cfb5 location:InputFileLocation offset:int limit:int = upload
 upload.saveBigFilePart#de7b673d file_id:long file_part:int file_total_parts:int bytes:bytes = Bool;
 upload.getWebFile#24e6818d location:InputWebFileLocation offset:int limit:int = upload.WebFile;
 upload.getCdnFile#2000bcc3 file_token:bytes offset:int limit:int = upload.CdnFile;
-upload.reuploadCdnFile#1af91c09 file_token:bytes request_token:bytes = Vector<CdnFileHash>;
-upload.getCdnFileHashes#f715c87b file_token:bytes offset:int = Vector<CdnFileHash>;
+upload.reuploadCdnFile#9b2754a8 file_token:bytes request_token:bytes = Vector<FileHash>;
+upload.getCdnFileHashes#4da54231 file_token:bytes offset:int = Vector<FileHash>;
+upload.getFileHashes#c7025931 location:InputFileLocation offset:int = Vector<FileHash>;
 
 help.getConfig#c4f9186b = Config;
 help.getNearestDc#1fb33026 = NearestDc;
@@ -1132,16 +1035,18 @@ help.saveAppLog#6f02f748 events:Vector<InputAppEvent> = Bool;
 help.getInviteText#4d392343 = help.InviteText;
 help.getSupport#9cdf08cd = help.Support;
 help.getAppChangelog#9010ef6f prev_app_version:string = Updates;
-help.getTermsOfService#350170f3 = help.TermsOfService;
 help.setBotUpdatesStatus#ec22cfcd pending_updates_count:int message:string = Bool;
 help.getCdnConfig#52029342 = CdnConfig;
 help.getRecentMeUrls#3dc0f114 referer:string = help.RecentMeUrls;
+help.getProxyData#3d7758e1 = help.ProxyData;
+help.getTermsOfServiceUpdate#2ca51fd1 = help.TermsOfServiceUpdate;
+help.acceptTermsOfService#ee72f79a id:DataJSON = Bool;
 
 channels.readHistory#cc104937 channel:InputChannel max_id:int = Bool;
 channels.deleteMessages#84c1fd4e channel:InputChannel id:Vector<int> = messages.AffectedMessages;
 channels.deleteUserHistory#d10dd71b channel:InputChannel user_id:InputUser = messages.AffectedHistory;
 channels.reportSpam#fe087810 channel:InputChannel user_id:InputUser id:Vector<int> = Bool;
-channels.getMessages#93d7b347 channel:InputChannel id:Vector<int> = messages.Messages;
+channels.getMessages#ad8c9a23 channel:InputChannel id:Vector<InputMessage> = messages.Messages;
 channels.getParticipants#123e05e9 channel:InputChannel filter:ChannelParticipantsFilter offset:int limit:int hash:int = channels.ChannelParticipants;
 channels.getParticipant#546dd7a6 channel:InputChannel user_id:InputUser = channels.ChannelParticipant;
 channels.getChannels#a7f6bbb id:Vector<InputChannel> = messages.Chats;
@@ -1199,4 +1104,4 @@ langpack.getStrings#2e1ee318 lang_code:string keys:Vector<string> = Vector<LangP
 langpack.getDifference#b2e4d7d from_version:int = LangPackDifference;
 langpack.getLanguages#800fd57d = Vector<LangPackLanguage>;
 
-// LAYER 75
+// LAYER 80

telethon_generator/docs_writer.py

@@ -4,7 +4,7 @@ import re
 
 class DocsWriter:
     """Utility class used to write the HTML files used on the documentation"""
-    def __init__(self, filename, type_to_path_function):
+    def __init__(self, filename, type_to_path):
         """Initializes the writer to the specified output file,
            creating the parent directories when used if required.
 
@@ -19,7 +19,7 @@ class DocsWriter:
         self.menu_separator_tag = None
 
         # Utility functions TODO There must be a better way
-        self.type_to_path = lambda t: type_to_path_function(
+        self.type_to_path = lambda t: type_to_path(
             t, relative_to=self.filename
         )
 
@@ -31,29 +31,32 @@ class DocsWriter:
         self._script = ''
 
     # High level writing
-    def write_head(self, title, relative_css_path):
+    def write_head(self, title, relative_css_path, default_css):
         """Writes the head part for the generated document,
            with the given title and CSS
         """
-        self.write('''<!DOCTYPE html>
+        self.write(
+            '''<!DOCTYPE html>
 <html>
 <head>
     <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
-    <title>''')
-
-        self.write(title)
-
-        self.write('''</title>
+    <title>{title}</title>
     <meta name="viewport" content="width=device-width, initial-scale=1.0">
-    <link href="''')
-
-        self.write(relative_css_path)
-
-        self.write('''" rel="stylesheet">
-    <link href="https://fonts.googleapis.com/css?family=Nunito|Source+Code+Pro" rel="stylesheet">
+    <link id="style" href="{rel_css}/docs.{def_css}.css" rel="stylesheet">
+    <script>
+    document.getElementById("style").href = "{rel_css}/docs."
+        + (document.cookie.split(";")[0].split("=")[1] || "{def_css}")
+        + ".css";
+    </script>
+    <link href="https://fonts.googleapis.com/css?family=Nunito|Source+Code+Pro"
+          rel="stylesheet">
 </head>
 <body>
-<div id="main_div">''')
+<div id="main_div">''',
+            title=title,
+            rel_css=relative_css_path.rstrip('/'),
+            def_css=default_css
+        )
 
     def set_menu_separator(self, relative_image_path):
         """Sets the menu separator.
@@ -77,9 +80,7 @@ class DocsWriter:
 
         self.write('<li>')
         if link:
-            self.write('<a href="')
-            self.write(link)
-            self.write('">')
+            self.write('<a href="{}">', link)
 
         # Write the real menu entry text
         self.write(name)
@@ -98,26 +99,21 @@ class DocsWriter:
         """Writes a title header in the document body,
            with an optional depth level
         """
-        self.write('<h%d>' % level)
-        self.write(title)
-        self.write('</h%d>' % level)
+        self.write('<h{level}>{title}</h{level}>', title=title, level=level)
 
     def write_code(self, tlobject):
         """Writes the code for the given 'tlobject' properly
            formatted with hyperlinks
         """
-        self.write('<pre>---')
-        self.write('functions' if tlobject.is_function else 'types')
-        self.write('---\n')
+        self.write('<pre>---{}---\n',
+                   'functions' if tlobject.is_function else 'types')
 
         # Write the function or type and its ID
         if tlobject.namespace:
             self.write(tlobject.namespace)
             self.write('.')
 
-        self.write(tlobject.name)
-        self.write('#')
-        self.write(hex(tlobject.id)[2:].rjust(8, '0'))
+        self.write('{}#{:08x}', tlobject.name, tlobject.id)
 
         # Write all the arguments (or do nothing if there's none)
         for arg in tlobject.args:
@@ -134,20 +130,19 @@ class DocsWriter:
 
             # "Opening" modifiers
             if arg.is_flag:
-                self.write('flags.%d?' % arg.flag_index)
+                self.write('flags.{}?', arg.flag_index)
 
             if arg.is_generic:
                 self.write('!')
 
             if arg.is_vector:
-                self.write(
-                    '<a href="%s">Vector</a>&lt;' % self.type_to_path('vector')
-                )
+                self.write('<a href="{}">Vector</a>&lt;',
+                           self.type_to_path('vector'))
 
             # Argument type
             if arg.type:
                 if add_link:
-                    self.write('<a href="%s">' % self.type_to_path(arg.type))
+                    self.write('<a href="{}">', self.type_to_path(arg.type))
                 self.write(arg.type)
                 if add_link:
                     self.write('</a>')
@@ -176,19 +171,14 @@ class DocsWriter:
                 # use a lower type name for it (see #81)
                 vector, inner = tlobject.result.split('<')
                 inner = inner.strip('>')
-                self.write('<a href="')
-                self.write(self.type_to_path(vector))
-                self.write('">%s</a>&lt;' % vector)
+                self.write('<a href="{}">{}</a>&lt;',
+                           self.type_to_path(vector), vector)
 
-                self.write('<a href="')
-                self.write(self.type_to_path(inner))
-                self.write('">%s</a>' % inner)
-
-                self.write('&gt;')
+                self.write('<a href="{}">{}</a>&gt;',
+                           self.type_to_path(inner), inner)
             else:
-                self.write('<a href="')
-                self.write(self.type_to_path(tlobject.result))
-                self.write('">%s</a>' % tlobject.result)
+                self.write('<a href="{}">{}</a>',
+                           self.type_to_path(tlobject.result), tlobject.result)
 
         self.write('</pre>')
 
@@ -209,17 +199,13 @@ class DocsWriter:
 
         self.write('<td')
         if align:
-            self.write(' style="text-align:')
-            self.write(align)
-            self.write('"')
+            self.write(' style="text-align:{}"', align)
         self.write('>')
 
         if bold:
             self.write('<b>')
         if link:
-            self.write('<a href="')
-            self.write(link)
-            self.write('">')
+            self.write('<a href="{}">', link)
 
         # Finally write the real table data, the given text
         self.write(text)
@@ -244,9 +230,7 @@ class DocsWriter:
 
     def write_text(self, text):
         """Writes a paragraph of text"""
-        self.write('<p>')
-        self.write(text)
-        self.write('</p>')
+        self.write('<p>{}</p>', text)
 
     def write_copy_button(self, text, text_to_copy):
         """Writes a button with 'text' which can be used
@@ -273,16 +257,18 @@ class DocsWriter:
                 'c.select();'
                 'try{document.execCommand("copy")}'
                 'catch(e){}}'
-                '</script>')
+                '</script>'
+            )
 
-        self.write('</div>')
-        self.write(self._script)
-        self.write('</body></html>')
+        self.write('</div>{}</body></html>', self._script)
 
     # "Low" level writing
-    def write(self, s):
+    def write(self, s, *args, **kwargs):
         """Wrapper around handle.write"""
-        self.handle.write(s)
+        if args or kwargs:
+            self.handle.write(s.format(*args, **kwargs))
+        else:
+            self.handle.write(s)
 
     # With block
     def __enter__(self):

telethon_generator/error_generator.py

@@ -1,176 +0,0 @@
-import json
-import re
-import urllib.request
-from collections import defaultdict
-
-URL = 'https://rpc.pwrtelegram.xyz/?all'
-
-known_base_classes = {
-    303: 'InvalidDCError',
-    400: 'BadRequestError',
-    401: 'UnauthorizedError',
-    403: 'ForbiddenError',
-    404: 'NotFoundError',
-    420: 'FloodError',
-    500: 'ServerError',
-}
-
-# The API doesn't return the code for some (vital) errors. They are
-# all assumed to be 400, except these well-known ones that aren't.
-known_codes = {
-    'ACTIVE_USER_REQUIRED': 401,
-    'AUTH_KEY_UNREGISTERED': 401,
-    'USER_DEACTIVATED': 401
-}
-
-
-def fetch_errors(output, url=URL):
-    print('Opening a connection to', url, '...')
-    r = urllib.request.urlopen(urllib.request.Request(
-        url, headers={'User-Agent' : 'Mozilla/5.0'}
-    ))
-    print('Checking response...')
-    data = json.loads(
-        r.read().decode(r.info().get_param('charset') or 'utf-8')
-    )
-    if data.get('ok'):
-        print('Response was okay, saving data')
-        with open(output, 'w', encoding='utf-8') as f:
-            json.dump(data, f, sort_keys=True)
-        return True
-    else:
-        print('The data received was not okay:')
-        print(json.dumps(data, indent=4, sort_keys=True))
-        return False
-
-
-def get_class_name(error_code):
-    if isinstance(error_code, int):
-        return known_base_classes.get(
-            error_code, 'RPCError' + str(error_code).replace('-', 'Neg')
-        )
-
-    if 'FIRSTNAME' in error_code:
-        error_code = error_code.replace('FIRSTNAME', 'FIRST_NAME')
-
-    result = re.sub(
-        r'_([a-z])', lambda m: m.group(1).upper(), error_code.lower()
-    )
-    return result[:1].upper() + result[1:].replace('_', '') + 'Error'
-
-
-def write_error(f, code, name, desc, capture_name):
-    f.write(
-        '\n\nclass {}({}):\n    def __init__(self, **kwargs):\n        '
-        ''.format(name, get_class_name(code))
-    )
-    if capture_name:
-        f.write(
-            "self.{} = int(kwargs.get('capture', 0))\n        ".format(capture_name)
-        )
-    f.write('super(Exception, self).__init__({}'.format(repr(desc)))
-    if capture_name:
-        f.write('.format(self.{})'.format(capture_name))
-    f.write(')\n')
-
-
-def generate_code(output, json_file, errors_desc):
-    with open(json_file, encoding='utf-8') as f:
-        data = json.load(f)
-
-    errors = defaultdict(set)
-    # PWRTelegram's API doesn't return all errors, which we do need here.
-    # Add some special known-cases manually first.
-    errors[420].update((
-        'FLOOD_WAIT_X', 'FLOOD_TEST_PHONE_WAIT_X'
-    ))
-    errors[401].update((
-        'AUTH_KEY_INVALID', 'SESSION_EXPIRED', 'SESSION_REVOKED'
-    ))
-    errors[303].update((
-        'FILE_MIGRATE_X', 'PHONE_MIGRATE_X',
-        'NETWORK_MIGRATE_X', 'USER_MIGRATE_X'
-    ))
-    for error_code, method_errors in data['result'].items():
-        for error_list in method_errors.values():
-            for error in error_list:
-                errors[int(error_code)].add(re.sub('_\d+', '_X', error).upper())
-
-    # Some errors are in the human result, but not with a code. Assume code 400
-    for error in data['human_result']:
-        if error[0] != '-' and not error.isdigit():
-            error = re.sub('_\d+', '_X', error).upper()
-            if not any(error in es for es in errors.values()):
-                errors[known_codes.get(error, 400)].add(error)
-
-    # Some error codes are not known, so create custom base classes if needed
-    needed_base_classes = [
-        (e, get_class_name(e)) for e in errors if e not in known_base_classes
-    ]
-
-    # Prefer the descriptions that are related with Telethon way of coding to
-    # those that PWRTelegram's API provides.
-    telethon_descriptions = {}
-    with open(errors_desc, encoding='utf-8') as f:
-        for line in f:
-            line = line.strip()
-            if line and not line.startswith('#'):
-                equal = line.index('=')
-                message, description = line[:equal], line[equal + 1:]
-                telethon_descriptions[message.rstrip()] = description.lstrip()
-
-    # Names for the captures, or 'x' if unknown
-    capture_names = {
-        'FloodWaitError': 'seconds',
-        'FloodTestPhoneWaitError': 'seconds',
-        'FileMigrateError': 'new_dc',
-        'NetworkMigrateError': 'new_dc',
-        'PhoneMigrateError': 'new_dc',
-        'UserMigrateError': 'new_dc',
-        'FilePartMissingError': 'which'
-    }
-
-    # Everything ready, generate the code
-    with open(output, 'w', encoding='utf-8') as f:
-        f.write(
-            'from .rpc_base_errors import RPCError, BadMessageError, {}\n'.format(
-                ", ".join(known_base_classes.values()))
-        )
-        for code, cls in needed_base_classes:
-            f.write(
-                '\n\nclass {}(RPCError):\n    code = {}\n'.format(cls, code)
-            )
-
-        patterns = []  # Save this dictionary later in the generated code
-        for error_code, error_set in errors.items():
-            for error in sorted(error_set):
-                description = telethon_descriptions.get(
-                    error, '\n'.join(data['human_result'].get(
-                        error, ['No description known.']
-                    ))
-                )
-                has_captures = '_X' in error
-                if has_captures:
-                    name = get_class_name(error.replace('_X', ''))
-                    pattern = error.replace('_X', r'_(\d+)')
-                else:
-                    name, pattern = get_class_name(error), error
-
-                patterns.append((pattern, name))
-                capture = capture_names.get(name, 'x') if has_captures else None
-                # TODO Some errors have the same name but different code,
-                # split this across different files?
-                write_error(f, error_code, name, description, capture)
-
-        f.write('\n\nrpc_errors_all = {\n')
-        for pattern, name in patterns:
-            f.write('    {}: {},\n'.format(repr(pattern), name))
-        f.write('}\n')
-
-
-if __name__ == '__main__':
-    if input('generate (y/n)?: ').lower() == 'y':
-        generate_code('../telethon/errors/rpc_error_list.py',
-                      'errors.json', 'error_descriptions')
-    elif input('fetch (y/n)?: ').lower() == 'y':
-        fetch_errors('errors.json')

telethon_generator/fetch_pwrtelegram.py

@@ -0,0 +1,30 @@
+import sys
+import json
+import urllib.request
+
+FILES = (
+    ('data/errors.json', 'https://rpc.pwrtelegram.xyz/?all'),
+    ('data/invalid_bot_methods.json', 'https://rpc.pwrtelegram.xyz/?bot')
+)
+
+
+def fetch_json(output, url):
+    r = urllib.request.urlopen(urllib.request.Request(
+        url, headers={'User-Agent' : 'Mozilla/5.0'}
+    ))
+    data = json.loads(
+        r.read().decode(r.info().get_param('charset') or 'utf-8')
+    )
+    if data.get('ok'):
+        del data['ok']
+        if len(data) == 1:
+            data = data[next(iter(data.keys()))]
+        with open(output, 'w', encoding='utf-8') as f:
+            json.dump(data, f, sort_keys=True)
+    else:
+        print(json.dumps(data, indent=4, sort_keys=True), file=sys.stderr)
+
+
+if __name__ == '__main__':
+    for output, url in FILES:
+        fetch_json(output, url)

telethon_generator/generators/__init__.py

@@ -0,0 +1,3 @@
+from .errors import generate_errors
+from .tlobject import generate_tlobjects, clean_tlobjects
+from .docs import generate_docs

telethon_generator/generators/docs.py

@@ -1,124 +1,75 @@
 #!/usr/bin/env python3
+import functools
 import os
 import re
-import sys
 import shutil
-try:
-    from .docs_writer import DocsWriter
-except (ImportError, SystemError):
-    from docs_writer import DocsWriter
+from collections import defaultdict
 
-# Small trick so importing telethon_generator works
-sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..'))
-
-from telethon_generator.parser import TLParser, TLObject
+from ..docs_writer import DocsWriter
+from ..parsers import TLObject
+from ..utils import snake_to_camel_case
 
 
-# TLObject -> Python class name
-def get_class_name(tlobject):
-    """Gets the class name following the Python style guidelines"""
-    # Courtesy of http://stackoverflow.com/a/31531797/4759433
+CORE_TYPES = {
+    'int', 'long', 'int128', 'int256', 'double',
+    'vector', 'string', 'bool', 'true', 'bytes', 'date'
+}
+
+
+def _get_file_name(tlobject):
+    """``ClassName -> class_name.html``."""
     name = tlobject.name if isinstance(tlobject, TLObject) else tlobject
-    result = re.sub(r'_([a-z])', lambda m: m.group(1).upper(), name)
-
-    # Replace '_' with '' once again to make sure it doesn't appear on the name
-    result = result[:1].upper() + result[1:].replace('_', '')
-
-    # If it's a function, let it end with "Request" to identify them more easily
-    if isinstance(tlobject, TLObject) and tlobject.is_function:
-        result += 'Request'
-
-    return result
-
-
-# TLObject -> filename
-def get_file_name(tlobject, add_extension=False):
-    """Gets the file name in file_name_format.html for the given TLObject.
-       Only its name may also be given if the full TLObject is not available"""
-    if isinstance(tlobject, TLObject):
-        name = tlobject.name
-    else:
-        name = tlobject
-
     # Courtesy of http://stackoverflow.com/a/1176023/4759433
     s1 = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', name)
     result = re.sub('([a-z0-9])([A-Z])', r'\1_\2', s1).lower()
-    if add_extension:
-        return result + '.html'
-    else:
-        return result
+    return '{}.html'.format(result)
 
 
-# TLObject -> from ... import ...
 def get_import_code(tlobject):
+    """``TLObject -> from ... import ...``."""
     kind = 'functions' if tlobject.is_function else 'types'
     ns = '.' + tlobject.namespace if tlobject.namespace else ''
-
     return 'from telethon.tl.{}{} import {}'\
-        .format(kind, ns, get_class_name(tlobject))
+        .format(kind, ns, tlobject.class_name)
 
 
-def get_create_path_for(tlobject):
-    """Gets the file path (and creates the parent directories)
-       for the given 'tlobject', relative to nothing; only its local path"""
-
-    # Determine the output directory
+def _get_create_path_for(root, tlobject, make=True):
+    """Creates and returns the path for the given TLObject at root."""
     out_dir = 'methods' if tlobject.is_function else 'constructors'
-
     if tlobject.namespace:
         out_dir = os.path.join(out_dir, tlobject.namespace)
 
-    # Ensure that it exists
-    os.makedirs(out_dir, exist_ok=True)
-
-    # Return the resulting filename
-    return os.path.join(out_dir, get_file_name(tlobject, add_extension=True))
+    out_dir = os.path.join(root, out_dir)
+    if make:
+        os.makedirs(out_dir, exist_ok=True)
+    return os.path.join(out_dir, _get_file_name(tlobject))
 
 
-def is_core_type(type_):
-    """Returns "true" if the type is considered a core type"""
-    return type_.lower() in {
-        'int', 'long', 'int128', 'int256', 'double',
-        'vector', 'string', 'bool', 'true', 'bytes', 'date'
-    }
-
-
-def get_path_for_type(type_, relative_to='.'):
-    """Similar to getting the path for a TLObject, it might not be possible
-       to have the TLObject itself but rather its name (the type);
-       this method works in the same way, returning a relative path"""
-    if is_core_type(type_):
+def _get_path_for_type(root, type_, relative_to='.'):
+    """Similar to `_get_create_path_for` but for only type names."""
+    if type_.lower() in CORE_TYPES:
         path = 'index.html#%s' % type_.lower()
-
     elif '.' in type_:
-        # If it's not a core type, then it has to be a custom Telegram type
         namespace, name = type_.split('.')
-        path = 'types/%s/%s' % (namespace, get_file_name(name, True))
+        path = 'types/%s/%s' % (namespace, _get_file_name(name))
     else:
-        path = 'types/%s' % get_file_name(type_, True)
+        path = 'types/%s' % _get_file_name(type_)
 
-    return get_relative_path(path, relative_to)
+    return _get_relative_path(os.path.join(root, path), relative_to)
 
 
-# Destination path from the current position -> relative to the given path
-def get_relative_path(destination, relative_to):
-    if os.path.isfile(relative_to):
+def _get_relative_path(destination, relative_to, folder=False):
+    """Return the relative path to destination from relative_to."""
+    if not folder:
         relative_to = os.path.dirname(relative_to)
 
     return os.path.relpath(destination, start=relative_to)
 
 
-def get_relative_paths(original, relative_to):
-    """Converts the dictionary of 'original' paths to relative paths
-       starting from the given 'relative_to' file"""
-    return {k: get_relative_path(v, relative_to) for k, v in original.items()}
-
-
-# Generate a index.html file for the given folder
-def find_title(html_file):
-    """Finds the <title> for the given HTML file, or (Unknown)"""
-    with open(html_file) as handle:
-        for line in handle:
+def _find_title(html_file):
+    """Finds the <title> for the given HTML file, or (Unknown)."""
+    with open(html_file) as fp:
+        for line in fp:
             if '<title>' in line:
                 # + 7 to skip len('<title>')
                 return line[line.index('<title>') + 7:line.index('</title>')]
@@ -126,10 +77,11 @@ def find_title(html_file):
     return '(Unknown)'
 
 
-def build_menu(docs, filename, relative_main_index):
+def _build_menu(docs, filename, root, relative_main_index):
     """Builds the menu using the given DocumentWriter up to 'filename',
        which must be a file (it cannot be a directory)"""
     # TODO Maybe this could be part of DocsWriter itself, "build path menu"
+    filename = _get_relative_path(filename, root)
     docs.add_menu('API', relative_main_index)
 
     items = filename.split('/')
@@ -144,9 +96,8 @@ def build_menu(docs, filename, relative_main_index):
     docs.end_menu()
 
 
-def generate_index(folder, original_paths):
+def _generate_index(folder, original_paths, root):
     """Generates the index file for the specified folder"""
-
     # Determine the namespaces listed here (as sub folders)
     # and the files (.html files) that we should link to
     namespaces = []
@@ -157,27 +108,30 @@ def generate_index(folder, original_paths):
         elif item != 'index.html':
             files.append(item)
 
-    # We work with relative paths
-    paths = get_relative_paths(original_paths, relative_to=folder)
+    paths = {k: _get_relative_path(v, folder, folder=True)
+             for k, v in original_paths.items()}
 
     # Now that everything is setup, write the index.html file
     filename = os.path.join(folder, 'index.html')
-    with DocsWriter(filename, type_to_path_function=get_path_for_type) as docs:
+    with DocsWriter(filename, type_to_path=_get_path_for_type) as docs:
         # Title should be the current folder name
-        docs.write_head(folder.title(), relative_css_path=paths['css'])
+        docs.write_head(folder.title(),
+                        relative_css_path=paths['css'],
+                        default_css=original_paths['default_css'])
 
         docs.set_menu_separator(paths['arrow'])
-        build_menu(docs, filename, relative_main_index=paths['index_all'])
-
-        docs.write_title(folder.title())
+        _build_menu(docs, filename, root,
+                    relative_main_index=paths['index_all'])
 
+        docs.write_title(_get_relative_path(folder, root, folder=True).title())
         if namespaces:
             docs.write_title('Namespaces', level=3)
             docs.begin_table(4)
             namespaces.sort()
             for namespace in namespaces:
                 # For every namespace, also write the index of it
-                generate_index(os.path.join(folder, namespace), original_paths)
+                _generate_index(os.path.join(folder, namespace),
+                                original_paths, root)
                 docs.add_row(namespace.title(),
                              link=os.path.join(namespace, 'index.html'))
 
@@ -186,7 +140,7 @@ def generate_index(folder, original_paths):
         docs.write_title('Available items')
         docs.begin_table(2)
 
-        files = [(f, find_title(os.path.join(folder, f))) for f in files]
+        files = [(f, _find_title(os.path.join(folder, f))) for f in files]
         files.sort(key=lambda t: t[1])
 
         for file, title in files:
@@ -196,8 +150,8 @@ def generate_index(folder, original_paths):
         docs.end_body()
 
 
-def get_description(arg):
-    """Generates a proper description for the given argument"""
+def _get_description(arg):
+    """Generates a proper description for the given argument."""
     desc = []
     otherwise = False
     if arg.can_be_inferred:
@@ -235,7 +189,7 @@ def get_description(arg):
     )
 
 
-def copy_replace(src, dst, replacements):
+def _copy_replace(src, dst, replacements):
     """Copies the src file into dst applying the replacements dict"""
     with open(src) as infile, open(dst, 'w') as outfile:
         outfile.write(re.sub(
@@ -245,12 +199,17 @@ def copy_replace(src, dst, replacements):
         ))
 
 
-def generate_documentation(scheme_file):
-    """Generates the documentation HTML files from from scheme.tl to
-       /methods and /constructors, etc.
+def _write_html_pages(tlobjects, errors, layer, input_res, output_dir):
     """
+    Generates the documentation HTML files from from ``scheme.tl``
+    to ``/methods`` and ``/constructors``, etc.
+    """
+    # Save 'Type: [Constructors]' for use in both:
+    # * Seeing the return type or constructors belonging to the same type.
+    # * Generating the types documentation, showing available constructors.
+    # TODO Tried using 'defaultdict(list)' with strange results, make it work.
     original_paths = {
-        'css': 'css/docs.css',
+        'css': 'css',
         'arrow': 'img/arrow.svg',
         'search.js': 'js/search.js',
         '404': '404.html',
@@ -259,46 +218,57 @@ def generate_documentation(scheme_file):
         'index_methods': 'methods/index.html',
         'index_constructors': 'constructors/index.html'
     }
-    tlobjects = tuple(TLParser.parse_file(scheme_file))
+    original_paths = {k: os.path.join(output_dir, v)
+                      for k, v in original_paths.items()}
 
-    print('Generating constructors and functions documentation...')
-
-    # Save 'Type: [Constructors]' for use in both:
-    # * Seeing the return type or constructors belonging to the same type.
-    # * Generating the types documentation, showing available constructors.
-    # TODO Tried using 'defaultdict(list)' with strange results, make it work.
-    tltypes = {}
-    tlfunctions = {}
+    original_paths['default_css'] = 'light'  # docs.<name>.css, local path
+    type_to_constructors = {}
+    type_to_functions = {}
     for tlobject in tlobjects:
-        # Select to which dictionary we want to store this type
-        dictionary = tlfunctions if tlobject.is_function else tltypes
-
-        if tlobject.result in dictionary:
-            dictionary[tlobject.result].append(tlobject)
+        d = type_to_functions if tlobject.is_function else type_to_constructors
+        if tlobject.result in d:
+            d[tlobject.result].append(tlobject)
         else:
-            dictionary[tlobject.result] = [tlobject]
+            d[tlobject.result] = [tlobject]
 
-    for tltype, constructors in tltypes.items():
-        tltypes[tltype] = list(sorted(constructors, key=lambda c: c.name))
+    for t, cs in type_to_constructors.items():
+        type_to_constructors[t] = list(sorted(cs, key=lambda c: c.name))
+
+    # Telegram may send errors with the same str_code but different int_code.
+    # They are all imported on telethon.errors anyway so makes no difference.
+    errors = list(sorted({e.str_code: e for e in errors}.values(),
+                         key=lambda e: e.name))
+
+    method_causes_errors = defaultdict(list)
+    for error in errors:
+        for method in error.caused_by:
+            method_causes_errors[method].append(error)
+
+    # Since the output directory is needed everywhere partially apply it now
+    create_path_for = functools.partial(_get_create_path_for, output_dir)
+    path_for_type = functools.partial(_get_path_for_type, output_dir)
 
     for tlobject in tlobjects:
-        filename = get_create_path_for(tlobject)
+        filename = create_path_for(tlobject)
+        paths = {k: _get_relative_path(v, filename)
+                 for k, v in original_paths.items()}
 
-        # Determine the relative paths for this file
-        paths = get_relative_paths(original_paths, relative_to=filename)
-
-        with DocsWriter(filename, type_to_path_function=get_path_for_type) \
-                as docs:
-            docs.write_head(
-                title=get_class_name(tlobject),
-                relative_css_path=paths['css'])
+        with DocsWriter(filename, type_to_path=path_for_type) as docs:
+            docs.write_head(title=tlobject.class_name,
+                            relative_css_path=paths['css'],
+                            default_css=original_paths['default_css'])
 
             # Create the menu (path to the current TLObject)
             docs.set_menu_separator(paths['arrow'])
-            build_menu(docs, filename, relative_main_index=paths['index_all'])
+            _build_menu(docs, filename, output_dir,
+                        relative_main_index=paths['index_all'])
 
             # Create the page title
-            docs.write_title(get_class_name(tlobject))
+            docs.write_title(tlobject.class_name)
+
+            if tlobject.is_function:
+                docs.write_text('Bots <strong>can{}</strong> use this method.'
+                                .format("" if tlobject.bot_usable else "'t"))
 
             # Write the code definition for this TLObject
             docs.write_code(tlobject)
@@ -328,24 +298,24 @@ def generate_documentation(scheme_file):
                     inner = tlobject.result
 
                 docs.begin_table(column_count=1)
-                docs.add_row(inner, link=get_path_for_type(
+                docs.add_row(inner, link=path_for_type(
                     inner, relative_to=filename
                 ))
                 docs.end_table()
 
-                constructors = tltypes.get(inner, [])
-                if not constructors:
+                cs = type_to_constructors.get(inner, [])
+                if not cs:
                     docs.write_text('This type has no instances available.')
-                elif len(constructors) == 1:
+                elif len(cs) == 1:
                     docs.write_text('This type can only be an instance of:')
                 else:
                     docs.write_text('This type can be an instance of either:')
 
                 docs.begin_table(column_count=2)
-                for constructor in constructors:
-                    link = get_create_path_for(constructor)
-                    link = get_relative_path(link, relative_to=filename)
-                    docs.add_row(get_class_name(constructor), link=link)
+                for constructor in cs:
+                    link = create_path_for(constructor)
+                    link = _get_relative_path(link, relative_to=filename)
+                    docs.add_row(constructor.class_name, link=link)
                 docs.end_table()
 
             # Return (or similar types) written. Now parameters/members
@@ -375,11 +345,11 @@ def generate_documentation(scheme_file):
                     else:
                         docs.add_row(
                             arg.type, align='center', link=
-                            get_path_for_type(arg.type, relative_to=filename)
+                            path_for_type(arg.type, relative_to=filename)
                          )
 
                     # Add a description for this argument
-                    docs.add_row(get_description(arg))
+                    docs.add_row(_get_description(arg))
 
                 docs.end_table()
             else:
@@ -388,6 +358,25 @@ def generate_documentation(scheme_file):
                 else:
                     docs.write_text('This type has no members.')
 
+            if tlobject.is_function:
+                docs.write_title('Known RPC errors')
+                errors = method_causes_errors[tlobject.fullname]
+                if not errors:
+                    docs.write_text("This request can't cause any RPC error "
+                                    "as far as we know.")
+                else:
+                    docs.write_text(
+                        'This request can cause {} known error{}:'.format(
+                            len(errors), '' if len(errors) == 1 else 's'
+                    ))
+                    docs.begin_table(column_count=2)
+                    for error in errors:
+                        docs.add_row('<code>{}</code>'.format(error.name))
+                        docs.add_row('{}.'.format(error.description))
+                    docs.end_table()
+                    docs.write_text('You can import these from '
+                                    '<code>telethon.errors</code>.')
+
             # TODO Bit hacky, make everything like this? (prepending '../')
             depth = '../' * (2 if tlobject.namespace else 1)
             docs.add_script(src='prependPath = "{}";'.format(depth))
@@ -396,55 +385,54 @@ def generate_documentation(scheme_file):
 
     # Find all the available types (which are not the same as the constructors)
     # Each type has a list of constructors associated to it, hence is a map
-    print('Generating types documentation...')
-    for tltype, constructors in tltypes.items():
-        filename = get_path_for_type(tltype)
+    for t, cs in type_to_constructors.items():
+        filename = path_for_type(t)
         out_dir = os.path.dirname(filename)
         if out_dir:
             os.makedirs(out_dir, exist_ok=True)
 
         # Since we don't have access to the full TLObject, split the type
-        if '.' in tltype:
-            namespace, name = tltype.split('.')
+        if '.' in t:
+            namespace, name = t.split('.')
         else:
-            namespace, name = None, tltype
+            namespace, name = None, t
 
-        # Determine the relative paths for this file
-        paths = get_relative_paths(original_paths, relative_to=out_dir)
+        paths = {k: _get_relative_path(v, out_dir, folder=True)
+                 for k, v in original_paths.items()}
 
-        with DocsWriter(filename, type_to_path_function=get_path_for_type) \
-                as docs:
-            docs.write_head(
-                title=get_class_name(name),
-                relative_css_path=paths['css'])
+        with DocsWriter(filename, type_to_path=path_for_type) as docs:
+            docs.write_head(title=snake_to_camel_case(name),
+                            relative_css_path=paths['css'],
+                            default_css=original_paths['default_css'])
 
             docs.set_menu_separator(paths['arrow'])
-            build_menu(docs, filename, relative_main_index=paths['index_all'])
+            _build_menu(docs, filename, output_dir,
+                        relative_main_index=paths['index_all'])
 
             # Main file title
-            docs.write_title(get_class_name(name))
+            docs.write_title(snake_to_camel_case(name))
 
             # List available constructors for this type
             docs.write_title('Available constructors', level=3)
-            if not constructors:
+            if not cs:
                 docs.write_text('This type has no constructors available.')
-            elif len(constructors) == 1:
+            elif len(cs) == 1:
                 docs.write_text('This type has one constructor available.')
             else:
                 docs.write_text('This type has %d constructors available.' %
-                                len(constructors))
+                                len(cs))
 
             docs.begin_table(2)
-            for constructor in constructors:
+            for constructor in cs:
                 # Constructor full name
-                link = get_create_path_for(constructor)
-                link = get_relative_path(link, relative_to=filename)
-                docs.add_row(get_class_name(constructor), link=link)
+                link = create_path_for(constructor)
+                link = _get_relative_path(link, relative_to=filename)
+                docs.add_row(constructor.class_name, link=link)
             docs.end_table()
 
             # List all the methods which return this type
             docs.write_title('Methods returning this type', level=3)
-            functions = tlfunctions.get(tltype, [])
+            functions = type_to_functions.get(t, [])
             if not functions:
                 docs.write_text('No method returns this type.')
             elif len(functions) == 1:
@@ -457,17 +445,17 @@ def generate_documentation(scheme_file):
 
             docs.begin_table(2)
             for func in functions:
-                link = get_create_path_for(func)
-                link = get_relative_path(link, relative_to=filename)
-                docs.add_row(get_class_name(func), link=link)
+                link = create_path_for(func)
+                link = _get_relative_path(link, relative_to=filename)
+                docs.add_row(func.class_name, link=link)
             docs.end_table()
 
             # List all the methods which take this type as input
             docs.write_title('Methods accepting this type as input', level=3)
             other_methods = sorted(
-                (t for t in tlobjects
-                 if any(tltype == a.type for a in t.args) and t.is_function),
-                key=lambda t: t.name
+                (u for u in tlobjects
+                 if any(a.type == t for a in u.args) and u.is_function),
+                key=lambda u: u.name
             )
             if not other_methods:
                 docs.write_text(
@@ -482,18 +470,17 @@ def generate_documentation(scheme_file):
 
             docs.begin_table(2)
             for ot in other_methods:
-                link = get_create_path_for(ot)
-                link = get_relative_path(link, relative_to=filename)
-                docs.add_row(get_class_name(ot), link=link)
+                link = create_path_for(ot)
+                link = _get_relative_path(link, relative_to=filename)
+                docs.add_row(ot.class_name, link=link)
             docs.end_table()
 
             # List every other type which has this type as a member
             docs.write_title('Other types containing this type', level=3)
             other_types = sorted(
-                (t for t in tlobjects
-                 if any(tltype == a.type for a in t.args)
-                 and not t.is_function
-                 ), key=lambda t: t.name
+                (u for u in tlobjects
+                 if any(a.type == t for a in u.args) and not u.is_function),
+                key=lambda u: u.name
             )
 
             if not other_types:
@@ -509,9 +496,9 @@ def generate_documentation(scheme_file):
 
             docs.begin_table(2)
             for ot in other_types:
-                link = get_create_path_for(ot)
-                link = get_relative_path(link, relative_to=filename)
-                docs.add_row(get_class_name(ot), link=link)
+                link = create_path_for(ot)
+                link = _get_relative_path(link, relative_to=filename)
+                docs.add_row(ot.class_name, link=link)
             docs.end_table()
             docs.end_body()
 
@@ -519,22 +506,21 @@ def generate_documentation(scheme_file):
     # This will be done automatically and not taking into account any extra
     # information that we have available, simply a file listing all the others
     # accessible by clicking on their title
-    print('Generating indices...')
     for folder in ['types', 'methods', 'constructors']:
-        generate_index(folder, original_paths)
+        _generate_index(os.path.join(output_dir, folder), original_paths,
+                        output_dir)
 
     # Write the final core index, the main index for the rest of files
-    layer = TLParser.find_layer(scheme_file)
     types = set()
     methods = []
-    constructors = []
+    cs = []
     for tlobject in tlobjects:
         if tlobject.is_function:
             methods.append(tlobject)
         else:
-            constructors.append(tlobject)
+            cs.append(tlobject)
 
-        if not is_core_type(tlobject.result):
+        if not tlobject.result.lower() in CORE_TYPES:
             if re.search('^vector<', tlobject.result, re.IGNORECASE):
                 types.add(tlobject.result.split('<')[1].strip('>'))
             else:
@@ -542,41 +528,47 @@ def generate_documentation(scheme_file):
 
     types = sorted(types)
     methods = sorted(methods, key=lambda m: m.name)
-    constructors = sorted(constructors, key=lambda c: c.name)
+    cs = sorted(cs, key=lambda c: c.name)
 
-    def fmt(xs):
-        ys = {x: get_class_name(x) for x in xs}  # cache TLObject: display
-        zs = {}  # create a dict to hold those which have duplicated keys
-        for y in ys.values():
-            zs[y] = y in zs
-        return ', '.join(
-            '"{}.{}"'.format(x.namespace, ys[x])
-            if zs[ys[x]] and getattr(x, 'namespace', None)
-            else '"{}"'.format(ys[x]) for x in xs
-        )
-
-    request_names = fmt(methods)
-    type_names = fmt(types)
-    constructor_names = fmt(constructors)
-
-    def fmt(xs, formatter):
-        return ', '.join('"{}"'.format(formatter(x)) for x in xs)
-
-    request_urls = fmt(methods, get_create_path_for)
-    type_urls = fmt(types, get_path_for_type)
-    constructor_urls = fmt(constructors, get_create_path_for)
-
-    shutil.copy('../res/404.html', original_paths['404'])
-    copy_replace('../res/core.html', original_paths['index_all'], {
+    shutil.copy(os.path.join(input_res, '404.html'), original_paths['404'])
+    _copy_replace(os.path.join(input_res, 'core.html'),
+                  original_paths['index_all'], {
         '{type_count}': len(types),
         '{method_count}': len(methods),
         '{constructor_count}': len(tlobjects) - len(methods),
         '{layer}': layer,
     })
+
+    def fmt(xs):
+        zs = {}  # create a dict to hold those which have duplicated keys
+        for x in xs:
+            zs[x.class_name] = x.class_name in zs
+        return ', '.join(
+            '"{}.{}"'.format(x.namespace, x.class_name)
+            if zs[x.class_name] and x.namespace
+            else '"{}"'.format(x.class_name) for x in xs
+        )
+
+    request_names = fmt(methods)
+    constructor_names = fmt(cs)
+
+    def fmt(xs, formatter):
+        return ', '.join('"{}"'.format(formatter(x)) for x in xs)
+
+    type_names = fmt(types, formatter=lambda x: x)
+
+    # Local URLs shouldn't rely on the output's root, so set empty root
+    create_path_for = functools.partial(_get_create_path_for, '', make=False)
+    path_for_type = functools.partial(_get_path_for_type, '')
+    request_urls = fmt(methods, create_path_for)
+    type_urls = fmt(types, path_for_type)
+    constructor_urls = fmt(cs, create_path_for)
+
     os.makedirs(os.path.abspath(os.path.join(
         original_paths['search.js'], os.path.pardir
     )), exist_ok=True)
-    copy_replace('../res/js/search.js', original_paths['search.js'], {
+    _copy_replace(os.path.join(input_res, 'js', 'search.js'),
+                  original_paths['search.js'], {
         '{request_names}': request_names,
         '{type_names}': type_names,
         '{constructor_names}': constructor_names,
@@ -585,23 +577,17 @@ def generate_documentation(scheme_file):
         '{constructor_urls}': constructor_urls
     })
 
-    # Everything done
-    print('Documentation generated.')
+
+def _copy_resources(res_dir, out_dir):
+    for dirname, files in [('css', ['docs.light.css', 'docs.dark.css']),
+                           ('img', ['arrow.svg'])]:
+        dirpath = os.path.join(out_dir, dirname)
+        os.makedirs(dirpath, exist_ok=True)
+        for file in files:
+            shutil.copy(os.path.join(res_dir, dirname, file), dirpath)
 
 
-def copy_resources():
-    for d in ('css', 'img'):
-        os.makedirs(d, exist_ok=True)
-
-    shutil.copy('../res/img/arrow.svg', 'img')
-    shutil.copy('../res/css/docs.css', 'css')
-
-
-if __name__ == '__main__':
-    os.makedirs('generated', exist_ok=True)
-    os.chdir('generated')
-    try:
-        generate_documentation('../../telethon_generator/scheme.tl')
-        copy_resources()
-    finally:
-        os.chdir(os.pardir)
+def generate_docs(tlobjects, errors, layer, input_res, output_dir):
+    os.makedirs(output_dir, exist_ok=True)
+    _write_html_pages(tlobjects, errors, layer, input_res, output_dir)
+    _copy_resources(input_res, output_dir)

telethon_generator/generators/errors.py

@@ -0,0 +1,52 @@
+import itertools
+
+
+def generate_errors(errors, f):
+    # Exact/regex match to create {CODE: ErrorClassName}
+    exact_match = []
+    regex_match = []
+
+    # Find out what subclasses to import and which to create
+    import_base, create_base = set(), {}
+    for error in errors:
+        if error.subclass_exists:
+            import_base.add(error.subclass)
+        else:
+            create_base[error.subclass] = error.int_code
+
+        if error.has_captures:
+            regex_match.append(error)
+        else:
+            exact_match.append(error)
+
+    # Imports and new subclass creation
+    f.write('from .rpc_base_errors import RPCError, {}\n'
+            .format(", ".join(sorted(import_base))))
+
+    for cls, int_code in sorted(create_base.items(), key=lambda t: t[1]):
+        f.write('\n\nclass {}(RPCError):\n    code = {}\n'
+                .format(cls, int_code))
+
+    # Error classes generation
+    for error in errors:
+        f.write('\n\nclass {}({}):\n    def __init__(self, **kwargs):\n'
+                '        '.format(error.name, error.subclass))
+
+        if error.has_captures:
+            f.write("self.{} = int(kwargs.get('capture', 0))\n        "
+                    .format(error.capture_name))
+
+        f.write('super(Exception, self).__init__({}'
+                .format(repr(error.description)))
+
+        if error.has_captures:
+            f.write('.format(self.{})'.format(error.capture_name))
+
+        f.write(')\n')
+
+    # Create the actual {CODE: ErrorClassName} dict once classes are defined
+    # TODO Actually make a difference between regex/exact
+    f.write('\n\nrpc_errors_all = {\n')
+    for error in itertools.chain(regex_match, exact_match):
+        f.write('    {}: {},\n'.format(repr(error.pattern), error.name))
+    f.write('}\n')

telethon_generator/generators/tlobject.py

@@ -0,0 +1,668 @@
+import functools
+import os
+import re
+import shutil
+import struct
+from collections import defaultdict
+from zlib import crc32
+
+from ..source_builder import SourceBuilder
+from ..utils import snake_to_camel_case
+
+AUTO_GEN_NOTICE = \
+    '"""File generated by TLObjects\' generator. All changes will be ERASED"""'
+
+
+AUTO_CASTS = {
+    'InputPeer': 'utils.get_input_peer(client.get_input_entity({}))',
+    'InputChannel': 'utils.get_input_channel(client.get_input_entity({}))',
+    'InputUser': 'utils.get_input_user(client.get_input_entity({}))',
+    'InputDialogPeer': 'utils.get_input_dialog(client.get_input_entity({}))',
+    'InputMedia': 'utils.get_input_media({})',
+    'InputPhoto': 'utils.get_input_photo({})',
+    'InputMessage': 'utils.get_input_message({})'
+}
+
+BASE_TYPES = ('string', 'bytes', 'int', 'long', 'int128',
+              'int256', 'double', 'Bool', 'true', 'date')
+
+
+def _write_modules(out_dir, depth, namespace_tlobjects, type_constructors):
+    # namespace_tlobjects: {'namespace', [TLObject]}
+    os.makedirs(out_dir, exist_ok=True)
+    for ns, tlobjects in namespace_tlobjects.items():
+        file = os.path.join(out_dir, '{}.py'.format(ns or '__init__'))
+        with open(file, 'w', encoding='utf-8') as f,\
+                SourceBuilder(f) as builder:
+            builder.writeln(AUTO_GEN_NOTICE)
+
+            builder.writeln('from {}.tl.tlobject import TLObject', '.' * depth)
+            builder.writeln('from typing import Optional, List, '
+                            'Union, TYPE_CHECKING')
+
+            # Add the relative imports to the namespaces,
+            # unless we already are in a namespace.
+            if not ns:
+                builder.writeln('from . import {}', ', '.join(
+                    x for x in namespace_tlobjects.keys() if x
+                ))
+
+            # Import 'os' for those needing access to 'os.urandom()'
+            # Currently only 'random_id' needs 'os' to be imported,
+            # for all those TLObjects with arg.can_be_inferred.
+            builder.writeln('import os')
+
+            # Import struct for the .__bytes__(self) serialization
+            builder.writeln('import struct')
+
+            tlobjects.sort(key=lambda x: x.name)
+
+            type_names = set()
+            type_defs = []
+
+            # Find all the types in this file and generate type definitions
+            # based on the types. The type definitions are written to the
+            # file at the end.
+            for t in tlobjects:
+                if not t.is_function:
+                    type_name = t.result
+                    if '.' in type_name:
+                        type_name = type_name[type_name.rindex('.'):]
+                    if type_name in type_names:
+                        continue
+                    type_names.add(type_name)
+                    constructors = type_constructors[type_name]
+                    if not constructors:
+                        pass
+                    elif len(constructors) == 1:
+                        type_defs.append('Type{} = {}'.format(
+                            type_name, constructors[0].class_name))
+                    else:
+                        type_defs.append('Type{} = Union[{}]'.format(
+                            type_name, ','.join(c.class_name
+                                                for c in constructors)))
+
+            imports = {}
+            primitives = ('int', 'long', 'int128', 'int256', 'string',
+                          'date', 'bytes', 'true')
+            # Find all the types in other files that are used in this file
+            # and generate the information required to import those types.
+            for t in tlobjects:
+                for arg in t.args:
+                    name = arg.type
+                    if not name or name in primitives:
+                        continue
+
+                    import_space = '{}.tl.types'.format('.' * depth)
+                    if '.' in name:
+                        namespace = name.split('.')[0]
+                        name = name.split('.')[1]
+                        import_space += '.{}'.format(namespace)
+
+                    if name not in type_names:
+                        type_names.add(name)
+                        if name == 'date':
+                            imports['datetime'] = ['datetime']
+                            continue
+                        elif import_space not in imports:
+                            imports[import_space] = set()
+                        imports[import_space].add('Type{}'.format(name))
+
+            # Add imports required for type checking
+            if imports:
+                builder.writeln('if TYPE_CHECKING:')
+                for namespace, names in imports.items():
+                    builder.writeln('from {} import {}',
+                                    namespace, ', '.join(names))
+
+                builder.end_block()
+
+            # Generate the class for every TLObject
+            for t in tlobjects:
+                _write_source_code(t, builder, type_constructors)
+                builder.current_indent = 0
+
+            # Write the type definitions generated earlier.
+            builder.writeln('')
+            for line in type_defs:
+                builder.writeln(line)
+
+
+def _write_source_code(tlobject, builder, type_constructors):
+    """
+    Writes the source code corresponding to the given TLObject
+    by making use of the ``builder`` `SourceBuilder`.
+
+    Additional information such as file path depth and
+    the ``Type: [Constructors]`` must be given for proper
+    importing and documentation strings.
+    """
+    _write_class_init(tlobject, type_constructors, builder)
+    _write_resolve(tlobject, builder)
+    _write_to_dict(tlobject, builder)
+    _write_to_bytes(tlobject, builder)
+    _write_from_reader(tlobject, builder)
+    _write_on_response(tlobject, builder)
+
+
+def _write_class_init(tlobject, type_constructors, builder):
+    builder.writeln()
+    builder.writeln()
+    builder.writeln('class {}(TLObject):', tlobject.class_name)
+
+    # Class-level variable to store its Telegram's constructor ID
+    builder.writeln('CONSTRUCTOR_ID = {:#x}', tlobject.id)
+    builder.writeln('SUBCLASS_OF_ID = {:#x}',
+                    crc32(tlobject.result.encode('ascii')))
+    builder.writeln()
+
+    # Convert the args to string parameters, flags having =None
+    args = [(a.name if not a.is_flag and not a.can_be_inferred
+             else '{}=None'.format(a.name)) for a in tlobject.real_args]
+
+    # Write the __init__ function
+    builder.writeln('def __init__({}):', ', '.join(['self'] + args))
+    if tlobject.real_args:
+        # Write the docstring, to know the type of the args
+        builder.writeln('"""')
+        for arg in tlobject.real_args:
+            if not arg.flag_indicator:
+                builder.writeln(':param {} {}:', arg.type_hint(), arg.name)
+                builder.current_indent -= 1  # It will auto-indent (':')
+
+        # We also want to know what type this request returns
+        # or to which type this constructor belongs to
+        builder.writeln()
+        if tlobject.is_function:
+            builder.write(':returns {}: ', tlobject.result)
+        else:
+            builder.write('Constructor for {}: ', tlobject.result)
+
+        constructors = type_constructors[tlobject.result]
+        if not constructors:
+            builder.writeln('This type has no constructors.')
+        elif len(constructors) == 1:
+            builder.writeln('Instance of {}.',
+                            constructors[0].class_name)
+        else:
+            builder.writeln('Instance of either {}.', ', '.join(
+                c.class_name for c in constructors))
+
+        builder.writeln('"""')
+
+    builder.writeln('super().__init__()')
+    # Functions have a result object and are confirmed by default
+    if tlobject.is_function:
+        builder.writeln('self.result = None')
+        builder.writeln('self.content_related = True')
+
+    # Set the arguments
+    if tlobject.real_args:
+        builder.writeln()
+
+    for arg in tlobject.real_args:
+        if not arg.can_be_inferred:
+            builder.writeln('self.{0} = {0}  # type: {1}',
+                            arg.name, arg.type_hint())
+
+        # Currently the only argument that can be
+        # inferred are those called 'random_id'
+        elif arg.name == 'random_id':
+            # Endianness doesn't really matter, and 'big' is shorter
+            code = "int.from_bytes(os.urandom({}), 'big', signed=True)" \
+                .format(8 if arg.type == 'long' else 4)
+
+            if arg.is_vector:
+                # Currently for the case of "messages.forwardMessages"
+                # Ensure we can infer the length from id:Vector<>
+                if not next(a for a in tlobject.real_args
+                            if a.name == 'id').is_vector:
+                    raise ValueError(
+                        'Cannot infer list of random ids for ', tlobject
+                    )
+                code = '[{} for _ in range(len(id))]'.format(code)
+
+            builder.writeln(
+                "self.random_id = random_id if random_id "
+                "is not None else {}", code
+            )
+        else:
+            raise ValueError('Cannot infer a value for ', arg)
+
+    builder.end_block()
+
+
+def _write_resolve(tlobject, builder):
+    if any(arg.type in AUTO_CASTS for arg in tlobject.real_args):
+        builder.writeln('def resolve(self, client, utils):')
+        for arg in tlobject.real_args:
+            ac = AUTO_CASTS.get(arg.type, None)
+            if not ac:
+                continue
+            if arg.is_vector:
+                builder.write('self.{0} = [{1} for _x in self.{0}]',
+                              arg.name, ac.format('_x'))
+            else:
+                builder.write('self.{} = {}', arg.name,
+                              ac.format('self.' + arg.name))
+            builder.writeln(' if self.{} else None'.format(arg.name)
+                            if arg.is_flag else '')
+        builder.end_block()
+
+
+def _write_to_dict(tlobject, builder):
+    builder.writeln('def to_dict(self):')
+    builder.writeln('return {')
+    builder.current_indent += 1
+
+    builder.write("'_': '{}'", tlobject.class_name)
+    for arg in tlobject.real_args:
+        builder.writeln(',')
+        builder.write("'{}': ", arg.name)
+        if arg.type in BASE_TYPES:
+            if arg.is_vector:
+                builder.write('[] if self.{0} is None else self.{0}[:]',
+                              arg.name)
+            else:
+                builder.write('self.{}', arg.name)
+        else:
+            if arg.is_vector:
+                builder.write(
+                    '[] if self.{0} is None else [None '
+                    'if x is None else x.to_dict() for x in self.{0}]',
+                    arg.name
+                )
+            else:
+                builder.write(
+                    'None if self.{0} is None else self.{0}.to_dict()',
+                    arg.name
+                )
+
+    builder.writeln()
+    builder.current_indent -= 1
+    builder.writeln("}")
+
+    builder.end_block()
+
+
+def _write_to_bytes(tlobject, builder):
+    builder.writeln('def __bytes__(self):')
+
+    # Some objects require more than one flag parameter to be set
+    # at the same time. In this case, add an assertion.
+    repeated_args = defaultdict(list)
+    for arg in tlobject.args:
+        if arg.is_flag:
+            repeated_args[arg.flag_index].append(arg)
+
+    for ra in repeated_args.values():
+        if len(ra) > 1:
+            cnd1 = ('(self.{0} or self.{0} is not None)'
+                    .format(a.name) for a in ra)
+            cnd2 = ('(self.{0} is None or self.{0} is False)'
+                    .format(a.name) for a in ra)
+            builder.writeln(
+                "assert ({}) or ({}), '{} parameters must all "
+                "be False-y (like None) or all me True-y'",
+                ' and '.join(cnd1), ' and '.join(cnd2),
+                ', '.join(a.name for a in ra)
+            )
+
+    builder.writeln("return b''.join((")
+    builder.current_indent += 1
+
+    # First constructor code, we already know its bytes
+    builder.writeln('{},', repr(struct.pack('<I', tlobject.id)))
+
+    for arg in tlobject.args:
+        if _write_arg_to_bytes(builder, arg, tlobject.args):
+            builder.writeln(',')
+
+    builder.current_indent -= 1
+    builder.writeln('))')
+    builder.end_block()
+
+
+def _write_from_reader(tlobject, builder):
+    builder.writeln('@classmethod')
+    builder.writeln('def from_reader(cls, reader):')
+    for arg in tlobject.args:
+        _write_arg_read_code(builder, arg, tlobject.args, name='_' + arg.name)
+
+    builder.writeln('return cls({})', ', '.join(
+        '{0}=_{0}'.format(a.name) for a in tlobject.real_args))
+
+
+def _write_on_response(tlobject, builder):
+    # Only requests can have a different response that's not their
+    # serialized body, that is, we'll be setting their .result.
+    #
+    # The default behaviour is reading a TLObject too, so no need
+    # to override it unless necessary.
+    if not tlobject.is_function:
+        return
+
+    # https://core.telegram.org/mtproto/serialize#boxed-and-bare-types
+    # TL;DR; boxed types start with uppercase always, so we can use
+    # this to check whether everything in it is boxed or not.
+    #
+    # Currently only un-boxed responses are Vector<int>/Vector<long>.
+    # If this weren't the case, we should check upper case after
+    # max(index('<'), index('.')) (and if it is, it's boxed, so return).
+    m = re.match(r'Vector<(int|long)>', tlobject.result)
+    if not m:
+        return
+
+    builder.end_block()
+    builder.writeln('def on_response(self, reader):')
+    builder.writeln('reader.read_int()  # Vector ID')
+    builder.writeln('self.result = [reader.read_{}() '
+                    'for _ in range(reader.read_int())]', m.group(1))
+
+
+def _write_arg_to_bytes(builder, arg, args, name=None):
+    """
+    Writes the .__bytes__() code for the given argument
+    :param builder: The source code builder
+    :param arg: The argument to write
+    :param args: All the other arguments in TLObject same __bytes__.
+                 This is required to determine the flags value
+    :param name: The name of the argument. Defaults to "self.argname"
+                 This argument is an option because it's required when
+                 writing Vectors<>
+    """
+    if arg.generic_definition:
+        return  # Do nothing, this only specifies a later type
+
+    if name is None:
+        name = 'self.{}'.format(arg.name)
+
+    # The argument may be a flag, only write if it's not None AND
+    # if it's not a True type.
+    # True types are not actually sent, but instead only used to
+    # determine the flags.
+    if arg.is_flag:
+        if arg.type == 'true':
+            return  # Exit, since True type is never written
+        elif arg.is_vector:
+            # Vector flags are special since they consist of 3 values,
+            # so we need an extra join here. Note that empty vector flags
+            # should NOT be sent either!
+            builder.write("b'' if {0} is None or {0} is False "
+                          "else b''.join((", name)
+        else:
+            builder.write("b'' if {0} is None or {0} is False "
+                          "else (", name)
+
+    if arg.is_vector:
+        if arg.use_vector_id:
+            # vector code, unsigned 0x1cb5c415 as little endian
+            builder.write(r"b'\x15\xc4\xb5\x1c',")
+
+        builder.write("struct.pack('<i', len({})),", name)
+
+        # Cannot unpack the values for the outer tuple through *[(
+        # since that's a Python >3.5 feature, so add another join.
+        builder.write("b''.join(")
+
+        # Temporary disable .is_vector, not to enter this if again
+        # Also disable .is_flag since it's not needed per element
+        old_flag = arg.is_flag
+        arg.is_vector = arg.is_flag = False
+        _write_arg_to_bytes(builder, arg, args, name='x')
+        arg.is_vector = True
+        arg.is_flag = old_flag
+
+        builder.write(' for x in {})', name)
+
+    elif arg.flag_indicator:
+        # Calculate the flags with those items which are not None
+        if not any(f.is_flag for f in args):
+            # There's a flag indicator, but no flag arguments so it's 0
+            builder.write(r"b'\0\0\0\0'")
+        else:
+            builder.write("struct.pack('<I', ")
+            builder.write(
+                ' | '.join('(0 if {0} is None or {0} is False else {1})'
+                           .format('self.{}'.format(flag.name),
+                                   1 << flag.flag_index)
+                           for flag in args if flag.is_flag)
+            )
+            builder.write(')')
+
+    elif 'int' == arg.type:
+        # struct.pack is around 4 times faster than int.to_bytes
+        builder.write("struct.pack('<i', {})", name)
+
+    elif 'long' == arg.type:
+        builder.write("struct.pack('<q', {})", name)
+
+    elif 'int128' == arg.type:
+        builder.write("{}.to_bytes(16, 'little', signed=True)", name)
+
+    elif 'int256' == arg.type:
+        builder.write("{}.to_bytes(32, 'little', signed=True)", name)
+
+    elif 'double' == arg.type:
+        builder.write("struct.pack('<d', {})", name)
+
+    elif 'string' == arg.type:
+        builder.write('TLObject.serialize_bytes({})', name)
+
+    elif 'Bool' == arg.type:
+        # 0x997275b5 if boolean else 0xbc799737
+        builder.write(r"b'\xb5ur\x99' if {} else b'7\x97y\xbc'", name)
+
+    elif 'true' == arg.type:
+        pass  # These are actually NOT written! Only used for flags
+
+    elif 'bytes' == arg.type:
+        builder.write('TLObject.serialize_bytes({})', name)
+
+    elif 'date' == arg.type:  # Custom format
+        builder.write('TLObject.serialize_datetime({})', name)
+
+    else:
+        # Else it may be a custom type
+        builder.write('bytes({})', name)
+
+        # If the type is not boxed (i.e. starts with lowercase) we should
+        # not serialize the constructor ID (so remove its first 4 bytes).
+        boxed = arg.type[arg.type.find('.') + 1].isupper()
+        if not boxed:
+            builder.write('[4:]')
+
+    if arg.is_flag:
+        builder.write(')')
+        if arg.is_vector:
+            builder.write(')')  # We were using a tuple
+
+    return True  # Something was written
+
+
+def _write_arg_read_code(builder, arg, args, name):
+    """
+    Writes the read code for the given argument, setting the
+    arg.name variable to its read value.
+
+    :param builder: The source code builder
+    :param arg: The argument to write
+    :param args: All the other arguments in TLObject same on_send.
+                 This is required to determine the flags value
+    :param name: The name of the argument. Defaults to "self.argname"
+                 This argument is an option because it's required when
+                 writing Vectors<>
+    """
+
+    if arg.generic_definition:
+        return  # Do nothing, this only specifies a later type
+
+    # The argument may be a flag, only write that flag was given!
+    was_flag = False
+    if arg.is_flag:
+        # Treat 'true' flags as a special case, since they're true if
+        # they're set, and nothing else needs to actually be read.
+        if 'true' == arg.type:
+            builder.writeln('{} = bool(flags & {})',
+                            name, 1 << arg.flag_index)
+            return
+
+        was_flag = True
+        builder.writeln('if flags & {}:', 1 << arg.flag_index)
+        # Temporary disable .is_flag not to enter this if
+        # again when calling the method recursively
+        arg.is_flag = False
+
+    if arg.is_vector:
+        if arg.use_vector_id:
+            # We have to read the vector's constructor ID
+            builder.writeln("reader.read_int()")
+
+        builder.writeln('{} = []', name)
+        builder.writeln('for _ in range(reader.read_int()):')
+        # Temporary disable .is_vector, not to enter this if again
+        arg.is_vector = False
+        _write_arg_read_code(builder, arg, args, name='_x')
+        builder.writeln('{}.append(_x)', name)
+        arg.is_vector = True
+
+    elif arg.flag_indicator:
+        # Read the flags, which will indicate what items we should read next
+        builder.writeln('flags = reader.read_int()')
+        builder.writeln()
+
+    elif 'int' == arg.type:
+        builder.writeln('{} = reader.read_int()', name)
+
+    elif 'long' == arg.type:
+        builder.writeln('{} = reader.read_long()', name)
+
+    elif 'int128' == arg.type:
+        builder.writeln('{} = reader.read_large_int(bits=128)', name)
+
+    elif 'int256' == arg.type:
+        builder.writeln('{} = reader.read_large_int(bits=256)', name)
+
+    elif 'double' == arg.type:
+        builder.writeln('{} = reader.read_double()', name)
+
+    elif 'string' == arg.type:
+        builder.writeln('{} = reader.tgread_string()', name)
+
+    elif 'Bool' == arg.type:
+        builder.writeln('{} = reader.tgread_bool()', name)
+
+    elif 'true' == arg.type:
+        # Arbitrary not-None value, don't actually read "true" flags
+        builder.writeln('{} = True', name)
+
+    elif 'bytes' == arg.type:
+        builder.writeln('{} = reader.tgread_bytes()', name)
+
+    elif 'date' == arg.type:  # Custom format
+        builder.writeln('{} = reader.tgread_date()', name)
+
+    else:
+        # Else it may be a custom type
+        if not arg.skip_constructor_id:
+            builder.writeln('{} = reader.tgread_object()', name)
+        else:
+            # Import the correct type inline to avoid cyclic imports.
+            # There may be better solutions so that we can just access
+            # all the types before the files have been parsed, but I
+            # don't know of any.
+            sep_index = arg.type.find('.')
+            if sep_index == -1:
+                ns, t = '.', arg.type
+            else:
+                ns, t = '.' + arg.type[:sep_index], arg.type[sep_index+1:]
+            class_name = snake_to_camel_case(t)
+
+            # There would be no need to import the type if we're in the
+            # file with the same namespace, but since it does no harm
+            # and we don't have information about such thing in the
+            # method we just ignore that case.
+            builder.writeln('from {} import {}', ns, class_name)
+            builder.writeln('{} = {}.from_reader(reader)',
+                            name, class_name)
+
+    # End vector and flag blocks if required (if we opened them before)
+    if arg.is_vector:
+        builder.end_block()
+
+    if was_flag:
+        builder.current_indent -= 1
+        builder.writeln('else:')
+        builder.writeln('{} = None', name)
+        builder.current_indent -= 1
+        # Restore .is_flag
+        arg.is_flag = True
+
+
+def _write_all_tlobjects(tlobjects, layer, builder):
+    builder.writeln(AUTO_GEN_NOTICE)
+    builder.writeln()
+
+    builder.writeln('from . import types, functions')
+    builder.writeln()
+
+    # Create a constant variable to indicate which layer this is
+    builder.writeln('LAYER = {}', layer)
+    builder.writeln()
+
+    # Then create the dictionary containing constructor_id: class
+    builder.writeln('tlobjects = {')
+    builder.current_indent += 1
+
+    # Fill the dictionary (0x1a2b3c4f: tl.full.type.path.Class)
+    for tlobject in tlobjects:
+        builder.write('{:#010x}: ', tlobject.id)
+        builder.write('functions' if tlobject.is_function else 'types')
+        if tlobject.namespace:
+            builder.write('.' + tlobject.namespace)
+
+        builder.writeln('.{},', tlobject.class_name)
+
+    builder.current_indent -= 1
+    builder.writeln('}')
+
+
+def generate_tlobjects(tlobjects, layer, import_depth, output_dir):
+    get_file = functools.partial(os.path.join, output_dir)
+    os.makedirs(get_file('functions'), exist_ok=True)
+    os.makedirs(get_file('types'), exist_ok=True)
+
+    # Group everything by {namespace: [tlobjects]} to generate __init__.py
+    namespace_functions = defaultdict(list)
+    namespace_types = defaultdict(list)
+
+    # Group {type: [constructors]} to generate the documentation
+    type_constructors = defaultdict(list)
+    for tlobject in tlobjects:
+        if tlobject.is_function:
+            namespace_functions[tlobject.namespace].append(tlobject)
+        else:
+            namespace_types[tlobject.namespace].append(tlobject)
+            type_constructors[tlobject.result].append(tlobject)
+
+    _write_modules(get_file('functions'), import_depth,
+                   namespace_functions, type_constructors)
+    _write_modules(get_file('types'), import_depth,
+                   namespace_types, type_constructors)
+
+    filename = os.path.join(get_file('all_tlobjects.py'))
+    with open(filename, 'w', encoding='utf-8') as file:
+        with SourceBuilder(file) as builder:
+            _write_all_tlobjects(tlobjects, layer, builder)
+
+
+def clean_tlobjects(output_dir):
+    get_file = functools.partial(os.path.join, output_dir)
+    for d in ('functions', 'types'):
+        d = get_file(d)
+        if os.path.isdir(d):
+            shutil.rmtree(d)
+
+    tl = get_file('all_tlobjects.py')
+    if os.path.isfile(tl):
+        os.remove(tl)

telethon_generator/parser/__init__.py

@@ -1,3 +0,0 @@
-from .source_builder import SourceBuilder
-from .tl_parser import TLParser
-from .tl_object import TLObject

telethon_generator/parser/tl_object.py

@@ -1,323 +0,0 @@
-import re
-from zlib import crc32
-
-
-class TLObject:
-    """.tl core types IDs (such as vector, booleans, etc.)"""
-    CORE_TYPES = (
-        0xbc799737,  # boolFalse#bc799737 = Bool;
-        0x997275b5,  # boolTrue#997275b5 = Bool;
-        0x3fedd339,  # true#3fedd339 = True;
-        0x1cb5c415,  # vector#1cb5c415 {t:Type} # [ t ] = Vector t;
-    )
-
-    def __init__(self, fullname, object_id, args, result, is_function):
-        """
-        Initializes a new TLObject, given its properties.
-        Usually, this will be called from `from_tl` instead
-        :param fullname: The fullname of the TL object (namespace.name)
-                         The namespace can be omitted
-        :param object_id: The hexadecimal string representing the object ID
-        :param args: The arguments, if any, of the TL object
-        :param result: The result type of the TL object
-        :param is_function: Is the object a function or a type?
-        """
-        # The name can or not have a namespace
-        if '.' in fullname:
-            self.namespace = fullname.split('.')[0]
-            self.name = fullname.split('.')[1]
-        else:
-            self.namespace = None
-            self.name = fullname
-
-        self.args = args
-        self.result = result
-        self.is_function = is_function
-
-        # The ID should be an hexadecimal string or None to be inferred
-        if object_id is None:
-            self.id = self.infer_id()
-        else:
-            self.id = int(object_id, base=16)
-            assert self.id == self.infer_id(),\
-                'Invalid inferred ID for ' + repr(self)
-
-    @staticmethod
-    def from_tl(tl, is_function):
-        """Returns a TL object from the given TL scheme line"""
-
-        # Regex to match the whole line
-        match = re.match(r'''
-            ^                  # We want to match from the beginning to the end
-            ([\w.]+)           # The .tl object can contain alpha_name or namespace.alpha_name
-            (?:
-                \#             # After the name, comes the ID of the object
-                ([0-9a-f]+)    # The constructor ID is in hexadecimal form
-            )?                 # If no constructor ID was given, CRC32 the 'tl' to determine it
-
-            (?:\s              # After that, we want to match its arguments (name:type)
-                {?             # For handling the start of the '{X:Type}' case
-                \w+            # The argument name will always be an alpha-only name
-                :              # Then comes the separator between name:type
-                [\w\d<>#.?!]+  # The type is slightly more complex, since it's alphanumeric and it can
-                               # also have Vector<type>, flags:# and flags.0?default, plus :!X as type
-                }?             # For handling the end of the '{X:Type}' case
-            )*                 # Match 0 or more arguments
-            \s                 # Leave a space between the arguments and the equal
-            =
-            \s                 # Leave another space between the equal and the result
-            ([\w\d<>#.?]+)     # The result can again be as complex as any argument type
-            ;$                 # Finally, the line should always end with ;
-            ''', tl, re.IGNORECASE | re.VERBOSE)
-
-        if match is None:
-            # Probably "vector#1cb5c415 {t:Type} # [ t ] = Vector t;"
-            raise ValueError('Cannot parse TLObject', tl)
-
-        # Sub-regex to match the arguments (sadly, it cannot be embedded in the first regex)
-        args_match = re.findall(r'''
-            ({)?             # We may or may not capture the opening brace
-            (\w+)            # First we capture any alpha name with length 1 or more
-            :                # Which is separated from its type by a colon
-            ([\w\d<>#.?!]+)  # The type is slightly more complex, since it's alphanumeric and it can
-                             # also have Vector<type>, flags:# and flags.0?default, plus :!X as type
-            (})?             # We may or not capture the closing brace
-            ''', tl, re.IGNORECASE | re.VERBOSE)
-
-        # Retrieve the matched arguments
-        args = [TLArg(name, arg_type, brace != '')
-                for brace, name, arg_type, _ in args_match]
-
-        # And initialize the TLObject
-        return TLObject(
-            fullname=match.group(1),
-            object_id=match.group(2),
-            args=args,
-            result=match.group(3),
-            is_function=is_function)
-
-    def class_name(self):
-        """Gets the class name following the Python style guidelines"""
-        return self.class_name_for(self.name, self.is_function)
-
-    @staticmethod
-    def class_name_for(typename, is_function=False):
-        """Gets the class name following the Python style guidelines"""
-        # Courtesy of http://stackoverflow.com/a/31531797/4759433
-        result = re.sub(r'_([a-z])', lambda m: m.group(1).upper(), typename)
-        result = result[:1].upper() + result[1:].replace('_', '')
-        # If it's a function, let it end with "Request" to identify them
-        if is_function:
-            result += 'Request'
-        return result
-
-    def sorted_args(self):
-        """Returns the arguments properly sorted and ready to plug-in
-           into a Python's method header (i.e., flags and those which
-           can be inferred will go last so they can default =None)
-        """
-        return sorted(self.args,
-                      key=lambda x: x.is_flag or x.can_be_inferred)
-
-    def is_core_type(self):
-        """Determines whether the TLObject is a "core type"
-           (and thus should be embedded in the generated code) or not"""
-        return self.id in TLObject.CORE_TYPES
-
-    def __repr__(self, ignore_id=False):
-        fullname = ('{}.{}'.format(self.namespace, self.name)
-                    if self.namespace is not None else self.name)
-
-        if getattr(self, 'id', None) is None or ignore_id:
-            hex_id = ''
-        else:
-            # Skip 0x and add 0's for padding
-            hex_id = '#' + hex(self.id)[2:].rjust(8, '0')
-
-        if self.args:
-            args = ' ' + ' '.join([repr(arg) for arg in self.args])
-        else:
-            args = ''
-
-        return '{}{}{} = {}'.format(fullname, hex_id, args, self.result)
-
-    def infer_id(self):
-        representation = self.__repr__(ignore_id=True)
-
-        # Clean the representation
-        representation = representation\
-            .replace(':bytes ', ':string ')\
-            .replace('?bytes ', '?string ')\
-            .replace('<', ' ').replace('>', '')\
-            .replace('{', '').replace('}', '')
-
-        representation = re.sub(
-            r' \w+:flags\.\d+\?true',
-            r'',
-            representation
-        )
-        return crc32(representation.encode('ascii'))
-
-    def __str__(self):
-        fullname = ('{}.{}'.format(self.namespace, self.name)
-                    if self.namespace is not None else self.name)
-
-        # Some arguments are not valid for being represented, such as the flag indicator or generic definition
-        # (these have no explicit values until used)
-        valid_args = [arg for arg in self.args
-                      if not arg.flag_indicator and not arg.generic_definition]
-
-        args = ', '.join(['{}={{}}'.format(arg.name) for arg in valid_args])
-
-        # Since Python's default representation for lists is using repr(), we need to str() manually on every item
-        args_format = ', '.join(
-            ['str(self.{})'.format(arg.name) if not arg.is_vector else
-             'None if not self.{0} else [str(_) for _ in self.{0}]'.format(
-                 arg.name) for arg in valid_args])
-
-        return ("'({} (ID: {}) = ({}))'.format({})"
-                .format(fullname, hex(self.id), args, args_format))
-
-
-class TLArg:
-    def __init__(self, name, arg_type, generic_definition):
-        """
-        Initializes a new .tl argument
-        :param name: The name of the .tl argument
-        :param arg_type: The type of the .tl argument
-        :param generic_definition: Is the argument a generic definition?
-                                   (i.e. {X:Type})
-        """
-        if name == 'self':  # This very only name is restricted
-            self.name = 'is_self'
-        else:
-            self.name = name
-
-        # Default values
-        self.is_vector = False
-        self.is_flag = False
-        self.skip_constructor_id = False
-        self.flag_index = -1
-
-        # Special case: some types can be inferred, which makes it
-        # less annoying to type. Currently the only type that can
-        # be inferred is if the name is 'random_id', to which a
-        # random ID will be assigned if left as None (the default)
-        self.can_be_inferred = name == 'random_id'
-
-        # The type can be an indicator that other arguments will be flags
-        if arg_type == '#':
-            self.flag_indicator = True
-            self.type = None
-            self.is_generic = False
-        else:
-            self.flag_indicator = False
-            self.is_generic = arg_type.startswith('!')
-            # Strip the exclamation mark always to have only the name
-            self.type = arg_type.lstrip('!')
-
-            # The type may be a flag (flags.IDX?REAL_TYPE)
-            # Note that 'flags' is NOT the flags name; this is determined by a previous argument
-            # However, we assume that the argument will always be called 'flags'
-            flag_match = re.match(r'flags.(\d+)\?([\w<>.]+)', self.type)
-            if flag_match:
-                self.is_flag = True
-                self.flag_index = int(flag_match.group(1))
-                # Update the type to match the exact type, not the "flagged" one
-                self.type = flag_match.group(2)
-
-            # Then check if the type is a Vector<REAL_TYPE>
-            vector_match = re.match(r'[Vv]ector<([\w\d.]+)>', self.type)
-            if vector_match:
-                self.is_vector = True
-
-                # If the type's first letter is not uppercase, then
-                # it is a constructor and we use (read/write) its ID
-                # as pinpointed on issue #81.
-                self.use_vector_id = self.type[0] == 'V'
-
-                # Update the type to match the one inside the vector
-                self.type = vector_match.group(1)
-
-            # See use_vector_id. An example of such case is ipPort in
-            # help.configSpecial
-            if self.type.split('.')[-1][0].islower():
-                self.skip_constructor_id = True
-
-            # The name may contain "date" in it, if this is the case and the type is "int",
-            # we can safely assume that this should be treated as a "date" object.
-            # Note that this is not a valid Telegram object, but it's easier to work with
-            if self.type == 'int' and (
-                        re.search(r'(\b|_)date\b', name) or
-                        name in ('expires', 'expires_at', 'was_online')):
-                self.type = 'date'
-
-        self.generic_definition = generic_definition
-
-    def doc_type_hint(self):
-        result = {
-            'int': 'int',
-            'long': 'int',
-            'int128': 'int',
-            'int256': 'int',
-            'string': 'str',
-            'date': 'datetime.datetime | None',  # None date = 0 timestamp
-            'bytes': 'bytes',
-            'true': 'bool',
-        }.get(self.type, self.type)
-        if self.is_vector:
-            result = 'list[{}]'.format(result)
-        if self.is_flag and self.type != 'date':
-            result += ' | None'
-
-        return result
-
-    def python_type_hint(self):
-        type = self.type
-        if '.' in type:
-            type = type.split('.')[1]
-        result = {
-            'int': 'int',
-            'long': 'int',
-            'int128': 'int',
-            'int256': 'int',
-            'string': 'str',
-            'date': 'Optional[datetime]',  # None date = 0 timestamp
-            'bytes': 'bytes',
-            'true': 'bool',
-        }.get(type, "Type{}".format(type))
-        if self.is_vector:
-            result = 'List[{}]'.format(result)
-        if self.is_flag and type != 'date':
-            result = 'Optional[{}]'.format(result)
-
-        return result
-
-    def __str__(self):
-        # Find the real type representation by updating it as required
-        real_type = self.type
-        if self.flag_indicator:
-            real_type = '#'
-
-        if self.is_vector:
-            if self.use_vector_id:
-                real_type = 'Vector<{}>'.format(real_type)
-            else:
-                real_type = 'vector<{}>'.format(real_type)
-
-        if self.is_generic:
-            real_type = '!{}'.format(real_type)
-
-        if self.is_flag:
-            real_type = 'flags.{}?{}'.format(self.flag_index, real_type)
-
-        if self.generic_definition:
-            return '{{{}:{}}}'.format(self.name, real_type)
-        else:
-            return '{}:{}'.format(self.name, real_type)
-
-    def __repr__(self):
-        # Get rid of our special type
-        return str(self)\
-            .replace(':date', ':int')\
-            .replace('?date', '?int')

telethon_generator/parser/tl_parser.py

@@ -1,51 +0,0 @@
-import re
-
-from .tl_object import TLObject
-
-
-class TLParser:
-    """Class used to parse .tl files"""
-
-    @staticmethod
-    def parse_file(file_path, ignore_core=False):
-        """This method yields TLObjects from a given .tl file"""
-
-        with open(file_path, encoding='utf-8') as file:
-            # Start by assuming that the next found line won't
-            # be a function (and will hence be a type)
-            is_function = False
-
-            # Read all the lines from the .tl file
-            for line in file:
-                # Strip comments from the line
-                comment_index = line.find('//')
-                if comment_index != -1:
-                    line = line[:comment_index]
-
-                line = line.strip()
-                if line:
-                    # Check whether the line is a type change
-                    # (types <-> functions) or not
-                    match = re.match('---(\w+)---', line)
-                    if match:
-                        following_types = match.group(1)
-                        is_function = following_types == 'functions'
-
-                    else:
-                        try:
-                            result = TLObject.from_tl(line, is_function)
-                            if not ignore_core or not result.is_core_type():
-                                yield result
-                        except ValueError as e:
-                            if 'vector#1cb5c415' not in str(e):
-                                raise
-
-    @staticmethod
-    def find_layer(file_path):
-        """Finds the layer used on the specified scheme.tl file"""
-        layer_regex = re.compile(r'^//\s*LAYER\s*(\d+)$')
-        with open(file_path, encoding='utf-8') as file:
-            for line in file:
-                match = layer_regex.match(line)
-                if match:
-                    return int(match.group(1))

telethon_generator/parsers/__init__.py

@@ -0,0 +1,2 @@
+from .errors import Error, parse_errors
+from .tlobject import TLObject, parse_tl, find_layer

telethon_generator/parsers/errors.py

@@ -0,0 +1,146 @@
+import json
+import re
+from collections import defaultdict
+
+from ..utils import snake_to_camel_case
+
+# Core base classes depending on the integer error code
+KNOWN_BASE_CLASSES = {
+    303: 'InvalidDCError',
+    400: 'BadRequestError',
+    401: 'UnauthorizedError',
+    403: 'ForbiddenError',
+    404: 'NotFoundError',
+    406: 'AuthKeyError',
+    420: 'FloodError',
+    500: 'ServerError',
+}
+
+# The API doesn't return the code for some (vital) errors. They are
+# all assumed to be 400, except these well-known ones that aren't.
+KNOWN_CODES = {
+    'ACTIVE_USER_REQUIRED': 401,
+    'AUTH_KEY_UNREGISTERED': 401,
+    'USER_DEACTIVATED': 401
+}
+
+# Give better semantic names to some captures
+CAPTURE_NAMES = {
+    'FloodWaitError': 'seconds',
+    'FloodTestPhoneWaitError': 'seconds',
+    'FileMigrateError': 'new_dc',
+    'NetworkMigrateError': 'new_dc',
+    'PhoneMigrateError': 'new_dc',
+    'UserMigrateError': 'new_dc',
+    'FilePartMissingError': 'which'
+}
+
+
+def _get_class_name(error_code):
+    """
+    Gets the corresponding class name for the given error code,
+    this either being an integer (thus base error name) or str.
+    """
+    if isinstance(error_code, int):
+        return KNOWN_BASE_CLASSES.get(
+            error_code, 'RPCError' + str(error_code).replace('-', 'Neg')
+        )
+
+    return snake_to_camel_case(
+        error_code.replace('FIRSTNAME', 'FIRST_NAME').lower(), suffix='Error')
+
+
+class Error:
+    def __init__(self, int_code, str_code, description, caused_by):
+        # TODO Some errors have the same str_code but different int_code
+        # Should these be split into different files or doesn't really matter?
+        # Telegram isn't exactly consistent with returned errors anyway.
+        self.int_code = int_code
+        self.str_code = str_code
+        self.subclass = _get_class_name(int_code)
+        self.subclass_exists = int_code in KNOWN_BASE_CLASSES
+        self.description = description
+        self.caused_by = list(sorted(caused_by))
+
+        self.has_captures = '_X' in str_code
+        if self.has_captures:
+            self.name = _get_class_name(str_code.replace('_X', ''))
+            self.pattern = str_code.replace('_X', r'_(\d+)')
+            self.capture_name = CAPTURE_NAMES.get(self.name, 'x')
+        else:
+            self.name = _get_class_name(str_code)
+            self.pattern = str_code
+            self.capture_name = None
+
+
+def parse_errors(json_file, descriptions_file):
+    """
+    Parses the given JSON file in the following format:
+        {
+            "ok": true,
+            "human_result": {"int_code": ["descriptions"]},
+            "result": {"int_code": {"full_method_name": ["str_error"]}}
+        }
+
+    The descriptions file, which has precedence over the JSON's human_result,
+    should have the following format:
+        # comment
+        str_error=Description
+
+    The method yields `Error` instances as a result.
+    """
+    with open(json_file, encoding='utf-8') as f:
+        data = json.load(f)
+
+    errors = defaultdict(set)
+    error_to_method = defaultdict(set)
+    # PWRTelegram's API doesn't return all errors, which we do need here.
+    # Add some special known-cases manually first.
+    errors[420].update((
+        'FLOOD_WAIT_X', 'FLOOD_TEST_PHONE_WAIT_X'
+    ))
+    errors[401].update((
+        'AUTH_KEY_INVALID', 'SESSION_EXPIRED', 'SESSION_REVOKED'
+    ))
+    errors[303].update((
+        'FILE_MIGRATE_X', 'PHONE_MIGRATE_X',
+        'NETWORK_MIGRATE_X', 'USER_MIGRATE_X'
+    ))
+    for int_code, method_errors in data['result'].items():
+        for method, error_list in method_errors.items():
+            for error in error_list:
+                error = re.sub('_\d+', '_X', error).upper()
+                errors[int(int_code)].add(error)
+                error_to_method[error].add(method)
+
+    # Some errors are in the human result, but not with a code. Assume 400
+    for error in data['human_result']:
+        if error[0] != '-' and not error.isdigit():
+            error = re.sub('_\d+', '_X', error).upper()
+            if not any(error in es for es in errors.values()):
+                errors[KNOWN_CODES.get(error, 400)].add(error)
+
+    # Prefer the descriptions that are related with Telethon way of coding
+    # to those that PWRTelegram's API provides.
+    telethon_descriptions = {}
+    with open(descriptions_file, encoding='utf-8') as f:
+        for line in f:
+            line = line.strip()
+            if line and not line.startswith('#'):
+                equal = line.index('=')
+                message, description = line[:equal], line[equal + 1:]
+                telethon_descriptions[message.rstrip()] = description.lstrip()
+
+    for int_code, error_set in errors.items():
+        for str_code in sorted(error_set):
+            description = telethon_descriptions.get(
+                str_code, '\n'.join(data['human_result'].get(
+                    str_code, ['No description known']
+                ))
+            )
+            yield Error(
+                int_code=int_code,
+                str_code=str_code,
+                description=description,
+                caused_by=error_to_method[str_code]
+            )

telethon_generator/parsers/tlobject.py

@@ -0,0 +1,297 @@
+import re
+from zlib import crc32
+
+from ..utils import snake_to_camel_case
+
+CORE_TYPES = (
+    0xbc799737,  # boolFalse#bc799737 = Bool;
+    0x997275b5,  # boolTrue#997275b5 = Bool;
+    0x3fedd339,  # true#3fedd339 = True;
+    0x1cb5c415,  # vector#1cb5c415 {t:Type} # [ t ] = Vector t;
+)
+
+# https://github.com/telegramdesktop/tdesktop/blob/4bf66cb6e93f3965b40084771b595e93d0b11bcd/Telegram/SourceFiles/codegen/scheme/codegen_scheme.py#L57-L62
+WHITELISTED_MISMATCHING_IDS = {
+    # 0 represents any layer
+    0: {'ipPortSecret', 'accessPointRule', 'help.configSimple'},
+    77: {'channel'},
+    78: {'channel'},
+    79: {'channel'},
+    80: {'channel'},
+}
+
+
+class TLObject:
+    def __init__(self, fullname, object_id, args, result, is_function, layer):
+        """
+        Initializes a new TLObject, given its properties.
+
+        :param fullname: The fullname of the TL object (namespace.name)
+                         The namespace can be omitted.
+        :param object_id: The hexadecimal string representing the object ID
+        :param args: The arguments, if any, of the TL object
+        :param result: The result type of the TL object
+        :param is_function: Is the object a function or a type?
+        :param layer: The layer this TLObject belongs to.
+        """
+        # The name can or not have a namespace
+        self.fullname = fullname
+        if '.' in fullname:
+            self.namespace, self.name = fullname.split('.', maxsplit=1)
+        else:
+            self.namespace, self.name = None, fullname
+
+        self.args = args
+        self.result = result
+        self.is_function = is_function
+        self.bot_usable = None
+        self.id = None
+        if object_id is None:
+            self.id = self.infer_id()
+        else:
+            self.id = int(object_id, base=16)
+            whitelist = WHITELISTED_MISMATCHING_IDS[0] |\
+                WHITELISTED_MISMATCHING_IDS.get(layer, set())
+
+            if self.fullname not in whitelist:
+                assert self.id == self.infer_id(),\
+                    'Invalid inferred ID for ' + repr(self)
+
+        self.class_name = snake_to_camel_case(
+            self.name, suffix='Request' if self.is_function else '')
+
+        self.real_args = list(a for a in self.sorted_args() if not
+                              (a.flag_indicator or a.generic_definition))
+
+    def sorted_args(self):
+        """Returns the arguments properly sorted and ready to plug-in
+           into a Python's method header (i.e., flags and those which
+           can be inferred will go last so they can default =None)
+        """
+        return sorted(self.args,
+                      key=lambda x: x.is_flag or x.can_be_inferred)
+
+    def __repr__(self, ignore_id=False):
+        if self.id is None or ignore_id:
+            hex_id = ''
+        else:
+            hex_id = '#{:08x}'.format(self.id)
+
+        if self.args:
+            args = ' ' + ' '.join([repr(arg) for arg in self.args])
+        else:
+            args = ''
+
+        return '{}{}{} = {}'.format(self.fullname, hex_id, args, self.result)
+
+    def infer_id(self):
+        representation = self.__repr__(ignore_id=True)
+        representation = representation\
+            .replace(':bytes ', ':string ')\
+            .replace('?bytes ', '?string ')\
+            .replace('<', ' ').replace('>', '')\
+            .replace('{', '').replace('}', '')
+
+        representation = re.sub(
+            r' \w+:flags\.\d+\?true',
+            r'',
+            representation
+        )
+        return crc32(representation.encode('ascii'))
+
+
+class TLArg:
+    def __init__(self, name, arg_type, generic_definition):
+        """
+        Initializes a new .tl argument
+        :param name: The name of the .tl argument
+        :param arg_type: The type of the .tl argument
+        :param generic_definition: Is the argument a generic definition?
+                                   (i.e. {X:Type})
+        """
+        self.name = 'is_self' if name == 'self' else name
+
+        # Default values
+        self.is_vector = False
+        self.is_flag = False
+        self.skip_constructor_id = False
+        self.flag_index = -1
+
+        # Special case: some types can be inferred, which makes it
+        # less annoying to type. Currently the only type that can
+        # be inferred is if the name is 'random_id', to which a
+        # random ID will be assigned if left as None (the default)
+        self.can_be_inferred = name == 'random_id'
+
+        # The type can be an indicator that other arguments will be flags
+        if arg_type == '#':
+            self.flag_indicator = True
+            self.type = None
+            self.is_generic = False
+        else:
+            self.flag_indicator = False
+            self.is_generic = arg_type.startswith('!')
+            # Strip the exclamation mark always to have only the name
+            self.type = arg_type.lstrip('!')
+
+            # The type may be a flag (flags.IDX?REAL_TYPE)
+            # Note that 'flags' is NOT the flags name; this
+            # is determined by a previous argument
+            # However, we assume that the argument will always be called 'flags'
+            flag_match = re.match(r'flags.(\d+)\?([\w<>.]+)', self.type)
+            if flag_match:
+                self.is_flag = True
+                self.flag_index = int(flag_match.group(1))
+                # Update the type to match the exact type, not the "flagged" one
+                self.type = flag_match.group(2)
+
+            # Then check if the type is a Vector<REAL_TYPE>
+            vector_match = re.match(r'[Vv]ector<([\w\d.]+)>', self.type)
+            if vector_match:
+                self.is_vector = True
+
+                # If the type's first letter is not uppercase, then
+                # it is a constructor and we use (read/write) its ID
+                # as pinpointed on issue #81.
+                self.use_vector_id = self.type[0] == 'V'
+
+                # Update the type to match the one inside the vector
+                self.type = vector_match.group(1)
+
+            # See use_vector_id. An example of such case is ipPort in
+            # help.configSpecial
+            if self.type.split('.')[-1][0].islower():
+                self.skip_constructor_id = True
+
+            # The name may contain "date" in it, if this is the case and the type is "int",
+            # we can safely assume that this should be treated as a "date" object.
+            # Note that this is not a valid Telegram object, but it's easier to work with
+            if self.type == 'int' and (
+                        re.search(r'(\b|_)date\b', name) or
+                        name in ('expires', 'expires_at', 'was_online')):
+                self.type = 'date'
+
+        self.generic_definition = generic_definition
+
+    def type_hint(self):
+        type = self.type
+        if '.' in type:
+            type = type.split('.')[1]
+        result = {
+            'int': 'int',
+            'long': 'int',
+            'int128': 'int',
+            'int256': 'int',
+            'string': 'str',
+            'date': 'Optional[datetime]',  # None date = 0 timestamp
+            'bytes': 'bytes',
+            'true': 'bool',
+        }.get(type, "Type{}".format(type))
+        if self.is_vector:
+            result = 'List[{}]'.format(result)
+        if self.is_flag and type != 'date':
+            result = 'Optional[{}]'.format(result)
+
+        return result
+
+    def __str__(self):
+        # Find the real type representation by updating it as required
+        real_type = self.type
+        if self.flag_indicator:
+            real_type = '#'
+
+        if self.is_vector:
+            if self.use_vector_id:
+                real_type = 'Vector<{}>'.format(real_type)
+            else:
+                real_type = 'vector<{}>'.format(real_type)
+
+        if self.is_generic:
+            real_type = '!{}'.format(real_type)
+
+        if self.is_flag:
+            real_type = 'flags.{}?{}'.format(self.flag_index, real_type)
+
+        if self.generic_definition:
+            return '{{{}:{}}}'.format(self.name, real_type)
+        else:
+            return '{}:{}'.format(self.name, real_type)
+
+    def __repr__(self):
+        return str(self).replace(':date', ':int').replace('?date', '?int')
+
+
+def _from_line(line, is_function, layer):
+    match = re.match(
+        r'^([\w.]+)'                     # 'name'
+        r'(?:#([0-9a-fA-F]+))?'          # '#optionalcode'
+        r'(?:\s{?\w+:[\w\d<>#.?!]+}?)*'  # '{args:.0?type}'
+        r'\s=\s'                         # ' = '
+        r'([\w\d<>#.?]+);$',             # '<result.type>;'
+        line
+    )
+    if match is None:
+        # Probably "vector#1cb5c415 {t:Type} # [ t ] = Vector t;"
+        raise ValueError('Cannot parse TLObject {}'.format(line))
+
+    args_match = re.findall(
+        r'({)?'
+        r'(\w+)'
+        r':'
+        r'([\w\d<>#.?!]+)'
+        r'}?',
+        line
+    )
+    return TLObject(
+        fullname=match.group(1),
+        object_id=match.group(2),
+        result=match.group(3),
+        is_function=is_function,
+        layer=layer,
+        args=[TLArg(name, arg_type, brace != '')
+              for brace, name, arg_type in args_match]
+    )
+
+
+def parse_tl(file_path, layer, ignore_core=False, invalid_bot_methods=None):
+    """This method yields TLObjects from a given .tl file."""
+    if invalid_bot_methods is None:
+        invalid_bot_methods = set()
+
+    with open(file_path, encoding='utf-8') as file:
+        is_function = False
+        for line in file:
+            comment_index = line.find('//')
+            if comment_index != -1:
+                line = line[:comment_index]
+
+            line = line.strip()
+            if not line:
+                continue
+
+            match = re.match('---(\w+)---', line)
+            if match:
+                following_types = match.group(1)
+                is_function = following_types == 'functions'
+                continue
+
+            try:
+                result = _from_line(line, is_function, layer=layer)
+                if not ignore_core or result.id not in CORE_TYPES:
+                    result.bot_usable =\
+                        result.fullname not in invalid_bot_methods
+
+                    yield result
+            except ValueError as e:
+                if 'vector#1cb5c415' not in str(e):
+                    raise
+
+
+def find_layer(file_path):
+    """Finds the layer used on the specified scheme.tl file."""
+    layer_regex = re.compile(r'^//\s*LAYER\s*(\d+)$')
+    with open(file_path, encoding='utf-8') as file:
+        for line in file:
+            match = layer_regex.match(line)
+            if match:
+                return int(match.group(1))

telethon_generator/source_builder.py

@@ -16,7 +16,7 @@ class SourceBuilder:
         """
         self.write(' ' * (self.current_indent * self.indent_size))
 
-    def write(self, string):
+    def write(self, string, *args, **kwargs):
         """Writes a string into the source code,
            applying indentation if required
         """
@@ -26,13 +26,16 @@ class SourceBuilder:
             if string.strip():
                 self.indent()
 
-        self.out_stream.write(string)
+        if args or kwargs:
+            self.out_stream.write(string.format(*args, **kwargs))
+        else:
+            self.out_stream.write(string)
 
-    def writeln(self, string=''):
+    def writeln(self, string='', *args, **kwargs):
         """Writes a string into the source code _and_ appends a new line,
            applying indentation if required
         """
-        self.write(string + '\n')
+        self.write(string + '\n', *args, **kwargs)
         self.on_new_line = True
 
         # If we're writing a block, increment indent for the next time

telethon_generator/tl_generator.py

@@ -1,781 +0,0 @@
-import os
-import re
-import shutil
-import struct
-from zlib import crc32
-from collections import defaultdict
-
-from .parser import SourceBuilder, TLParser, TLObject
-AUTO_GEN_NOTICE = \
-    '"""File generated by TLObjects\' generator. All changes will be ERASED"""'
-
-
-AUTO_CASTS = {
-    'InputPeer': 'utils.get_input_peer(client.get_input_entity({}))',
-    'InputChannel': 'utils.get_input_channel(client.get_input_entity({}))',
-    'InputUser': 'utils.get_input_user(client.get_input_entity({}))',
-    'InputMedia': 'utils.get_input_media({})',
-    'InputPhoto': 'utils.get_input_photo({})'
-}
-
-
-class TLGenerator:
-    def __init__(self, output_dir):
-        self.output_dir = output_dir
-
-    def _get_file(self, *paths):
-        return os.path.join(self.output_dir, *paths)
-
-    def _rm_if_exists(self, filename):
-        file = self._get_file(filename)
-        if os.path.exists(file):
-            if os.path.isdir(file):
-                shutil.rmtree(file)
-            else:
-                os.remove(file)
-
-    def tlobjects_exist(self):
-        """Determines whether the TLObjects were previously
-           generated (hence exist) or not
-        """
-        return os.path.isfile(self._get_file('all_tlobjects.py'))
-
-    def clean_tlobjects(self):
-        """Cleans the automatically generated TLObjects from disk"""
-        for name in ('functions', 'types', 'all_tlobjects.py'):
-            self._rm_if_exists(name)
-
-    def generate_tlobjects(self, scheme_file, import_depth):
-        """Generates all the TLObjects from scheme.tl to
-           tl/functions and tl/types
-        """
-
-        # First ensure that the required parent directories exist
-        os.makedirs(self._get_file('functions'), exist_ok=True)
-        os.makedirs(self._get_file('types'), exist_ok=True)
-
-        # Step 0: Cache the parsed file on a tuple
-        tlobjects = tuple(TLParser.parse_file(scheme_file, ignore_core=True))
-
-        # Step 1: Group everything by {namespace: [tlobjects]} so we can
-        # easily generate __init__.py files with all the TLObjects on them.
-        namespace_functions = defaultdict(list)
-        namespace_types = defaultdict(list)
-
-        # Make use of this iteration to also store 'Type: [Constructors]',
-        # used when generating the documentation for the classes.
-        type_constructors = defaultdict(list)
-        for tlobject in tlobjects:
-            if tlobject.is_function:
-                namespace_functions[tlobject.namespace].append(tlobject)
-            else:
-                namespace_types[tlobject.namespace].append(tlobject)
-                type_constructors[tlobject.result].append(tlobject)
-
-        # Step 2: Generate the actual code
-        self._write_init_py(
-            self._get_file('functions'), import_depth,
-            namespace_functions, type_constructors
-        )
-        self._write_init_py(
-            self._get_file('types'), import_depth,
-            namespace_types, type_constructors
-        )
-
-        # Step 4: Once all the objects have been generated,
-        #         we can now group them in a single file
-        filename = os.path.join(self._get_file('all_tlobjects.py'))
-        with open(filename, 'w', encoding='utf-8') as file:
-            with SourceBuilder(file) as builder:
-                builder.writeln(AUTO_GEN_NOTICE)
-                builder.writeln()
-
-                builder.writeln('from . import types, functions')
-                builder.writeln()
-
-                # Create a constant variable to indicate which layer this is
-                builder.writeln('LAYER = {}'.format(
-                    TLParser.find_layer(scheme_file))
-                )
-                builder.writeln()
-
-                # Then create the dictionary containing constructor_id: class
-                builder.writeln('tlobjects = {')
-                builder.current_indent += 1
-
-                # Fill the dictionary (0x1a2b3c4f: tl.full.type.path.Class)
-                for tlobject in tlobjects:
-                    constructor = hex(tlobject.id)
-                    if len(constructor) != 10:
-                        # Make it a nice length 10 so it fits well
-                        constructor = '0x' + constructor[2:].zfill(8)
-
-                    builder.write('{}: '.format(constructor))
-                    builder.write(
-                        'functions' if tlobject.is_function else 'types')
-
-                    if tlobject.namespace:
-                        builder.write('.' + tlobject.namespace)
-
-                    builder.writeln('.{},'.format(tlobject.class_name()))
-
-                builder.current_indent -= 1
-                builder.writeln('}')
-
-    @staticmethod
-    def _write_init_py(out_dir, depth, namespace_tlobjects, type_constructors):
-        # namespace_tlobjects: {'namespace', [TLObject]}
-        os.makedirs(out_dir, exist_ok=True)
-        for ns, tlobjects in namespace_tlobjects.items():
-            file = os.path.join(out_dir, ns + '.py' if ns else '__init__.py')
-            with open(file, 'w', encoding='utf-8') as f, \
-                    SourceBuilder(f) as builder:
-                builder.writeln(AUTO_GEN_NOTICE)
-
-                # Both types and functions inherit from the TLObject class
-                # so they all can be serialized and sent, however, only the
-                # functions are "content_related".
-                builder.writeln(
-                    'from {}.tl.tlobject import TLObject'.format('.' * depth)
-                )
-                builder.writeln('from typing import Optional, List, Union, TYPE_CHECKING')
-
-                # Add the relative imports to the namespaces,
-                # unless we already are in a namespace.
-                if not ns:
-                    builder.writeln('from . import {}'.format(', '.join(
-                        x for x in namespace_tlobjects.keys() if x
-                    )))
-
-                # Import 'os' for those needing access to 'os.urandom()'
-                # Currently only 'random_id' needs 'os' to be imported,
-                # for all those TLObjects with arg.can_be_inferred.
-                builder.writeln('import os')
-
-                # Import struct for the .__bytes__(self) serialization
-                builder.writeln('import struct')
-
-                tlobjects.sort(key=lambda x: x.name)
-
-                type_names = set()
-                type_defs = []
-
-                # Find all the types in this file and generate type definitions
-                # based on the types. The type definitions are written to the
-                # file at the end.
-                for t in tlobjects:
-                    if not t.is_function:
-                        type_name = t.result
-                        if '.' in type_name:
-                            type_name = type_name[type_name.rindex('.'):]
-                        if type_name in type_names:
-                            continue
-                        type_names.add(type_name)
-                        constructors = type_constructors[type_name]
-                        if not constructors:
-                            pass
-                        elif len(constructors) == 1:
-                            type_defs.append('Type{} = {}'.format(
-                                type_name, constructors[0].class_name()))
-                        else:
-                            type_defs.append('Type{} = Union[{}]'.format(
-                                type_name, ','.join(c.class_name()
-                                                    for c in constructors)))
-
-                imports = {}
-                primitives = ('int', 'long', 'int128', 'int256', 'string',
-                              'date', 'bytes', 'true')
-                # Find all the types in other files that are used in this file
-                # and generate the information required to import those types.
-                for t in tlobjects:
-                    for arg in t.args:
-                        name = arg.type
-                        if not name or name in primitives:
-                            continue
-
-                        import_space = '{}.tl.types'.format('.' * depth)
-                        if '.' in name:
-                            namespace = name.split('.')[0]
-                            name = name.split('.')[1]
-                            import_space += '.{}'.format(namespace)
-
-                        if name not in type_names:
-                            type_names.add(name)
-                            if name == 'date':
-                                imports['datetime'] = ['datetime']
-                                continue
-                            elif not import_space in imports:
-                                imports[import_space] = set()
-                            imports[import_space].add('Type{}'.format(name))
-
-                # Add imports required for type checking.
-                builder.writeln('if TYPE_CHECKING:')
-                for namespace, names in imports.items():
-                    builder.writeln('from {} import {}'.format(
-                        namespace, ', '.join(names)))
-                else:
-                    builder.writeln('pass')
-                builder.end_block()
-
-                # Generate the class for every TLObject
-                for t in tlobjects:
-                    TLGenerator._write_source_code(
-                        t, builder, depth, type_constructors
-                    )
-                    builder.current_indent = 0
-
-                # Write the type definitions generated earlier.
-                builder.writeln('')
-                for line in type_defs:
-                    builder.writeln(line)
-
-
-    @staticmethod
-    def _write_source_code(tlobject, builder, depth, type_constructors):
-        """Writes the source code corresponding to the given TLObject
-           by making use of the 'builder' SourceBuilder.
-
-           Additional information such as file path depth and
-           the Type: [Constructors] must be given for proper
-           importing and documentation strings.
-        """
-        builder.writeln()
-        builder.writeln()
-        builder.writeln('class {}(TLObject):'.format(tlobject.class_name()))
-
-        # Class-level variable to store its Telegram's constructor ID
-        builder.writeln('CONSTRUCTOR_ID = {}'.format(hex(tlobject.id)))
-        builder.writeln('SUBCLASS_OF_ID = {}'.format(
-            hex(crc32(tlobject.result.encode('ascii'))))
-        )
-        builder.writeln()
-
-        # Flag arguments must go last
-        args = [
-            a for a in tlobject.sorted_args()
-            if not a.flag_indicator and not a.generic_definition
-        ]
-
-        # Convert the args to string parameters, flags having =None
-        args = [
-            (a.name if not a.is_flag and not a.can_be_inferred
-             else '{}=None'.format(a.name))
-            for a in args
-        ]
-
-        # Write the __init__ function
-        if args:
-            builder.writeln(
-                'def __init__(self, {}):'.format(', '.join(args))
-            )
-        else:
-            builder.writeln('def __init__(self):')
-
-        # Now update args to have the TLObject arguments, _except_
-        # those which are calculated on send or ignored, this is
-        # flag indicator and generic definitions.
-        #
-        # We don't need the generic definitions in Python
-        # because arguments can be any type
-        args = [arg for arg in tlobject.args
-                if not arg.flag_indicator and
-                not arg.generic_definition]
-
-        if args:
-            # Write the docstring, to know the type of the args
-            builder.writeln('"""')
-            for arg in args:
-                if not arg.flag_indicator:
-                    builder.writeln(':param {} {}:'.format(
-                        arg.doc_type_hint(), arg.name
-                    ))
-                    builder.current_indent -= 1  # It will auto-indent (':')
-
-            # We also want to know what type this request returns
-            # or to which type this constructor belongs to
-            builder.writeln()
-            if tlobject.is_function:
-                builder.write(':returns {}: '.format(tlobject.result))
-            else:
-                builder.write('Constructor for {}: '.format(tlobject.result))
-
-            constructors = type_constructors[tlobject.result]
-            if not constructors:
-                builder.writeln('This type has no constructors.')
-            elif len(constructors) == 1:
-                builder.writeln('Instance of {}.'.format(
-                    constructors[0].class_name()
-                ))
-            else:
-                builder.writeln('Instance of either {}.'.format(
-                    ', '.join(c.class_name() for c in constructors)
-                ))
-
-            builder.writeln('"""')
-
-        builder.writeln('super().__init__()')
-        # Functions have a result object and are confirmed by default
-        if tlobject.is_function:
-            builder.writeln('self.result = None')
-            builder.writeln(
-                'self.content_related = True')
-
-        # Set the arguments
-        if args:
-            # Leave an empty line if there are any args
-            builder.writeln()
-
-        for arg in args:
-            if not arg.can_be_inferred:
-                builder.writeln('self.{0} = {0}  # type: {1}'.format(
-                    arg.name, arg.python_type_hint()))
-                continue
-
-            # Currently the only argument that can be
-            # inferred are those called 'random_id'
-            if arg.name == 'random_id':
-                # Endianness doesn't really matter, and 'big' is shorter
-                code = "int.from_bytes(os.urandom({}), 'big', signed=True)" \
-                    .format(8 if arg.type == 'long' else 4)
-
-                if arg.is_vector:
-                    # Currently for the case of "messages.forwardMessages"
-                    # Ensure we can infer the length from id:Vector<>
-                    if not next(
-                            a for a in args if a.name == 'id').is_vector:
-                        raise ValueError(
-                            'Cannot infer list of random ids for ', tlobject
-                        )
-                    code = '[{} for _ in range(len(id))]'.format(code)
-
-                builder.writeln(
-                    "self.random_id = random_id if random_id "
-                    "is not None else {}".format(code)
-                )
-            else:
-                raise ValueError('Cannot infer a value for ', arg)
-
-        builder.end_block()
-
-        # Write the resolve(self, client, utils) method
-        if any(arg.type in AUTO_CASTS for arg in args):
-            builder.writeln('def resolve(self, client, utils):')
-            for arg in args:
-                ac = AUTO_CASTS.get(arg.type, None)
-                if ac:
-                    TLGenerator._write_self_assign(builder, arg, ac)
-            builder.end_block()
-
-        # Write the to_dict(self) method
-        builder.writeln('def to_dict(self):')
-        builder.writeln('return {')
-        builder.current_indent += 1
-
-        base_types = ('string', 'bytes', 'int', 'long', 'int128',
-                      'int256', 'double', 'Bool', 'true', 'date')
-
-        builder.write("'_': '{}'".format(tlobject.class_name()))
-        for arg in args:
-            builder.writeln(',')
-            builder.write("'{}': ".format(arg.name))
-            if arg.type in base_types:
-                if arg.is_vector:
-                    builder.write('[] if self.{0} is None else self.{0}[:]'
-                                  .format(arg.name))
-                else:
-                    builder.write('self.{}'.format(arg.name))
-            else:
-                if arg.is_vector:
-                    builder.write(
-                        '[] if self.{0} is None else [None '
-                        'if x is None else x.to_dict() for x in self.{0}]'
-                        .format(arg.name)
-                    )
-                else:
-                    builder.write(
-                        'None if self.{0} is None else self.{0}.to_dict()'
-                        .format(arg.name)
-                    )
-
-        builder.writeln()
-        builder.current_indent -= 1
-        builder.writeln("}")
-
-        builder.end_block()
-
-        # Write the .__bytes__() function
-        builder.writeln('def __bytes__(self):')
-
-        # Some objects require more than one flag parameter to be set
-        # at the same time. In this case, add an assertion.
-        repeated_args = defaultdict(list)
-        for arg in tlobject.args:
-            if arg.is_flag:
-                repeated_args[arg.flag_index].append(arg)
-
-        for ra in repeated_args.values():
-            if len(ra) > 1:
-                cnd1 = ('(self.{0} or self.{0} is not None)'
-                        .format(a.name) for a in ra)
-                cnd2 = ('(self.{0} is None or self.{0} is False)'
-                        .format(a.name) for a in ra)
-                builder.writeln(
-                    "assert ({}) or ({}), '{} parameters must all "
-                    "be False-y (like None) or all me True-y'".format(
-                        ' and '.join(cnd1), ' and '.join(cnd2),
-                        ', '.join(a.name for a in ra)
-                    )
-                )
-
-        builder.writeln("return b''.join((")
-        builder.current_indent += 1
-
-        # First constructor code, we already know its bytes
-        builder.writeln('{},'.format(repr(struct.pack('<I', tlobject.id))))
-
-        for arg in tlobject.args:
-            if TLGenerator.write_to_bytes(builder, arg, tlobject.args):
-                builder.writeln(',')
-
-        builder.current_indent -= 1
-        builder.writeln('))')
-        builder.end_block()
-
-        # Write the static from_reader(reader) function
-        builder.writeln('@staticmethod')
-        builder.writeln('def from_reader(reader):')
-        for arg in tlobject.args:
-            TLGenerator.write_read_code(
-                builder, arg, tlobject.args, name='_' + arg.name
-            )
-
-        builder.writeln('return {}({})'.format(
-            tlobject.class_name(), ', '.join(
-                '{0}=_{0}'.format(a.name) for a in tlobject.sorted_args()
-                if not a.flag_indicator and not a.generic_definition
-            )
-        ))
-
-        # Only requests can have a different response that's not their
-        # serialized body, that is, we'll be setting their .result.
-        #
-        # The default behaviour is reading a TLObject too, so no need
-        # to override it unless necessary.
-        if tlobject.is_function and not TLGenerator._is_boxed(tlobject.result):
-            builder.end_block()
-            builder.writeln('def on_response(self, reader):')
-            TLGenerator.write_request_result_code(builder, tlobject)
-
-    @staticmethod
-    def _is_boxed(type_):
-        # https://core.telegram.org/mtproto/serialize#boxed-and-bare-types
-        # TL;DR; boxed types start with uppercase always, so we can use
-        # this to check whether everything in it is boxed or not.
-        #
-        # The API always returns a boxed type, but it may inside a Vector<>
-        # or a namespace, and the Vector may have a not-boxed type. For this
-        # reason we find whatever index, '<' or '.'. If neither are present
-        # we will get -1, and the 0th char is always upper case thus works.
-        # For Vector types and namespaces, it will check in the right place.
-        check_after = max(type_.find('<'), type_.find('.'))
-        return type_[check_after + 1].isupper()
-
-    @staticmethod
-    def _write_self_assign(builder, arg, get_input_code):
-        """Writes self.arg = input.format(self.arg), considering vectors"""
-        if arg.is_vector:
-            builder.write('self.{0} = [{1} for _x in self.{0}]'
-                          .format(arg.name, get_input_code.format('_x')))
-        else:
-            builder.write('self.{} = {}'.format(
-                arg.name, get_input_code.format('self.' + arg.name)))
-
-        builder.writeln(
-            ' if self.{} else None'.format(arg.name) if arg.is_flag else ''
-        )
-
-    @staticmethod
-    def get_file_name(tlobject, add_extension=False):
-        """Gets the file name in file_name_format.py for the given TLObject"""
-
-        # Courtesy of http://stackoverflow.com/a/1176023/4759433
-        s1 = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', tlobject.name)
-        result = re.sub('([a-z0-9])([A-Z])', r'\1_\2', s1).lower()
-        if add_extension:
-            return result + '.py'
-        else:
-            return result
-
-    @staticmethod
-    def write_to_bytes(builder, arg, args, name=None):
-        """
-        Writes the .__bytes__() code for the given argument
-        :param builder: The source code builder
-        :param arg: The argument to write
-        :param args: All the other arguments in TLObject same __bytes__.
-                     This is required to determine the flags value
-        :param name: The name of the argument. Defaults to "self.argname"
-                     This argument is an option because it's required when
-                     writing Vectors<>
-        """
-        if arg.generic_definition:
-            return  # Do nothing, this only specifies a later type
-
-        if name is None:
-            name = 'self.{}'.format(arg.name)
-
-        # The argument may be a flag, only write if it's not None AND
-        # if it's not a True type.
-        # True types are not actually sent, but instead only used to
-        # determine the flags.
-        if arg.is_flag:
-            if arg.type == 'true':
-                return  # Exit, since True type is never written
-            elif arg.is_vector:
-                # Vector flags are special since they consist of 3 values,
-                # so we need an extra join here. Note that empty vector flags
-                # should NOT be sent either!
-                builder.write("b'' if {0} is None or {0} is False "
-                              "else b''.join((".format(name))
-            else:
-                builder.write("b'' if {0} is None or {0} is False "
-                              "else (".format(name))
-
-        if arg.is_vector:
-            if arg.use_vector_id:
-                # vector code, unsigned 0x1cb5c415 as little endian
-                builder.write(r"b'\x15\xc4\xb5\x1c',")
-
-            builder.write("struct.pack('<i', len({})),".format(name))
-
-            # Cannot unpack the values for the outer tuple through *[(
-            # since that's a Python >3.5 feature, so add another join.
-            builder.write("b''.join(")
-
-            # Temporary disable .is_vector, not to enter this if again
-            # Also disable .is_flag since it's not needed per element
-            old_flag = arg.is_flag
-            arg.is_vector = arg.is_flag = False
-            TLGenerator.write_to_bytes(builder, arg, args, name='x')
-            arg.is_vector = True
-            arg.is_flag = old_flag
-
-            builder.write(' for x in {})'.format(name))
-
-        elif arg.flag_indicator:
-            # Calculate the flags with those items which are not None
-            if not any(f.is_flag for f in args):
-                # There's a flag indicator, but no flag arguments so it's 0
-                builder.write(r"b'\0\0\0\0'")
-            else:
-                builder.write("struct.pack('<I', ")
-                builder.write(
-                    ' | '.join('(0 if {0} is None or {0} is False else {1})'
-                               .format('self.{}'.format(flag.name),
-                                       1 << flag.flag_index)
-                               for flag in args if flag.is_flag)
-                )
-                builder.write(')')
-
-        elif 'int' == arg.type:
-            # struct.pack is around 4 times faster than int.to_bytes
-            builder.write("struct.pack('<i', {})".format(name))
-
-        elif 'long' == arg.type:
-            builder.write("struct.pack('<q', {})".format(name))
-
-        elif 'int128' == arg.type:
-            builder.write("{}.to_bytes(16, 'little', signed=True)".format(name))
-
-        elif 'int256' == arg.type:
-            builder.write("{}.to_bytes(32, 'little', signed=True)".format(name))
-
-        elif 'double' == arg.type:
-            builder.write("struct.pack('<d', {})".format(name))
-
-        elif 'string' == arg.type:
-            builder.write('TLObject.serialize_bytes({})'.format(name))
-
-        elif 'Bool' == arg.type:
-            # 0x997275b5 if boolean else 0xbc799737
-            builder.write(
-                r"b'\xb5ur\x99' if {} else b'7\x97y\xbc'".format(name)
-            )
-
-        elif 'true' == arg.type:
-            pass  # These are actually NOT written! Only used for flags
-
-        elif 'bytes' == arg.type:
-            builder.write('TLObject.serialize_bytes({})'.format(name))
-
-        elif 'date' == arg.type:  # Custom format
-            builder.write('TLObject.serialize_datetime({})'.format(name))
-
-        else:
-            # Else it may be a custom type
-            builder.write('bytes({})'.format(name))
-
-        if arg.is_flag:
-            builder.write(')')
-            if arg.is_vector:
-                builder.write(')')  # We were using a tuple
-
-        return True  # Something was written
-
-    @staticmethod
-    def write_read_code(builder, arg, args, name):
-        """
-        Writes the read code for the given argument, setting the
-        arg.name variable to its read value.
-
-        :param builder: The source code builder
-        :param arg: The argument to write
-        :param args: All the other arguments in TLObject same on_send.
-                     This is required to determine the flags value
-        :param name: The name of the argument. Defaults to "self.argname"
-                     This argument is an option because it's required when
-                     writing Vectors<>
-        """
-
-        if arg.generic_definition:
-            return  # Do nothing, this only specifies a later type
-
-        # The argument may be a flag, only write that flag was given!
-        was_flag = False
-        if arg.is_flag:
-            # Treat 'true' flags as a special case, since they're true if
-            # they're set, and nothing else needs to actually be read.
-            if 'true' == arg.type:
-                builder.writeln(
-                    '{} = bool(flags & {})'.format(name, 1 << arg.flag_index)
-                )
-                return
-
-            was_flag = True
-            builder.writeln('if flags & {}:'.format(
-                1 << arg.flag_index
-            ))
-            # Temporary disable .is_flag not to enter this if
-            # again when calling the method recursively
-            arg.is_flag = False
-
-        if arg.is_vector:
-            if arg.use_vector_id:
-                # We have to read the vector's constructor ID
-                builder.writeln("reader.read_int()")
-
-            builder.writeln('{} = []'.format(name))
-            builder.writeln('for _ in range(reader.read_int()):')
-            # Temporary disable .is_vector, not to enter this if again
-            arg.is_vector = False
-            TLGenerator.write_read_code(builder, arg, args, name='_x')
-            builder.writeln('{}.append(_x)'.format(name))
-            arg.is_vector = True
-
-        elif arg.flag_indicator:
-            # Read the flags, which will indicate what items we should read next
-            builder.writeln('flags = reader.read_int()')
-            builder.writeln()
-
-        elif 'int' == arg.type:
-            builder.writeln('{} = reader.read_int()'.format(name))
-
-        elif 'long' == arg.type:
-            builder.writeln('{} = reader.read_long()'.format(name))
-
-        elif 'int128' == arg.type:
-            builder.writeln(
-                '{} = reader.read_large_int(bits=128)'.format(name)
-            )
-
-        elif 'int256' == arg.type:
-            builder.writeln(
-                '{} = reader.read_large_int(bits=256)'.format(name)
-            )
-
-        elif 'double' == arg.type:
-            builder.writeln('{} = reader.read_double()'.format(name))
-
-        elif 'string' == arg.type:
-            builder.writeln('{} = reader.tgread_string()'.format(name))
-
-        elif 'Bool' == arg.type:
-            builder.writeln('{} = reader.tgread_bool()'.format(name))
-
-        elif 'true' == arg.type:
-            # Arbitrary not-None value, don't actually read "true" flags
-            builder.writeln('{} = True'.format(name))
-
-        elif 'bytes' == arg.type:
-            builder.writeln('{} = reader.tgread_bytes()'.format(name))
-
-        elif 'date' == arg.type:  # Custom format
-            builder.writeln('{} = reader.tgread_date()'.format(name))
-
-        else:
-            # Else it may be a custom type
-            if not arg.skip_constructor_id:
-                builder.writeln('{} = reader.tgread_object()'.format(name))
-            else:
-                # Import the correct type inline to avoid cyclic imports.
-                # There may be better solutions so that we can just access
-                # all the types before the files have been parsed, but I
-                # don't know of any.
-                sep_index = arg.type.find('.')
-                if sep_index == -1:
-                    ns, t = '.', arg.type
-                else:
-                    ns, t = '.' + arg.type[:sep_index], arg.type[sep_index+1:]
-                class_name = TLObject.class_name_for(t)
-
-                # There would be no need to import the type if we're in the
-                # file with the same namespace, but since it does no harm
-                # and we don't have information about such thing in the
-                # method we just ignore that case.
-                builder.writeln('from {} import {}'.format(ns, class_name))
-                builder.writeln('{} = {}.from_reader(reader)'.format(
-                    name, class_name
-                ))
-
-        # End vector and flag blocks if required (if we opened them before)
-        if arg.is_vector:
-            builder.end_block()
-
-        if was_flag:
-            builder.current_indent -= 1
-            builder.writeln('else:')
-            builder.writeln('{} = None'.format(name))
-            builder.current_indent -= 1
-            # Restore .is_flag
-            arg.is_flag = True
-
-    @staticmethod
-    def write_request_result_code(builder, tlobject):
-        """
-        Writes the receive code for the given function
-
-        :param builder: The source code builder
-        :param tlobject: The TLObject for which the 'self.result = '
-                         will be written
-        """
-        if tlobject.result.startswith('Vector<'):
-            # Vector results are a bit special since they can also be composed
-            # of integer values and such; however, the result of requests is
-            # not parsed as arguments are and it's a bit harder to tell which
-            # is which.
-            if tlobject.result == 'Vector<int>':
-                builder.writeln('reader.read_int()  # Vector ID')
-                builder.writeln('count = reader.read_int()')
-                builder.writeln(
-                    'self.result = [reader.read_int() for _ in range(count)]'
-                )
-            elif tlobject.result == 'Vector<long>':
-                builder.writeln('reader.read_int()  # Vector ID')
-                builder.writeln('count = reader.read_long()')
-                builder.writeln(
-                    'self.result = [reader.read_long() for _ in range(count)]'
-                )
-            else:
-                builder.writeln('self.result = reader.tgread_vector()')
-        else:
-            builder.writeln('self.result = reader.tgread_object()')

telethon_generator/utils.py

@@ -0,0 +1,8 @@
+import re
+
+
+def snake_to_camel_case(name, suffix=None):
+    # Courtesy of http://stackoverflow.com/a/31531797/4759433
+    result = re.sub(r'_([a-z])', lambda m: m.group(1).upper(), name)
+    result = result[:1].upper() + result[1:].replace('_', '')
+    return result + suffix if suffix else result

telethon_tests/test_higher_level.py

@@ -34,7 +34,7 @@ class HigherLevelTests(unittest.TestCase):
                 progress_callback=lambda c, t:
                     print('test_cdn_download:uploading {:.2%}...'.format(c/t))
             )
-            msg = client.get_message_history(me)[1][0]
+            msg = client.get_messages(me)[1][0]
 
             out = BytesIO()
             client.download_media(msg, out)