Patched code generator and updated README.rst removing markdown leftovers

README.rst

@@ -21,11 +21,11 @@ Being written **entirely** on Python, Telethon can run as a script under any env
 or use it in any other script you have. Want to send a message to someone when you're available? Write a script.
 Do you want check for new messages at a given time and find relevant ones? Write a script.
 
-Hungry for more API calls which the :code:`TelegramClient` class doesn't *seem* to have implemented?
+Hungry for more API calls which the ``TelegramClient`` class doesn't *seem* to have implemented?
-Please read [this section](#using-more-than-just-telegramclient).
+Please read `Using more than just TelegramClient`_.
 
-Obtaining your Telegram :code:`API ID` and :code:`Hash`
+Obtaining your Telegram ``API ID`` and ``Hash``
-=======================================================
+===============================================
 In order to use Telethon, you first need to obtain your very own API ID and Hash:
 
 1. Follow `this link <https://my.telegram.org>`_ and login with your phone number.
@@ -35,12 +35,12 @@ In order to use Telethon, you first need to obtain your very own API ID and Hash
    can be changed later as long as I'm aware.
 4. Click on *Create application* at the end.
 
-Now that you know your :code:`API ID` and :code:`Hash`, you can continue installing Telethon.
+Now that you know your ``API ID`` and ``Hash``, you can continue installing Telethon.
 
 Installing Telethon
 ===================
-Installing Telethon via :code:`pip`
+Installing Telethon via ``pip``
------------------------------------
+-------------------------------
 On a terminal, issue the following command:
 
 .. code:: sh
@@ -52,11 +52,11 @@ You're ready to go.
 Installing Telethon manually
 ----------------------------
 
-1. Install the required :code:`pyaes` module: :code:`sudo -H pip install pyaes`
+1. Install the required ``pyaes`` module: ``sudo -H pip install pyaes``
    (`GitHub <https://github.com/ricmoo/pyaes>`_, `package index <https://pypi.python.org/pypi/pyaes>`_)
-2. Clone Telethon's GitHub repository: :code:`git clone https://github.com/LonamiWebs/Telethon.git`
+2. Clone Telethon's GitHub repository: ``git clone https://github.com/LonamiWebs/Telethon.git``
-3. Enter the cloned repository: :code:`cd Telethon`
+3. Enter the cloned repository: ``cd Telethon``
-4. Run the code generator: :code:`python3 telethon_generator/tl_generator.py`
+4. Run the code generator: ``python3 telethon_generator/tl_generator.py``
 5. Done!
 
 Running Telethon
@@ -72,45 +72,48 @@ If you've installed Telethon via pip, launch an interactive python3 session and
   >>> client = InteractiveTelegramClient('sessionid', '+34600000000',
   ...     api_id=12345, api_hash='0123456789abcdef0123456789abcdef')
 
-  ┌───────────────────────────────────────────────────────────┐
+  ┌─────────────────────────────────────────────┐
-  │                      Initialization                       │
+  │               Initialization                │
-  └───────────────────────────────────────────────────────────┘
+  └─────────────────────────────────────────────┘
   Initializing interactive example...
   Connecting to Telegram servers...
   >>> client.run()
 
-If, on the other hand, you've installed Telethon manually, head to the :code:`api/` directory and create a
+If, on the other hand, you've installed Telethon manually, head to the ``api/`` directory and create a
-copy of the :code:`settings_example` file, naming it :code:`settings` (lowercase!). Then fill the file with the
+copy of the ``settings_example`` file, naming it ``settings`` (lowercase!). Then fill the file with the
-corresponding values (your :code:`api_id`, :code:`api_hash` and phone number in international format).
+corresponding values (your ``api_id``, ``api_hash`` and phone number in international format).
 
-Then, simply run :code:`python3 try_telethon.py` to start the interactive example.
+Then, simply run ``python3 try_telethon.py`` to start the interactive example.
 
 Advanced uses
 =============
-Using more than just `TelegramClient`
--------------------------------------
-The :code:`TelegramClient` class should be used to provide a quick, well-documented and simplified starting point.
-It is **not** meant to be a place for _all_ the available Telegram :code:`Request`'s, because there are simply too many.
 
-However, this doesn't mean that you cannot :code:`invoke` all the power of Telegram's API.
+.. _Using more than just TelegramClient:
-Whenever you need to :code:`invoke` a Telegram :code:`Request`, all you need to do is the following:
+
+Using more than just ``TelegramClient``
+---------------------------------------
+The ``TelegramClient`` class should be used to provide a quick, well-documented and simplified starting point.
+It is **not** meant to be a place for *all* the available Telegram ``Request``'s, because there are simply too many.
+
+However, this doesn't mean that you cannot ``invoke`` all the power of Telegram's API.
+Whenever you need to ``invoke`` a Telegram ``Request``, all you need to do is the following:
 
 .. code:: python
 
   result = client.invoke(SomeRequest(...))
 
-You have just :code:`invoke`'d :code:`SomeRequest` and retrieved its :code:`result`! That wasn't hard at all, was it?
+You have just ``invoke``'d ``SomeRequest`` and retrieved its ``result``! That wasn't hard at all, was it?
-Now you may wonder, what's the deal with *all the power of Telegram's API*? Have a look under :code:`tl/functions/`.
+Now you may wonder, what's the deal with *all the power of Telegram's API*? Have a look under ``tl/functions/``.
-That is *everything* you can do. You have **over 200 API `Request`'s** at your disposal.
+That is *everything* you can do. You have **over 200 API** ``Request``'s at your disposal.
 
-However, we don't pretty know *how* that :code:`result` looks like. Easy. :code:`print(str(result))` should
+However, we don't pretty know *how* that ``result`` looks like. Easy. ``print(str(result))`` should
-give you a quick overview. Nevertheless, there may be more than a single :code:`result`! Let's have a look at
+give you a quick overview. Nevertheless, there may be more than a single ``result``! Let's have a look at
-this seemingly innocent :code:`TL` definition:
+this seemingly innocent ``TL`` definition:
 
-`messages.getWebPagePreview#25223e24 message:string = MessageMedia;`
+``messages.getWebPagePreview#25223e24 message:string = MessageMedia;``
 
-Focusing on the end, we can see that the `result` of invoking `GetWebPagePreviewRequest` is `MessageMedia`. But how
+Focusing on the end, we can see that the ``result`` of invoking ``GetWebPagePreviewRequest`` is ``MessageMedia``.
-can `MessageMedia` exactly look like? It's time to have another look, but this time under `tl/types/`:
+But how can ``MessageMedia`` exactly look like? It's time to have another look, but this time under ``tl/types/``:
 
 .. code:: sh
 
@@ -128,59 +131,43 @@ can `MessageMedia` exactly look like? It's time to have another look, but this t
   │       └── message_media_web_page.py
 
 Those are *eight* different types! How do we know what exact type it is to determine its properties? A simple
-:code:`if type(result) == MessageMediaContact:` or similar will do. Now you're ready to take advantage of
+``if type(result) == MessageMediaContact:`` or similar will do. Now you're ready to take advantage of
 Telegram's polymorphism.
 
 Tips for porting Telethon
 -------------------------
-First of all, you need to understand how the :code:`scheme.tl` (:code:`TL` language) works. Every object
+First of all, you need to understand how the ``scheme.tl`` (``TL`` language) works. Every object
 definition is written as follows:
 
-:code:`name#id argument_name:argument_type = CommonType`
+``name#id argument_name:argument_type = CommonType``
 
-This means that in a single line you know what the :code:`TLObject` name is. You know it's unique ID, and you
+This means that in a single line you know what the ``TLObject`` name is. You know it's unique ID, and you
 know what arguments it has. It really isn't that hard to write a generator for generating code to any platform!
 
-The generated code should also be able to *encode* the :code:`Request` into bytes, so they can be sent over
+The generated code should also be able to *encode* the ``Request`` into bytes, so they can be sent over
-the network. This isn't a big deal either, because you know how the :code:`TLObject`'s are made.
+the network. This isn't a big deal either, because you know how the ``TLObject``'s are made.
 
 Once you have your own [code generator](telethon_generator/tl_generator.py), start by looking at the
 `first release <https://github.com/LonamiWebs/Telethon/releases/tag/v0.1>`_ of Telethon.
 The code there is simple to understand, easy to read and hence easy to port. No extra useless features.
 Only the bare bones. Perfect for starting a *new implementation*.
 
-P.S.: I may have lied a bit. The :code:`TL` language is not that easy. But it's not that hard either.
+P.S.: I may have lied a bit. The ``TL`` language is not that easy. But it's not that hard either.
-You're free to sniff the :code:`parser/` files and learn how to parse other more complex lines.
+You're free to sniff the ``parser/`` files and learn how to parse other more complex lines.
 Or simply use that code and change the [SourceBuilder](telethon_generator/parser/source_builder.py)!
 
-Code generator limitations
+Notes about the code generator
---------------------------
-The current code generator is not complete, yet adding the missing features would only over-complicate an
-already hard-to-read code. Some parts of the :code:`.tl` file *should* be omitted, because they're "built-in"
-in the generated code (such as writing booleans, etc.).
-
-In order to make sure that all the generated files will work, please make sure to **always** comment out these
-lines in :code:`scheme.tl` (the latest version can always be found
-`here <https://github.com/telegramdesktop/tdesktop/blob/master/Telegram/SourceFiles/mtproto/scheme.tl>`_):
-
-.. code:: c
-
-  // boolFalse#bc799737 = Bool;
-  // boolTrue#997275b5 = Bool;
-  // true#3fedd339 = True;
-  // vector#1cb5c415 {t:Type} # [ t ] = Vector t;
-
-Also please make sure to rename :code:`updates#74ae4240 ...` to :code:`updates_tg#74ae4240 ...` or similar to
-avoid confusion between the :code:`updates` folder and the :code:`updates.py` file! Note that depending on the name,
-it may break things somewhere else. So please stick with the suggested name or give one which is still descriptive
-enough and easy to remember.
-
-Updating the :code:`scheme.tl`
 ------------------------------
-Have you found a more updated version of the :code:`scheme.tl` file? Those are great news! Updating is as simple
+The code generator will skip the types considered as *core types*. These types are usually included in
+almost every programming language, such as boolean values or lists, and also the Telegram True flag,
+which is *not* sent but rather used to determine whether that flag should be enabled or not.
+
+Updating the ``scheme.tl``
+--------------------------
+Have you found a more updated version of the ``scheme.tl`` file? Those are great news! Updating is as simple
 as grabbing the
 `latest version <https://github.com/telegramdesktop/tdesktop/blob/master/Telegram/SourceFiles/mtproto/scheme.tl>`_
 and replacing the one you can find in this same directory by the updated one.
-Don't forget to run :code:`python3 tl_generator.py`.
+Don't forget to run ``python3 tl_generator.py``.
 
 If the changes weren't too big, everything should still work the same way as it did before; but with extra features.

telethon_generator/parser/tl_object.py

@@ -2,6 +2,9 @@ import re
 
 
 class TLObject:
+    """.tl core types IDs (such as vector, booleans, etc.)"""
+    CORE_TYPES = (0x1cb5c415, 0xbc799737, 0x997275b5, 0x3fedd339)
+
     def __init__(self, fullname, id, args, result, is_function):
         """
         Initializes a new TLObject, given its properties.
@@ -73,6 +76,11 @@ class TLObject:
                         result=match.group(3),
                         is_function=is_function)
 
+    def is_core_type(self):
+        """Determines whether the TLObject is a "core type"
+           (and thus should be embedded in the generated code) or not"""
+        return self.id in TLObject.CORE_TYPES
+
     def __repr__(self):
         fullname = ('{}.{}'.format(self.namespace, self.name) if self.namespace is not None
                     else self.name)

telethon_generator/scheme.tl

@@ -410,7 +410,7 @@ updateShortMessage#914fbf11 flags:# out:flags.1?true mentioned:flags.4?true medi
 updateShortChatMessage#16812688 flags:# out:flags.1?true mentioned:flags.4?true media_unread:flags.5?true silent:flags.13?true id:int from_id:int chat_id:int message:string pts:int pts_count:int date:int fwd_from:flags.2?MessageFwdHeader via_bot_id:flags.11?int reply_to_msg_id:flags.3?int entities:flags.7?Vector<MessageEntity> = Updates;
 updateShort#78d4dec1 update:Update date:int = Updates;
 updatesCombined#725b04c3 updates:Vector<Update> users:Vector<User> chats:Vector<Chat> date:int seq_start:int seq:int = Updates;
-updates_tg#74ae4240 updates:Vector<Update> users:Vector<User> chats:Vector<Chat> date:int seq:int = Updates;
+updates#74ae4240 updates:Vector<Update> users:Vector<User> chats:Vector<Chat> date:int seq:int = Updates;
 updateShortSentMessage#11f1331c flags:# out:flags.1?true id:int pts:int pts_count:int date:int media:flags.9?MessageMedia entities:flags.7?Vector<MessageEntity> = Updates;
 
 photos.photos#8dca6aa5 photos:Vector<Photo> users:Vector<User> = photos.Photos;

telethon_generator/tl_generator.py

@@ -35,9 +35,30 @@ class TLGenerator:
         os.makedirs(get_output_path('functions'), exist_ok=True)
         os.makedirs(get_output_path('types'), exist_ok=True)
 
-        # Store the parsed file in a tuple for iterating it more than once
+        # Step 0: Store the parsed file in a tuple to avoid parsing it on each iteration
         tlobjects = tuple(TLParser.parse_file(scheme_file))
+
+        # Step 1: Ensure that no object has the same name as a namespace
+        # We must check this because Python will complain if it sees a
+        # file and a directory with the same name, which happens for example with "updates"
+        namespace_directories = set()
         for tlobject in tlobjects:
+            namespace_directories.add(tlobject.namespace)
+
+        for tlobject in tlobjects:
+            if TLGenerator.get_file_name(tlobject, add_extension=False) \
+                    in namespace_directories:
+                # If this TLObject isn't under the same directory as its name (i.e. "contacts"),
+                # append "_tg" to avoid confusion between the file and the directory (i.e. "updates")
+                if tlobject.namespace != tlobject.name:
+                    tlobject.name += '_tg'
+
+        # Step 2: Generate the actual code
+        for tlobject in tlobjects:
+            # Omit core types, these are embedded in the generated code
+            if tlobject.is_core_type():
+                continue
+
             # Determine the output directory and create it
             out_dir = get_output_path('functions' if tlobject.is_function
                                       else 'types')
@@ -168,7 +189,7 @@ class TLGenerator:
                     builder.writeln("return {}".format(str(tlobject)))
                     # builder.end_block()  # There is no need to end the last block
 
-        # Once all the objects have been generated, we can now group them in a single file
+        # Step 3: Once all the objects have been generated, we can now group them in a single file
         filename = os.path.join(get_output_path('all_tlobjects.py'))
         with open(filename, 'w', encoding='utf-8') as file:
             with SourceBuilder(file) as builder: