.. _asyncio-daemon-tutorial: Asyncio daemon tutorial ======================= .. meta:: :keywords: Python,asyncio,Daemon,Monitoring,Tutorial,Education,Web,API,REST API,Example,DI, Dependency injection,IoC,Inversion of control,Refactoring,Tests,Unit tests,Pytest, py.test,docker,docker-compose,backend :description: This tutorial shows how to build an asyncio application following the dependency injection principle. You will create the monitoring daemon, use docker & docker-compose, cover the daemon with the unit test and make some refactoring. This tutorial shows how to build an ``asyncio`` daemon following the dependency injection principle. In this tutorial we will use: - Python 3 - Docker - Docker-compose Start from the scratch or jump to the section: .. contents:: :local: :backlinks: none You can find complete project on the `Github `_. What are we going to build? --------------------------- We will build a monitoring daemon that monitors web services availability. The daemon will send the requests to the `example.com `_ and `httpbin.org `_ every couple of seconds. For each successfully completed response it will log: - The response code - The amount of bytes in the response - The time took to complete the response .. image:: asyncio_images/diagram.png Prerequisites ------------- We will use `Docker `_ and `docker-compose `_ in this tutorial. Let's check the versions: .. code-block:: bash docker --version docker-compose --version The output should look something like: .. code-block:: bash Docker version 19.03.12, build 48a66213fe docker-compose version 1.26.2, build eefe0d31 .. note:: If you don't have ``Docker`` or ``docker-compose`` you need to install them before proceeding. Follow these installation guides: - `Install Docker `_ - `Install docker-compose `_ The prerequisites are satisfied. Let's get started with the project layout. Project layout -------------- Create the project root folder and set it as a working directory: .. code-block:: bash mkdir monitoring-daemon-tutorial cd monitoring-daemon-tutorial Now we need to create the initial project structure. Create the files and folders following next layout. All files should be empty for now. We will fill them later. Initial project layout: .. code-block:: bash ./ ├── monitoringdaemon/ │ ├── __init__.py │ ├── __main__.py │ └── containers.py ├── config.yml ├── docker-compose.yml ├── Dockerfile └── requirements.txt Initial project layout is ready. We will extend it in the next sections. Let's proceed to the environment preparation. Prepare the environment ----------------------- In this section we are going to prepare the environment for running our daemon. First we need to specify the project requirements. We will use next packages: - ``dependency-injector`` - the dependency injection framework - ``aiohttp`` - the web framework (we need only http client) - ``pyyaml`` - the YAML files parsing library, used for the reading of the configuration files - ``pytest`` - the test framework - ``pytest-asyncio`` - the helper library for the testing of the ``asyncio`` application - ``pytest-cov`` - the helper library for measuring the test coverage Put next lines into the ``requirements.txt`` file: .. code-block:: bash dependency-injector aiohttp pyyaml pytest pytest-asyncio pytest-cov Second, we need to create the ``Dockerfile``. It will describe the daemon's build process and specify how to run it. We will use ``python:3.8-buster`` as a base image. Put next lines into the ``Dockerfile`` file: .. code-block:: bash FROM python:3.8-buster ENV PYTHONUNBUFFERED=1 WORKDIR /code COPY . /code/ RUN apt-get install openssl \ && pip install --upgrade pip \ && pip install -r requirements.txt \ && rm -rf ~/.cache CMD ["python", "-m", "monitoringdaemon"] Third, we need to define the container in the docker-compose configuration. Put next lines into the ``docker-compose.yml`` file: .. code-block:: yaml version: "3.7" services: monitor: build: ./ image: monitoring-daemon volumes: - "./:/code" All is ready. Let's check that the environment is setup properly. Run in the terminal: .. code-block:: bash docker-compose build The build process may take a couple of minutes. You should see something like this in the end: .. code-block:: bash Successfully built 5b4ee5e76e35 Successfully tagged monitoring-daemon:latest After the build is done run the container: .. code-block:: bash docker-compose up The output should look like: .. code-block:: bash Creating network "monitoring-daemon-tutorial_default" with the default driver Creating monitoring-daemon-tutorial_monitor_1 ... done Attaching to monitoring-daemon-tutorial_monitor_1 monitoring-daemon-tutorial_monitor_1 exited with code 0 The environment is ready. The application does not do any work and just exits with a code ``0``. Next step is to configure the logging and configuration file parsing. Logging and configuration ------------------------- In this section we will configure the logging and configuration file parsing. Let's start with the the main part of our application - the container. Container will keep all of the application components and their dependencies. First two components that we're going to add are the config object and the provider for configuring the logging. Put next lines into the ``containers.py`` file: .. code-block:: python """Application containers module.""" import logging import sys from dependency_injector import containers, providers class ApplicationContainer(containers.DeclarativeContainer): """Application container.""" config = providers.Configuration() configure_logging = providers.Callable( logging.basicConfig, stream=sys.stdout, level=config.log.level, format=config.log.format, ) .. note:: We have used the configuration value before it was defined. That's the principle how the ``Configuration`` provider works. Use first, define later. The configuration file will keep the logging settings. Put next lines into the ``config.yml`` file: .. code-block:: yaml log: level: "INFO" format: "[%(asctime)s] [%(levelname)s] [%(name)s]: %(message)s" Now let's create the function that will run our daemon. It's traditionally called ``main()``. The ``main()`` function will create the container. Then it will use the container to parse the ``config.yml`` file and call the logging configuration provider. Put next lines into the ``__main__.py`` file: .. code-block:: python """Main module.""" from .containers import ApplicationContainer def main() -> None: """Run the application.""" container = ApplicationContainer() container.config.from_yaml('config.yml') container.configure_logging() if __name__ == '__main__': main() .. note:: Container is the first object in the application. The container is used to create all other objects. Logging and configuration parsing part is done. In the next section we will create the monitoring checks dispatcher. Dispatcher ---------- Now let's add the monitoring checks dispatcher. The dispatcher will control a list of the monitoring tasks. It will execute each task according to the configured schedule. The ``Monitor`` class is the base class for all the monitors. You can create different monitors by subclassing it and implementing the ``check()`` method. .. image:: asyncio_images/class_1.png Let's create dispatcher and the monitor base classes. Create ``dispatcher.py`` and ``monitors.py`` in the ``monitoringdaemon`` package: .. code-block:: bash :emphasize-lines: 6-7 ./ ├── monitoringdaemon/ │ ├── __init__.py │ ├── __main__.py │ ├── containers.py │ ├── dispatcher.py │ └── monitors.py ├── config.yml ├── docker-compose.yml ├── Dockerfile └── requirements.txt Put next into the ``monitors.py``: .. code-block:: python """Monitors module.""" import logging class Monitor: def __init__(self, check_every: int) -> None: self.check_every = check_every self.logger = logging.getLogger(self.__class__.__name__) async def check(self) -> None: raise NotImplementedError() and next into the ``dispatcher.py``: .. code-block:: python """"Dispatcher module.""" import asyncio import logging import signal import time from typing import List from .monitors import Monitor class Dispatcher: def __init__(self, monitors: List[Monitor]) -> None: self._monitors = monitors self._monitor_tasks: List[asyncio.Task] = [] self._logger = logging.getLogger(self.__class__.__name__) self._stopping = False def run(self) -> None: asyncio.run(self.start()) async def start(self) -> None: self._logger.info('Starting up') for monitor in self._monitors: self._monitor_tasks.append( asyncio.create_task(self._run_monitor(monitor)), ) asyncio.get_event_loop().add_signal_handler(signal.SIGTERM, self.stop) asyncio.get_event_loop().add_signal_handler(signal.SIGINT, self.stop) await asyncio.gather(*self._monitor_tasks, return_exceptions=True) self.stop() def stop(self) -> None: if self._stopping: return self._stopping = True self._logger.info('Shutting down') for task, monitor in zip(self._monitor_tasks, self._monitors): task.cancel() self._logger.info('Shutdown finished successfully') @staticmethod async def _run_monitor(monitor: Monitor) -> None: def _until_next(last: float) -> float: time_took = time.time() - last return monitor.check_every - time_took while True: time_start = time.time() try: await monitor.check() except asyncio.CancelledError: break except Exception: monitor.logger.exception('Error executing monitor check') await asyncio.sleep(_until_next(last=time_start)) Now we need to add the dispatcher to the container. Edit ``containers.py``: .. code-block:: python :emphasize-lines: 8,23-28 """Application containers module.""" import logging import sys from dependency_injector import containers, providers from . import dispatcher class ApplicationContainer(containers.DeclarativeContainer): """Application container.""" config = providers.Configuration() configure_logging = providers.Callable( logging.basicConfig, stream=sys.stdout, level=config.log.level, format=config.log.format, ) dispatcher = providers.Factory( dispatcher.Dispatcher, monitors=providers.List( # TODO: add monitors ), ) .. note:: Every component should be added to the container. At the last we will add the dispatcher in the ``main()`` function. We will retrieve the dispatcher instance from the container and call the ``run()`` method. Edit ``__main__.py``: .. code-block:: python :emphasize-lines: 13-14 """Main module.""" from .containers import ApplicationContainer def main() -> None: """Run the application.""" container = ApplicationContainer() container.config.from_yaml('config.yml') container.configure_logging() dispatcher = container.dispatcher() dispatcher.run() if __name__ == '__main__': main() Finally let's start the daemon to check that all works. Run in the terminal: .. code-block:: bash docker-compose up The output should look like: .. code-block:: bash Starting monitoring-daemon-tutorial_monitor_1 ... done Attaching to monitoring-daemon-tutorial_monitor_1 monitor_1 | [2020-08-08 16:12:35,772] [INFO] [Dispatcher]: Starting up monitor_1 | [2020-08-08 16:12:35,774] [INFO] [Dispatcher]: Shutting down monitor_1 | [2020-08-08 16:12:35,774] [INFO] [Dispatcher]: Shutdown finished successfully monitoring-daemon-tutorial_monitor_1 exited with code 0 Everything works properly. Dispatcher starts up and exits because there are no monitoring tasks. By the end of this section we have the application skeleton ready. In the next section will will add first monitoring task. Example.com monitor ------------------- In this section we will add the monitoring task that will check the availability of the `http://example.com `_. We will start from the extending of our class model with a new type of the monitoring check, the ``HttpMonitor``. The ``HttpMonitor`` is a subclass of the ``Monitor``. We will implement the ``check()`` method that will send the HTTP request to the specified URL. The http request sending will be delegated to the ``HttpClient``. .. image:: asyncio_images/class_2.png First, we need to create the ``HttpClient``. Create ``http.py`` in the ``monitoringdaemon`` package: .. code-block:: bash :emphasize-lines: 7 ./ ├── monitoringdaemon/ │ ├── __init__.py │ ├── __main__.py │ ├── containers.py │ ├── dispatcher.py │ ├── http.py │ └── monitors.py ├── config.yml ├── docker-compose.yml ├── Dockerfile └── requirements.txt and put next into it: .. code-block:: python """Http client module.""" from aiohttp import ClientSession, ClientTimeout, ClientResponse class HttpClient: async def request(self, method: str, url: str, timeout: int) -> ClientResponse: async with ClientSession(timeout=ClientTimeout(timeout)) as session: async with session.request(method, url) as response: return response Now we need to add the ``HttpClient`` to the container. Edit ``containers.py``: .. code-block:: python :emphasize-lines: 8, 23 """Application containers module.""" import logging import sys from dependency_injector import containers, providers from . import http, dispatcher class ApplicationContainer(containers.DeclarativeContainer): """Application container.""" config = providers.Configuration() configure_logging = providers.Callable( logging.basicConfig, stream=sys.stdout, level=config.log.level, format=config.log.format, ) http_client = providers.Factory(http.HttpClient) dispatcher = providers.Factory( dispatcher.Dispatcher, monitors=providers.List( # TODO: add monitors ), ) Now we're ready to add the ``HttpMonitor``. We will add it to the ``monitors`` module. Edit ``monitors.py``: .. code-block:: python :emphasize-lines: 4-5,7,20-56 """Monitors module.""" import logging import time from typing import Dict, Any from .http import HttpClient class Monitor: def __init__(self, check_every: int) -> None: self.check_every = check_every self.logger = logging.getLogger(self.__class__.__name__) async def check(self) -> None: raise NotImplementedError() class HttpMonitor(Monitor): def __init__( self, http_client: HttpClient, options: Dict[str, Any], ) -> None: self._client = http_client self._method = options.pop('method') self._url = options.pop('url') self._timeout = options.pop('timeout') super().__init__(check_every=options.pop('check_every')) async def check(self) -> None: time_start = time.time() response = await self._client.request( method=self._method, url=self._url, timeout=self._timeout, ) time_end = time.time() time_took = time_end - time_start self.logger.info( 'Check\n' ' %s %s\n' ' response code: %s\n' ' content length: %s\n' ' request took: %s seconds\n', self._method, self._url, response.status, response.content_length, round(time_took, 3) ) We have everything ready to add the `http://example.com `_ monitoring check. We make two changes in the container: - Add the factory provider ``example_monitor``. - Inject the ``example_monitor`` into the dispatcher. Edit ``containers.py``: .. code-block:: python :emphasize-lines: 8,25-29,34 """Application containers module.""" import logging import sys from dependency_injector import containers, providers from . import http, monitors, dispatcher class ApplicationContainer(containers.DeclarativeContainer): """Application container.""" config = providers.Configuration() configure_logging = providers.Callable( logging.basicConfig, stream=sys.stdout, level=config.log.level, format=config.log.format, ) http_client = providers.Factory(http.HttpClient) example_monitor = providers.Factory( monitors.HttpMonitor, http_client=http_client, options=config.monitors.example, ) dispatcher = providers.Factory( dispatcher.Dispatcher, monitors=providers.List( example_monitor, ), ) Provider ``example_monitor`` has a dependency on the configuration options. Let's define these options. Edit ``config.yml``: .. code-block:: yaml :emphasize-lines: 5-11 log: level: "INFO" format: "[%(asctime)s] [%(levelname)s] [%(name)s]: %(message)s" monitors: example: method: "GET" url: "http://example.com" timeout: 5 check_every: 5 All set. Start the daemon to check that all works. Run in the terminal: .. code-block:: bash docker-compose up You should see: .. code-block:: bash Starting monitoring-daemon-tutorial_monitor_1 ... done Attaching to monitoring-daemon-tutorial_monitor_1 monitor_1 | [2020-08-08 17:06:41,965] [INFO] [Dispatcher]: Starting up monitor_1 | [2020-08-08 17:06:42,033] [INFO] [HttpMonitor]: Check monitor_1 | GET http://example.com monitor_1 | response code: 200 monitor_1 | content length: 648 monitor_1 | request took: 0.067 seconds monitor_1 | monitor_1 | [2020-08-08 17:06:47,040] [INFO] [HttpMonitor]: Check monitor_1 | GET http://example.com monitor_1 | response code: 200 monitor_1 | content length: 648 monitor_1 | request took: 0.073 seconds Our daemon can monitor `http://example.com `_ availability. Let's add the monitor for the `http://httpbin.org `_. Httpbin.org monitor ------------------- Adding of the monitor for the `httpbin.org`_ will be much easier because we have all the components ready. We just need to create a new provider in the container and update the configuration. Edit ``containers.py``: .. code-block:: python :emphasize-lines: 31-35,41 """Application containers module.""" import logging import sys from dependency_injector import containers, providers from . import http, monitors, dispatcher class ApplicationContainer(containers.DeclarativeContainer): """Application container.""" config = providers.Configuration() configure_logging = providers.Callable( logging.basicConfig, stream=sys.stdout, level=config.log.level, format=config.log.format, ) http_client = providers.Factory(http.HttpClient) example_monitor = providers.Factory( monitors.HttpMonitor, http_client=http_client, options=config.monitors.example, ) httpbin_monitor = providers.Factory( monitors.HttpMonitor, http_client=http_client, options=config.monitors.httpbin, ) dispatcher = providers.Factory( dispatcher.Dispatcher, monitors=providers.List( example_monitor, httpbin_monitor, ), ) Edit ``config.yml``: .. code-block:: yaml :emphasize-lines: 13-17 log: level: "INFO" format: "[%(asctime)s] [%(levelname)s] [%(name)s]: %(message)s" monitors: example: method: "GET" url: "http://example.com" timeout: 5 check_every: 5 httpbin: method: "GET" url: "https://httpbin.org/get" timeout: 5 check_every: 5 Let's start the daemon and check the logs. Run in the terminal: .. code-block:: bash docker-compose up You should see: .. code-block:: bash Starting monitoring-daemon-tutorial_monitor_1 ... done Attaching to monitoring-daemon-tutorial_monitor_1 monitor_1 | [2020-08-08 18:09:08,540] [INFO] [Dispatcher]: Starting up monitor_1 | [2020-08-08 18:09:08,618] [INFO] [HttpMonitor]: Check monitor_1 | GET http://example.com monitor_1 | response code: 200 monitor_1 | content length: 648 monitor_1 | request took: 0.077 seconds monitor_1 | monitor_1 | [2020-08-08 18:09:08,722] [INFO] [HttpMonitor]: Check monitor_1 | GET https://httpbin.org/get monitor_1 | response code: 200 monitor_1 | content length: 310 monitor_1 | request took: 0.18 seconds monitor_1 | monitor_1 | [2020-08-08 18:09:13,619] [INFO] [HttpMonitor]: Check monitor_1 | GET http://example.com monitor_1 | response code: 200 monitor_1 | content length: 648 monitor_1 | request took: 0.066 seconds monitor_1 | monitor_1 | [2020-08-08 18:09:13,681] [INFO] [HttpMonitor]: Check monitor_1 | GET https://httpbin.org/get monitor_1 | response code: 200 monitor_1 | content length: 310 monitor_1 | request took: 0.126 seconds The functional part is done. Daemon monitors `http://example.com `_ and `https://httpbin.org `_. In the next section we will add some tests. Tests ----- It would be nice to add some tests. Let's do it. We will use `pytest `_ and `coverage `_. Create ``tests.py`` in the ``monitoringdaemon`` package: .. code-block:: bash :emphasize-lines: 9 ./ ├── monitoringdaemon/ │ ├── __init__.py │ ├── __main__.py │ ├── containers.py │ ├── dispatcher.py │ ├── http.py │ ├── monitors.py │ └── tests.py ├── config.yml ├── docker-compose.yml ├── Dockerfile └── requirements.txt and put next into it: .. code-block:: python :emphasize-lines: 54,70-71 """Tests module.""" import asyncio import dataclasses from unittest import mock import pytest from .containers import ApplicationContainer @dataclasses.dataclass class RequestStub: status: int content_length: int @pytest.fixture def container(): container = ApplicationContainer() container.config.from_dict({ 'log': { 'level': 'INFO', 'formant': '[%(asctime)s] [%(levelname)s] [%(name)s]: %(message)s', }, 'monitors': { 'example': { 'method': 'GET', 'url': 'http://fake-example.com', 'timeout': 1, 'check_every': 1, }, 'httpbin': { 'method': 'GET', 'url': 'https://fake-httpbin.org/get', 'timeout': 1, 'check_every': 1, }, }, }) return container @pytest.mark.asyncio async def test_example_monitor(container, caplog): caplog.set_level('INFO') http_client_mock = mock.AsyncMock() http_client_mock.request.return_value = RequestStub( status=200, content_length=635, ) with container.http_client.override(http_client_mock): example_monitor = container.example_monitor() await example_monitor.check() assert 'http://fake-example.com' in caplog.text assert 'response code: 200' in caplog.text assert 'content length: 635' in caplog.text @pytest.mark.asyncio async def test_dispatcher(container, caplog, event_loop): caplog.set_level('INFO') example_monitor_mock = mock.AsyncMock() httpbin_monitor_mock = mock.AsyncMock() with container.example_monitor.override(example_monitor_mock), \ container.httpbin_monitor.override(httpbin_monitor_mock): dispatcher = container.dispatcher() event_loop.create_task(dispatcher.start()) await asyncio.sleep(0.1) dispatcher.stop() assert example_monitor_mock.check.called assert httpbin_monitor_mock.check.called Run in the terminal: .. code-block:: bash docker-compose run --rm monitor py.test monitoringdaemon/tests.py --cov=monitoringdaemon You should see: .. code-block:: bash platform linux -- Python 3.8.3, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 rootdir: /code plugins: asyncio-0.14.0, cov-2.10.0 collected 2 items monitoringdaemon/tests.py .. [100%] ----------- coverage: platform linux, python 3.8.3-final-0 ----------- Name Stmts Miss Cover ---------------------------------------------------- monitoringdaemon/__init__.py 0 0 100% monitoringdaemon/__main__.py 9 9 0% monitoringdaemon/containers.py 11 0 100% monitoringdaemon/dispatcher.py 43 5 88% monitoringdaemon/http.py 6 3 50% monitoringdaemon/monitors.py 23 1 96% monitoringdaemon/tests.py 37 0 100% ---------------------------------------------------- TOTAL 129 18 86% .. note:: Take a look at the highlights in the ``tests.py``. In the ``test_example_monitor`` it emphasizes the overriding of the ``HttpClient``. The real HTTP calls are mocked. In the ``test_dispatcher`` we override both monitors with the mocks. Conclusion ---------- In this tutorial we've built an ``asyncio`` monitoring daemon following the dependency injection principle. We've used the ``Dependency Injector`` as a dependency injection framework. The benefit you get with the ``Dependency Injector`` is the container. It starts to payoff when you need to understand or change your application structure. It's easy with the container, cause you have everything defined explicitly in one place: .. code-block:: python """Application containers module.""" import logging import sys from dependency_injector import containers, providers from . import http, monitors, dispatcher class ApplicationContainer(containers.DeclarativeContainer): """Application container.""" config = providers.Configuration() configure_logging = providers.Callable( logging.basicConfig, stream=sys.stdout, level=config.log.level, format=config.log.format, ) http_client = providers.Factory(http.HttpClient) example_monitor = providers.Factory( monitors.HttpMonitor, http_client=http_client, options=config.monitors.example, ) httpbin_monitor = providers.Factory( monitors.HttpMonitor, http_client=http_client, options=config.monitors.httpbin, ) dispatcher = providers.Factory( dispatcher.Dispatcher, monitors=providers.List( example_monitor, httpbin_monitor, ), ) What's next? - Look at the other :ref:`tutorials` - Know more about the :ref:`providers` - Go to the :ref:`contents` .. disqus::