diff --git a/docs/providers/selector.rst b/docs/providers/selector.rst index 883e4fa0..29867217 100644 --- a/docs/providers/selector.rst +++ b/docs/providers/selector.rst @@ -1,3 +1,5 @@ +.. _selector-provider: + Selector providers ------------------ diff --git a/docs/tutorials/cli-images/classes_01.png b/docs/tutorials/cli-images/classes_01.png new file mode 100644 index 00000000..ee4575a5 Binary files /dev/null and b/docs/tutorials/cli-images/classes_01.png differ diff --git a/docs/tutorials/cli-images/classes_02.png b/docs/tutorials/cli-images/classes_02.png new file mode 100644 index 00000000..f768d187 Binary files /dev/null and b/docs/tutorials/cli-images/classes_02.png differ diff --git a/docs/tutorials/cli.rst b/docs/tutorials/cli.rst new file mode 100644 index 00000000..6b09eff9 --- /dev/null +++ b/docs/tutorials/cli.rst @@ -0,0 +1,1053 @@ +.. _cli-tutorial: + +CLI application tutorial +======================== + +.. meta:: + :keywords: Python,CLI,Tutorial,Education,Web,Example,DI,Dependency injection,IoC, + Inversion of control,Refactoring,Tests,Unit tests,Pytest,py.test + :description: This tutorial shows how to build a CLI application following the dependency + injection principle. You will create the CLI script, use CSV files and sqlite + database, cover the application with the unit the tests and make some refactoring. + +This tutorial shows how to build a CLI application following the dependency injection +principle. + +Start from the scratch or jump to the section: + +.. contents:: + :local: + :backlinks: none + +You can find complete project on the +`Github `_. + +What are we going to build? +--------------------------- + +We will build a CLI application that helps to search for the movies. Let's call it Movie Lister. + +How does Movie Lister work? + +- Application uses a movies database to search for the movies +- Application can search the movies by: + - Director's name + - Year of the release +- Each movie has next fields: + - Title + - Year of the release + - Director's name +- The database can be in the next formats: + - Csv + - Sqlite +- Other database formats can be added later + +Movie Lister is a naive example from Martin Fowler's article about the dependency injection and +inversion of control: + + http://www.martinfowler.com/articles/injection.html + +Here is a class diagram of the Movie Lister application: + +.. image:: cli-images/classes_01.png + +The responsibilities are split that way: + +- ``MovieLister`` - is responsible for the search +- ``MovieFinder`` - is responsible for the fetching from the database +- ``Movie`` - the movie entity + +Prepare the environment +----------------------- + +Let's create the environment for the project. + +First we need to create a project folder and the virtual environment: + +.. code-block:: bash + + mkdir movie-lister-tutorial + cd movie-lister-tutorial + python3 -m venv venv + +Now let's activate the virtual environment: + +.. code-block:: bash + + . venv/bin/activate + +Project layout +-------------- + +Create next structure in the project root directory. All files are empty. That's ok for now. + +Initial project layout: + +.. code-block:: bash + + ./ + ├── movies/ + │ ├── __init__.py + │ ├── __main__.py + │ └── containers.py + ├── venv/ + ├── config.yml + └── requirements.txt + +Move on to the project requirements. + +Install the requirements +------------------------ + +Now it's time to install the project requirements. We will use next packages: + +- ``dependency-injector`` - the dependency injection framework +- ``pyyaml`` - the YAML files parsing library, used for the reading of the configuration files +- ``pytest`` - the test framework +- ``pytest-cov`` - the helper library for measuring the test coverage + +Put next lines into the ``requirements.txt`` file: + +.. code-block:: bash + + dependency-injector + pyyaml + pytest + pytest-cov + +and run next in the terminal: + +.. code-block:: bash + + pip install -r requirements.txt + +The requirements are setup. Now we will add the fixtures. + +Fixtures +-------- + +In this section we will add the fixtures. + +We will create a script that creates database files. + +First add the folder ``data/`` in the root of the project and then add the file +``fixtures.py`` inside of it: + +.. code-block:: bash + :emphasize-lines: 2-3 + + ./ + ├── data/ + │ └── fixtures.py + ├── movies/ + │ ├── __init__.py + │ ├── __main__.py + │ └── containers.py + ├── venv/ + ├── config.yml + └── requirements.txt + +Second put next in the ``fixtures.py``: + +.. code-block:: python + + """Fixtures module.""" + + import csv + import sqlite3 + import pathlib + + + SAMPLE_DATA = [ + ('The Hunger Games: Mockingjay - Part 2', 2015, 'Francis Lawrence'), + ('Rogue One: A Star Wars Story', 2016, 'Gareth Edwards'), + ('The Jungle Book', 2016, 'Jon Favreau'), + ] + + FILE = pathlib.Path(__file__) + DIR = FILE.parent + CSV_FILE = DIR / 'movies.csv' + SQLITE_FILE = DIR / 'movies.db' + + + def create_csv(movies_data, path): + with open(path, 'w') as opened_file: + writer = csv.writer(opened_file) + for row in movies_data: + writer.writerow(row) + + + def create_sqlite(movies_data, path): + with sqlite3.connect(path) as db: + db.execute( + 'CREATE TABLE IF NOT EXISTS movies ' + '(title text, year int, director text)' + ) + db.execute('DELETE FROM movies') + db.executemany('INSERT INTO movies VALUES (?,?,?)', movies_data) + + + def main(): + create_csv(SAMPLE_DATA, CSV_FILE) + create_sqlite(SAMPLE_DATA, SQLITE_FILE) + print('OK') + + + if __name__ == '__main__': + main() + +Now run in the terminal: + +.. code-block:: bash + + python data/fixtures.py + +You should see: + +.. code-block:: bash + + OK + +Check that files ``movies.csv`` and ``movies.db`` have appeared in the ``data/`` folder: + +.. code-block:: bash + :emphasize-lines: 4-5 + + ./ + ├── data/ + │ ├── fixtures.py + │ ├── movies.csv + │ └── movies.db + ├── movies/ + │ ├── __init__.py + │ ├── __main__.py + │ └── containers.py + ├── venv/ + ├── config.yml + └── requirements.txt + +Fixtures are created. Let's move on. + +Container +--------- + +In this section we will add the main part of our application - the container. + +Container will keep all of the application components and their dependencies. + +Edit ``containers.py``: + +.. code-block:: python + + """Containers module.""" + + from dependency_injector import containers + + + class ApplicationContainer(containers.DeclarativeContainer): + ... + +Container is empty for now. We will add the providers in the following sections. + +Let's also create the ``main()`` function. Its responsibility is to run our application. For now +it will just create the container. + +Edit ``__main__.py``: + +.. code-block:: python + + """Main module.""" + + from .containers import ApplicationContainer + + + def main(): + container = ApplicationContainer() + + + if __name__ == '__main__': + main() + +.. note:: + + Container is the first object in the application. + + The container is used to create all other objects. + +Csv finder +---------- + +In this section we will build everything we need for working with the csv file formats. + +We will add: + +- The ``Movie`` entity +- The ``MovieFinder`` base class +- The ``CsvMovieFinder`` finder implementation +- The ``MovieLister`` class + +After each step we will add the provider to the container. + +.. image:: cli-images/classes_02.png + +Create the ``entities.py`` in the ``movies`` package: + +.. code-block:: bash + :emphasize-lines: 10 + + ./ + ├── data/ + │ ├── fixtures.py + │ ├── movies.csv + │ └── movies.db + ├── movies/ + │ ├── __init__.py + │ ├── __main__.py + │ ├── containers.py + │ └── entities.py + ├── venv/ + ├── config.yml + └── requirements.txt + +and put next into it: + +.. code-block:: python + + """Movie entities module.""" + + + class Movie: + + def __init__(self, title: str, year: int, director: str): + self.title = str(title) + self.year = int(year) + self.director = str(director) + + def __repr__(self): + return '{0}(title={1}, year={2}, director={3})'.format( + self.__class__.__name__, + repr(self.title), + repr(self.year), + repr(self.director), + ) + +Now we need to add the ``Movie`` factory to the container. We need to add import of the +``providers`` module from the ``dependency_injector`` package, import ``entities`` module. + +Edit ``containers.py``: + +.. code-block:: python + :emphasize-lines: 3,5,9 + + """Containers module.""" + + from dependency_injector import containers, providers + + from . import entities + + class ApplicationContainer(containers.DeclarativeContainer): + + movie = providers.Factory(entities.Movie) + +.. note:: + + Don't forget to remove the Ellipsis ``...`` from the container. We don't need it anymore + since we container is not empty. + +Let's move on to the finders. + +Create the ``finders.py`` in the ``movies`` package: + +.. code-block:: bash + :emphasize-lines: 11 + + ./ + ├── data/ + │ ├── fixtures.py + │ ├── movies.csv + │ └── movies.db + ├── movies/ + │ ├── __init__.py + │ ├── __main__.py + │ ├── containers.py + │ ├── entities.py + │ └── finders.py + ├── venv/ + ├── config.yml + └── requirements.txt + +and put next into it: + +.. code-block:: python + + """Movie finders module.""" + + import csv + from typing import Callable, List + + from .entities import Movie + + + class MovieFinder: + + def __init__(self, movie_factory: Callable[..., Movie]) -> None: + self._movie_factory = movie_factory + + def find_all(self) -> List[Movie]: + raise NotImplementedError() + + + class CsvMovieFinder(MovieFinder): + + def __init__( + self, + movie_factory: Callable[..., Movie], + path: str, + delimiter: str, + ) -> None: + self._csv_file_path = path + self._delimiter = delimiter + super().__init__(movie_factory) + + def find_all(self) -> List[Movie]: + with open(self._csv_file_path) as csv_file: + csv_reader = csv.reader(csv_file, delimiter=self._delimiter) + return [self._movie_factory(*row) for row in csv_reader] + +Now let's add the csv finder into the container. + +Edit ``containers.py``: + +.. code-block:: python + :emphasize-lines: 5,9,13-18 + + """Containers module.""" + + from dependency_injector import containers, providers + + from . import finders, entities + + class ApplicationContainer(containers.DeclarativeContainer): + + config = providers.Configuration() + + movie = providers.Factory(entities.Movie) + + csv_finder = providers.Singleton( + finders.CsvMovieFinder, + movie_factory=movie.provider, + path=config.finder.csv.path, + delimiter=config.finder.csv.delimiter, + ) + +The csv finder needs the movie factory. It needs it to create the ``Movie`` entities when +reads the csv rows. To provide the factory we use ``.provider`` factory attribute. +This is also called the delegation of the provider. If we just pass the movie factory +as the dependency, it will be called when csv finder is created and the ``Movie`` instance will +be injected. With the ``.provider`` attribute the provider itself will be injected. + +The csv finder also has a few dependencies on the configuration options. We added configuration +provider to provide these dependencies. + +.. note:: + + We have used the configuration value before it was defined. That's the principle how the + Configuration provider works. + + Use first, define later. + +Not let's define the configuration values. + +Edit ``config.yml``: + +.. code-block:: yaml + + finder: + + csv: + path: "data/movies.csv" + delimiter: "," + +The configuration file is ready. Now let's update the ``main()`` function to specify its location. + +Edit ``__main__.py``: + +.. code-block:: python + :emphasize-lines: 9 + + """Main module.""" + + from .containers import ApplicationContainer + + + def main(): + container = ApplicationContainer() + + container.config.from_yaml('config.yml') + + + if __name__ == '__main__': + main() + +Move on to the lister. + +Create the ``listers.py`` in the ``movies`` package: + +.. code-block:: bash + :emphasize-lines: 12 + + ./ + ├── data/ + │ ├── fixtures.py + │ ├── movies.csv + │ └── movies.db + ├── movies/ + │ ├── __init__.py + │ ├── __main__.py + │ ├── containers.py + │ ├── entities.py + │ ├── finders.py + │ └── listers.py + ├── venv/ + ├── config.yml + └── requirements.txt + +and put next into it: + +.. code-block:: python + + """Movie listers module.""" + + from .finders import MovieFinder + + + class MovieLister: + + def __init__(self, movie_finder: MovieFinder): + self._movie_finder = movie_finder + + def movies_directed_by(self, director): + return [ + movie for movie in self._movie_finder.find_all() + if movie.director == director + ] + + def movies_released_in(self, year): + return [ + movie for movie in self._movie_finder.find_all() + if movie.year == year + ] + +and edit ``containers.py``: + +.. code-block:: python + :emphasize-lines: 5,20-23 + + """Containers module.""" + + from dependency_injector import containers, providers + + from . import finders, listers, entities + + class ApplicationContainer(containers.DeclarativeContainer): + + config = providers.Configuration() + + movie = providers.Factory(entities.Movie) + + csv_finder = providers.Singleton( + finders.CsvMovieFinder, + movie_factory=movie.provider, + path=config.finder.csv.path, + delimiter=config.finder.csv.delimiter, + ) + + lister = providers.Factory( + listers.MovieLister, + movie_finder=csv_finder, + ) + +All the components are created and added to the container. + +Finally let's update the ``main()`` function. + +Edit ``__main__.py``: + +.. code-block:: python + :emphasize-lines: 11-20 + + """Main module.""" + + from .containers import ApplicationContainer + + + def main(): + container = ApplicationContainer() + + container.config.from_yaml('config.yml') + + lister = container.lister() + + print( + 'Francis Lawrence movies:', + lister.movies_directed_by('Francis Lawrence'), + ) + print( + '2016 movies:', + lister.movies_released_in(2016), + ) + + + if __name__ == '__main__': + main() + +All set. Now we run the application. + +Run in the terminal: + +.. code-block:: bash + + python -m movies + +You should see: + +.. code-block:: bash + + Francis Lawrence movies: [Movie(title='The Hunger Games: Mockingjay - Part 2', year=2015, director='Francis Lawrence')] + 2016 movies: [Movie(title='Rogue One: A Star Wars Story', year=2016, director='Gareth Edwards'), Movie(title='The Jungle Book', year=2016, director='Jon Favreau')] + +Our application can work with the movies database in the csv format. We also need to support +the sqlite format. We will deal with it in the next section. + +Sqlite finder +------------- + +In this section we will add another type of the finder - the sqlite finder. + +Let's get to work. + +Edit ``finders.py``: + +.. code-block:: python + :emphasize-lines: 4,37-50 + + """Movie finders module.""" + + import csv + import sqlite3 + from typing import Callable, List + + from .entities import Movie + + + class MovieFinder: + + def __init__(self, movie_factory: Callable[..., Movie]) -> None: + self._movie_factory = movie_factory + + def find_all(self) -> List[Movie]: + raise NotImplementedError() + + + class CsvMovieFinder(MovieFinder): + + def __init__( + self, + movie_factory: Callable[..., Movie], + path: str, + delimiter: str, + ) -> None: + self._csv_file_path = path + self._delimiter = delimiter + super().__init__(movie_factory) + + def find_all(self) -> List[Movie]: + with open(self._csv_file_path) as csv_file: + csv_reader = csv.reader(csv_file, delimiter=self._delimiter) + return [self._movie_factory(*row) for row in csv_reader] + + + class SqliteMovieFinder(MovieFinder): + + def __init__( + self, + movie_factory: Callable[..., Movie], + path: str, + ) -> None: + self._database = sqlite3.connect(path) + super().__init__(movie_factory) + + def find_all(self) -> List[Movie]: + with self._database as db: + rows = db.execute('SELECT title, year, director FROM movies') + return [self._movie_factory(*row) for row in rows] + +Now we need to add the sqlite finder to the container and update lister's dependency to use it. + +Edit ``containers.py``: + +.. code-block:: python + :emphasize-lines: 20-24,28 + + """Containers module.""" + + from dependency_injector import containers, providers + + from . import finders, listers, entities + + class ApplicationContainer(containers.DeclarativeContainer): + + config = providers.Configuration() + + movie = providers.Factory(entities.Movie) + + csv_finder = providers.Singleton( + finders.CsvMovieFinder, + movie_factory=movie.provider, + path=config.finder.csv.path, + delimiter=config.finder.csv.delimiter, + ) + + sqlite_finder = providers.Singleton( + finders.SqliteMovieFinder, + movie_factory=movie.provider, + path=config.finder.sqlite.path, + ) + + lister = providers.Factory( + listers.MovieLister, + movie_finder=sqlite_finder, + ) + +The sqlite finder has a dependency on the configuration option. Let's update the configuration +file. + +Edit ``config.yml``: + +.. code-block:: yaml + :emphasize-lines: 7-8 + + finder: + + csv: + path: "data/movies.csv" + delimiter: "," + + sqlite: + path: "data/movies.db" + +All is ready. Let's check. + +Run in the terminal: + +.. code-block:: bash + + python -m movies + +You should see: + +.. code-block:: bash + + Francis Lawrence movies: [Movie(title='The Hunger Games: Mockingjay - Part 2', year=2015, director='Francis Lawrence')] + 2016 movies: [Movie(title='Rogue One: A Star Wars Story', year=2016, director='Gareth Edwards'), Movie(title='The Jungle Book', year=2016, director='Jon Favreau')] + +Our application now supports both formats: csv files and sqlite databases. Every time when we +need to work with the different format we need to make a code change in the container. We will +improve this in the next section. + +Selector +-------- + +In this section we will make our application more flexible. + +The code change will not be needed to switch between csv and sqlite formats. We implement the +switch based on the environment variable ``MOVIE_FINDER_TYPE``: + +- When ``MOVIE_FINDER_TYPE=csv`` application uses csv finder. +- When ``MOVIE_FINDER_TYPE=sqlite`` application uses sqlite finder. + +We will use the ``Selector`` provider. It selects the provider based on the configuration option +(docs - :ref:`selector-provider`). + +Edit ``containers.py``: + +.. code-block:: python + :emphasize-lines: 27-31,35 + + """Containers module.""" + + from dependency_injector import containers, providers + + from . import finders, listers, entities + + + class ApplicationContainer(containers.DeclarativeContainer): + + config = providers.Configuration() + + movie = providers.Factory(entities.Movie) + + csv_finder = providers.Singleton( + finders.CsvMovieFinder, + movie_factory=movie.provider, + path=config.finder.csv.path, + delimiter=config.finder.csv.delimiter, + ) + + sqlite_finder = providers.Singleton( + finders.SqliteMovieFinder, + movie_factory=movie.provider, + path=config.finder.sqlite.path, + ) + + finder = providers.Selector( + config.finder.type, + csv=csv_finder, + sqlite=sqlite_finder, + ) + + lister = providers.Factory( + listers.MovieLister, + movie_finder=finder, + ) + +The switch is the ``config.finder.type`` option. When its value is ``csv``, the provider under +``csv`` key is used. The same is for ``sqlite``. + +Now we need to read the value of the ``config.finder.type`` option from the environment variable +``MOVIE_FINDER_TYPE``. + +Edit ``__main__.py``: + +.. code-block:: python + :emphasize-lines: 10 + + """Main module.""" + + from .containers import ApplicationContainer + + + def main(): + container = ApplicationContainer() + + container.config.from_yaml('config.yml') + container.config.finder.type.from_env('MOVIE_FINDER_TYPE') + + lister = container.lister() + + print( + 'Francis Lawrence movies:', + lister.movies_directed_by('Francis Lawrence'), + ) + print( + '2016 movies:', + lister.movies_released_in(2016), + ) + + + if __name__ == '__main__': + main() + +Done. + +Run in the terminal line by line: + +.. code-block:: bash + + MOVIE_FINDER_TYPE=csv python -m movies + MOVIE_FINDER_TYPE=sqlite python -m movies + +The output should be something like this for each command: + +.. code-block:: bash + + Francis Lawrence movies: [Movie(title='The Hunger Games: Mockingjay - Part 2', year=2015, director='Francis Lawrence')] + 2016 movies: [Movie(title='Rogue One: A Star Wars Story', year=2016, director='Gareth Edwards'), Movie(title='The Jungle Book', year=2016, director='Jon Favreau')] + +In the next section we will add some tests. + +Tests +----- + +It would be nice to add some tests. Let's do it. + +We will use `pytest `_ and +`coverage `_. + +Create ``tests.py`` in the ``movies`` package: + +.. code-block:: bash + :emphasize-lines: 13 + + ./ + ├── data/ + │ ├── fixtures.py + │ ├── movies.csv + │ └── movies.db + ├── movies/ + │ ├── __init__.py + │ ├── __main__.py + │ ├── containers.py + │ ├── entities.py + │ ├── finders.py + │ ├── listers.py + │ └── tests.py + ├── venv/ + ├── config.yml + └── requirements.txt + +and put next into it: + +.. code-block:: python + :emphasize-lines: 35,50 + + """Tests module.""" + + from unittest import mock + + import pytest + + from .containers import ApplicationContainer + + + @pytest.fixture + def container(): + container = ApplicationContainer() + container.config.from_dict({ + 'finder': { + 'type': 'csv', + 'csv': { + 'path': '/fake-movies.csv', + 'delimiter': ',', + }, + 'sqlite': { + 'path': '/fake-movies.db', + }, + }, + }) + return container + + + def test_movies_directed_by(container): + finder_mock = mock.Mock() + finder_mock.find_all.return_value = [ + container.movie('The 33', 2015, 'Patricia Riggen'), + container.movie('The Jungle Book', 2016, 'Jon Favreau'), + ] + + with container.finder.override(finder_mock): + lister = container.lister() + movies = lister.movies_directed_by('Jon Favreau') + + assert len(movies) == 1 + assert movies[0].title == 'The Jungle Book' + + + def test_movies_released_in(container): + finder_mock = mock.Mock() + finder_mock.find_all.return_value = [ + container.movie('The 33', 2015, 'Patricia Riggen'), + container.movie('The Jungle Book', 2016, 'Jon Favreau'), + ] + + with container.finder.override(finder_mock): + lister = container.lister() + movies = lister.movies_released_in(2015) + + assert len(movies) == 1 + assert movies[0].title == 'The 33' + +Run in the terminal: + +.. code-block:: bash + + pytest movies/tests.py --cov=movies + +You should see: + +.. code-block:: bash + + platform darwin -- Python 3.8.3, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 + plugins: cov-2.10.0 + collected 2 items + + movies/tests.py .. [100%] + + ---------- coverage: platform darwin, python 3.8.3-final-0 ----------- + Name Stmts Miss Cover + ------------------------------------------ + movies/__init__.py 0 0 100% + movies/__main__.py 10 10 0% + movies/containers.py 9 0 100% + movies/entities.py 7 1 86% + movies/finders.py 26 13 50% + movies/listers.py 8 0 100% + movies/tests.py 24 0 100% + ------------------------------------------ + TOTAL 84 24 71% + +.. note:: + + Take a look at the highlights in the ``tests.py``. + + We use ``.override()`` method of the ``finder`` provider. Provider is overridden by the mock. + Every time when any other provider will request ``finder`` provider to provide the dependency, + the mock will be returned. So when we call the ``lister`` provider, the ``MovieLister`` + instance is created with the mock, not an actual ``MovieFinder``. + +Conclusion +---------- + +In this tutorial we've built a CLI application following the dependency injection principle. +We've used the ``Dependency Injector`` as a dependency injection framework. + +The benefit you get with the ``Dependency Injector`` is the container. It starts to payoff +when you need to understand or change your application structure. It's easy with the container, +cause you have everything defined explicitly in one place: + +.. code-block:: python + + """Containers module.""" + + from dependency_injector import containers, providers + + from . import finders, listers, entities + + + class ApplicationContainer(containers.DeclarativeContainer): + + config = providers.Configuration() + + movie = providers.Factory(entities.Movie) + + csv_finder = providers.Singleton( + finders.CsvMovieFinder, + movie_factory=movie.provider, + path=config.finder.csv.path, + delimiter=config.finder.csv.delimiter, + ) + + sqlite_finder = providers.Singleton( + finders.SqliteMovieFinder, + movie_factory=movie.provider, + path=config.finder.sqlite.path, + ) + + finder = providers.Selector( + config.finder.type, + csv=csv_finder, + sqlite=sqlite_finder, + ) + + lister = providers.Factory( + listers.MovieLister, + movie_finder=finder, + ) + +What's next? + +- Look at the other :ref:`tutorials`. +- Know more about the :ref:`providers`. +- Go to the :ref:`contents`. + +.. disqus:: diff --git a/docs/tutorials/index.rst b/docs/tutorials/index.rst index bfe6a2c2..f686fe4f 100644 --- a/docs/tutorials/index.rst +++ b/docs/tutorials/index.rst @@ -12,5 +12,6 @@ frameworks. flask aiohttp asyncio-daemon + cli .. disqus::