From ea5ee0f04786bd8a1808933bf0d101bf83ecfd89 Mon Sep 17 00:00:00 2001 From: Radoslav Georgiev Date: Tue, 16 Nov 2021 17:05:31 +0200 Subject: [PATCH 1/5] Reiterate over the initial APIs & Serializers paragraphs --- README.md | 35 +++++++++++++++++++++++++---------- 1 file changed, 25 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index b629e71..9f2f8ed 100644 --- a/README.md +++ b/README.md @@ -574,17 +574,32 @@ class ItemBuyTests(TestCase): When using services & selectors, all of your APIs should look simple & identical. -General rules for an API is: +**General rules for an API is:** -* Do 1 API per operation. For CRUD on a model, this means 4 APIs. -* Use the most simple `APIView` or `GenericAPIView` -* Use services / selectors & don't do business logic in your API. -* Use serializers for fetching objects from params - passed either via `GET` or `POST` -* Serializer should be nested in the API and be named either `InputSerializer` or `OutputSerializer` - * `OutputSerializer` can subclass `ModelSerializer`, if needed. - * `InputSerializer` should always be a plain `Serializer` - * Reuse serializers as little as possible - * If you need a nested serializer, use the `inline_serializer` util +* Have 1 API per operation. This means, for CRUD on a model, having 4 APIs. +* Inherit from the most simple `APIView` or `GenericAPIView`. + * Avoid the more abstract classes, since they tend to manage things via serializers & we want to do that via services & selectors. +* **Don't do business logic in your API.** +* You can do object fetching / data manipulation in your APIs (potentially, you can extract that to somewhere else). + * If you are calling `some_service` in your API, you can extract data manipulation to `some_service_parse`. +* Basically, keep the APIs are simple as possible. They are an interface towards your core business logic. + +When we are talking about APIs, we need a way to parse data in & parse data out. + +**Here are our rules for API serialization:** + +* There should be a dedicated **input serializer** & a dedicated **output serializer**. +* **Input serializer** takes care of the data coming in. +* **Output serializer** takes care of the data coming out. +* Use whatever abstraction works for you, in terms of serialization. + +**In case you are using DRF's serializers, here are our rules:** + +* Serializer should be nested in the API and be named either `InputSerializer` or `OutputSerializer`. +* Our preference is for both serializers to inherit from the simpler `serializer.Serializers` and avoid using `serializers.ModelSerializer` +* If you need a nested serializer, use the `inline_serializer` util. +* Reuse serializers as little as possible. + * Once you start reusing serializers by inheriting them, you'll be exposed to unexpected behaviors, when something in the base serializer changes. ### Naming convention From f200a06334f4b1e6a6055581fca3c47e20b75774 Mon Sep 17 00:00:00 2001 From: Radoslav Georgiev Date: Tue, 16 Nov 2021 17:09:44 +0200 Subject: [PATCH 2/5] Change `ModelSerializer` to `Serializer` in examples --- README.md | 35 ++++++++++++++++------------------- 1 file changed, 16 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index 9f2f8ed..74128bc 100644 --- a/README.md +++ b/README.md @@ -623,13 +623,9 @@ from styleguide_example.users.models import BaseUser class UserListApi(APIView): - class OutputSerializer(serializers.ModelSerializer): - class Meta: - model = BaseUser - fields = ( - 'id', - 'email' - ) + class OutputSerializer(serializers.Serializer): + id = serializers.CharField() + email = serializers.CharField() def get(self, request): users = user_list() @@ -677,14 +673,10 @@ class UserListApi(ApiErrorsMixin, APIView): is_admin = serializers.NullBooleanField(required=False) email = serializers.EmailField(required=False) - class OutputSerializer(serializers.ModelSerializer): - class Meta: - model = BaseUser - fields = ( - 'id', - 'email', - 'is_admin' - ) + class OutputSerializer(serializers.Serializer): + id = serializers.CharField() + email = serializers.CharField() + is_admin = serializers.BooleanField() def get(self, request): # Make sure the filters are valid, if passed @@ -806,10 +798,11 @@ You can find the code for the example list API with filters & pagination in the ```python class CourseDetailApi(SomeAuthenticationMixin, APIView): - class OutputSerializer(serializers.ModelSerializer): - class Meta: - model = Course - fields = ('id', 'name', 'start_date', 'end_date') + class OutputSerializer(serializers.Serializer): + id = serializers.CharField() + name = serializers.CharField() + start_date = serializers.DateField() + end_date = serializers.DateField() def get(self, request, course_id): course = get_course(id=course_id) @@ -869,6 +862,10 @@ class Serializer(serializers.Serializer): The implementation of `inline_serializer` can be found [here](https://github.com/HackSoftware/Styleguide-Example/blob/master/styleguide_example/common/utils.py#L34), in the [Styleguide-Example](https://github.com/HackSoftware/Styleguide-Example) repo. +### Advanced serialization + +*Coming soon* + ## Urls We usually organize our urls the same way we organize our APIs - 1 url per API, meaning 1 url per action. From 4a5a6e36043f9f36336931b544293d1824ee9049 Mon Sep 17 00:00:00 2001 From: Radoslav Georgiev Date: Tue, 16 Nov 2021 17:26:51 +0200 Subject: [PATCH 3/5] Expand section for advanced serialization --- README.md | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 69 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 74128bc..a63a0f2 100644 --- a/README.md +++ b/README.md @@ -864,7 +864,75 @@ The implementation of `inline_serializer` can be found [here](https://github.com ### Advanced serialization -*Coming soon* +Sometimes, the end result of an API can be quite complex. Sometimes, we want to optimize the queries that we do and the optimization itself is quite complex. + +Trying to stick with just an `OutputSerializer` in that case might limit our options. + +In those cases, we can implement our output serialization as a function, and have the optimizations we need there. + +Lets take this API as an example: + +```python +class SomeGenericFeedApi(BaseApi): + def get(self, request): + feed = some_feed_get( + user=request.user, + ) + + data = some_feed_serialize(feed) + + return Response(data) +``` + +In this scenario, `some_feed_get` has the responsibility of returning a list of feed items (can be ORM objects, can be just IDs). + +And we want to push the complexity of serializing this feed, in an optimal manner, to the serializer function - `some_feed_serialize`. + +This means we don't have to do any unnecessary prefetches & optimizations in `some_feed_get`. + +Here's an example of `some_feed_serialize`: + +```python +class FeedItemSerializer(serializers.Serializer): + ... some fields here ... + calculated_field = serializers.IntegerField(source="_calculated_field") + + +def some_feed_serialize(feed: List[FeedItem]): + feed_ids = [feed_item.id for feed_item in feed] + + # Refetch items with more optimizations + # Based on the relations that are going in + objects = FeedItem.objects.select_related( + ... as complex as you want ... + ).prefetch_related( + ... as complex as you want ... + ).filter( + id__in=feed_ids + ).order_by( + "-some_timestamp" + ) + + some_cache = get_some_cache(feed_ids) + + result = [] + + for feed_item in objects: + # An example, adding additional fields for the serializer + # That are based on values outside of our current object + # This may be some optimization to save queries + feed_item._calculated_field = some_cache.get(feed_item.id) + + result.append(FeedItemSerializer(feed_item).data) + + return result +``` + +As you can see, this is a pretty generic example, but the idea is simple: + +1. Refetch your data, with the needed joins & prefetches. +1. Fetch or build in-memory caches, that will save you queries for specific computed values. +1. Return a result, that's ready to be an API response. ## Urls From 5424ce3681373b83f23fd645995b5b11965ceef3 Mon Sep 17 00:00:00 2001 From: Radoslav Georgiev Date: Wed, 17 Nov 2021 14:38:48 +0200 Subject: [PATCH 4/5] Reiterate again --- README.md | 101 +++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 70 insertions(+), 31 deletions(-) diff --git a/README.md b/README.md index a63a0f2..f323a9c 100644 --- a/README.md +++ b/README.md @@ -569,37 +569,37 @@ class ItemBuyTests(TestCase): payment_charge_mock.assert_called() ``` - ## APIs & Serializers When using services & selectors, all of your APIs should look simple & identical. -**General rules for an API is:** +**When we are creating new APIs, we follow those general rules:** * Have 1 API per operation. This means, for CRUD on a model, having 4 APIs. * Inherit from the most simple `APIView` or `GenericAPIView`. * Avoid the more abstract classes, since they tend to manage things via serializers & we want to do that via services & selectors. * **Don't do business logic in your API.** -* You can do object fetching / data manipulation in your APIs (potentially, you can extract that to somewhere else). - * If you are calling `some_service` in your API, you can extract data manipulation to `some_service_parse`. +* You can do **object fetching / data manipulation in your APIs** (potentially, you can extract that to somewhere else). + * If you are calling `some_service` in your API, you can extract object fetching / data manipulation to `some_service_parse`. * Basically, keep the APIs are simple as possible. They are an interface towards your core business logic. -When we are talking about APIs, we need a way to parse data in & parse data out. +When we are talking about APIs, we need a way to deal with data serialization - both incoming & outgoing data. **Here are our rules for API serialization:** * There should be a dedicated **input serializer** & a dedicated **output serializer**. * **Input serializer** takes care of the data coming in. * **Output serializer** takes care of the data coming out. -* Use whatever abstraction works for you, in terms of serialization. +* In terms of serialization, Use whatever abstraction works for you. **In case you are using DRF's serializers, here are our rules:** -* Serializer should be nested in the API and be named either `InputSerializer` or `OutputSerializer`. -* Our preference is for both serializers to inherit from the simpler `serializer.Serializers` and avoid using `serializers.ModelSerializer` +* Serializer should be **nested in the API** and be named either `InputSerializer` or `OutputSerializer`. +* Our preference is for both serializers to inherit from the simpler `Serializer` and avoid using `ModelSerializer` + * This is a matter of preference and choice. If `ModelSerializer` is working fine for you, use it. * If you need a nested serializer, use the `inline_serializer` util. * Reuse serializers as little as possible. - * Once you start reusing serializers by inheriting them, you'll be exposed to unexpected behaviors, when something in the base serializer changes. + * Reusing serializers may expose you to unexpected behavior, when something changes in the base serializers. ### Naming convention @@ -607,11 +607,11 @@ For our APIs we use the following naming convention: `Api`. Here are few examples: `UserCreateApi`, `UserSendResetPasswordApi`, `UserDeactivateApi`, etc. -### An example list API +### List APIs #### Plain -A dead-simple list API would look like that: +A dead-simple list API should look like that: ```python from rest_framework.views import APIView @@ -635,7 +635,7 @@ class UserListApi(APIView): return Response(data) ``` -Keep in mind this API is public by default. Authentication is up to you. +*Keep in mind this API is public by default. Authentication is up to you.* #### Filters + Pagination @@ -648,9 +648,10 @@ That's why, we take the following approach: 1. Selectors take care of the actual filtering. 1. APIs take care of filter parameter serialization. -1. APIs take care of pagination. +1. If you need some of the generic paginations, provided by DRF, the API should take care of that. +1. If you need a different pagination, or you are implementing it yourself, either add a new layer to handle pagination or let the selector do that for you. -Let's look at the example: +**Let's look at the example, where we rely on pagination, provided by DRF:** ```python from rest_framework.views import APIView @@ -724,9 +725,7 @@ def user_list(*, filters=None): As you can see, we are leveraging the powerful [`django-filter`](https://django-filter.readthedocs.io/en/stable/) library. -But you can do whatever suits you best here. We have projects, where we implemented our own filtering layer & used it here. - -The key thing is - **selectors take care of filtering**. +> 👀 The key thing here is that the selector is responsible for the filtering. You can always use something else, as a filtering abstaction. For most of the cases, `django-filter` is more than enough. Finally, let's look at `get_paginated_response`: @@ -750,7 +749,7 @@ def get_paginated_response(*, pagination_class, serializer_class, queryset, requ This is basically a code, extracted from within DRF. -Same goes for the `LimitOffsetPagination: +Same goes for the `LimitOffsetPagination`: ```python from collections import OrderedDict @@ -788,13 +787,15 @@ class LimitOffsetPagination(_LimitOffsetPagination): ])) ``` -What we basically did is reverse-engineered the generic APIs, since pagination should be able to live outside the layers of complexity there. +What we basically did is reverse-engineered the generic APIs. -**A possible future implementation should be able to paginate without needing the request / response of the APIView.** +> 👀 Again, if you need something else for pagination, you can always implement it & use it in the same manner. There are cases, where the selector needs to take care of the pagination. We approach those cases the same way we approach filtering. You can find the code for the example list API with filters & pagination in the [Styleguide Example](https://github.com/HackSoftware/Styleguide-Example#example-list-api) project. -### An example detail API +### Detail API + +Here's an example: ```python class CourseDetailApi(SomeAuthenticationMixin, APIView): @@ -805,14 +806,16 @@ class CourseDetailApi(SomeAuthenticationMixin, APIView): end_date = serializers.DateField() def get(self, request, course_id): - course = get_course(id=course_id) + course = course_get(id=course_id) serializer = self.OutputSerializer(course) return Response(serializer.data) ``` -### An example create API +### Create API + +Here's an example: ```python class CourseCreateApi(SomeAuthenticationMixin, APIView): @@ -825,12 +828,14 @@ class CourseCreateApi(SomeAuthenticationMixin, APIView): serializer = self.InputSerializer(data=request.data) serializer.is_valid(raise_exception=True) - create_course(**serializer.validated_data) + course_create(**serializer.validated_data) return Response(status=status.HTTP_201_CREATED) ``` -### An example update API +### Update API + +Here's an example: ```python class CourseUpdateApi(SomeAuthenticationMixin, APIView): @@ -843,11 +848,41 @@ class CourseUpdateApi(SomeAuthenticationMixin, APIView): serializer = self.InputSerializer(data=request.data) serializer.is_valid(raise_exception=True) - update_course(course_id=course_id, **serializer.validated_data) + course_update(course_id=course_id, **serializer.validated_data) return Response(status=status.HTTP_200_OK) ``` +### Fetching objects + +When our APIs receive an `object_id`, the question that arises is: **Where should we fetch that object?** + +We have several options: + +1. We can pass that object to a serializer, which has a [`PrimaryKeyRelatedField`](https://www.django-rest-framework.org/api-guide/relations/#primarykeyrelatedfield) (or a [`SlugRelatedField`](https://www.django-rest-framework.org/api-guide/relations/#slugrelatedfield) for that matter) +1. We can do some kind of object fetching in the API & pass the object to a service or a selector. +1. We can pass the id to the service / selector and do the object fetching there. + +What approach we take is a matter of project context & preference. + +What we usually do is to fetch objects on the API level, using a special `get_object` util: + +```python +def get_object(model_or_queryset, **kwargs): + """ + Reuse get_object_or_404 since the implementation supports both Model && queryset. + Catch Http404 & return None + """ + try: + return get_object_or_404(model_or_queryset, **kwargs) + except Http404: + return None +``` + +This is a very basic utility, that handles the exception and returns `None` instead. + +Whatever you do, make sure to keep it consistent. + ### Nested serializers In case you need to use a nested serializer, you can do the following thing: @@ -864,11 +899,11 @@ The implementation of `inline_serializer` can be found [here](https://github.com ### Advanced serialization -Sometimes, the end result of an API can be quite complex. Sometimes, we want to optimize the queries that we do and the optimization itself is quite complex. +Sometimes, the end result of an API can be quite complex. Sometimes, we want to optimize the queries that we do and the optimization itself can be quite complex. Trying to stick with just an `OutputSerializer` in that case might limit our options. -In those cases, we can implement our output serialization as a function, and have the optimizations we need there. +In those cases, we can implement our output serialization as a function, and have the optimizations we need there, **instead of having all the optimizations in the selector.** Lets take this API as an example: @@ -884,7 +919,7 @@ class SomeGenericFeedApi(BaseApi): return Response(data) ``` -In this scenario, `some_feed_get` has the responsibility of returning a list of feed items (can be ORM objects, can be just IDs). +In this scenario, `some_feed_get` has the responsibility of returning a list of feed items (can be ORM objects, can be just IDs, can be whatever works for you). And we want to push the complexity of serializing this feed, in an optimal manner, to the serializer function - `some_feed_serialize`. @@ -904,9 +939,9 @@ def some_feed_serialize(feed: List[FeedItem]): # Refetch items with more optimizations # Based on the relations that are going in objects = FeedItem.objects.select_related( - ... as complex as you want ... + # ... as complex as you want ... ).prefetch_related( - ... as complex as you want ... + # ... as complex as you want ... ).filter( id__in=feed_ids ).order_by( @@ -934,6 +969,10 @@ As you can see, this is a pretty generic example, but the idea is simple: 1. Fetch or build in-memory caches, that will save you queries for specific computed values. 1. Return a result, that's ready to be an API response. +Even though this is labeled as "advanced serialization", the pattern is really powerful and can be used for all serializations. + +Such serializer functions usually live in a `serializers.py` module, in the corresponding Django app. + ## Urls We usually organize our urls the same way we organize our APIs - 1 url per API, meaning 1 url per action. From 7cd9d321fc250b7f7e41961b06f7999abcd8a3ef Mon Sep 17 00:00:00 2001 From: Radoslav Georgiev Date: Wed, 17 Nov 2021 14:39:11 +0200 Subject: [PATCH 5/5] Update TOC --- README.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index f323a9c..e2bede7 100644 --- a/README.md +++ b/README.md @@ -27,13 +27,15 @@ Django styleguide that we use in [HackSoft](https://hacksoft.io). * [Testing](#testing-1) - [APIs & Serializers](#apis--serializers) * [Naming convention](#naming-convention-1) - * [An example list API](#an-example-list-api) + * [List APIs](#list-apis) + [Plain](#plain) + [Filters + Pagination](#filters--pagination) - * [An example detail API](#an-example-detail-api) - * [An example create API](#an-example-create-api) - * [An example update API](#an-example-update-api) + * [Detail API](#detail-api) + * [Create API](#create-api) + * [Update API](#update-api) + * [Fetching objects](#fetching-objects) * [Nested serializers](#nested-serializers) + * [Advanced serialization](#advanced-serialization) - [Urls](#urls) - [Exception Handling](#exception-handling) * [Raising Exceptions in Services / Selectors](#raising-exceptions-in-services--selectors)