Merge pull request #744 from tomchristie/basic-bulk-edit

Basic bulk create and bulk update
This commit is contained in:
Tom Christie 2013-03-22 14:47:29 -07:00
commit 870d5c7d78
5 changed files with 494 additions and 98 deletions

View File

@ -273,6 +273,49 @@ Django's regular [FILE_UPLOAD_HANDLERS] are used for handling uploaded files.
---
# Custom fields
If you want to create a custom field, you'll probably want to override either one or both of the `.to_native()` and `.from_native()` methods. These two methods are used to convert between the intial datatype, and a primative, serializable datatype. Primative datatypes may be any of a number, string, date/time/datetime or None. They may also be any list or dictionary like object that only contains other primative objects.
The `.to_native()` method is called to convert the initial datatype into a primative, serializable datatype. The `from_native()` method is called to restore a primative datatype into it's initial representation.
## Examples
Let's look at an example of serializing a class that represents an RGB color value:
class Color(object):
"""
A color represented in the RGB colorspace.
"""
def __init__(self, red, green, blue):
assert(red >= 0 and green >= 0 and blue >= 0)
assert(red < 256 and green < 256 and blue < 256)
self.red, self.green, self.blue = red, green, blue
class ColourField(serializers.WritableField):
"""
Color objects are serialized into "rgb(#, #, #)" notation.
"""
def to_native(self, obj):
return "rgb(%d, %d, %d)" % (obj.red, obj.green, obj.blue)
def from_native(self, data):
data = data.strip('rgb(').rstrip(')')
red, green, blue = [int(col) for col in data.split(',')]
return Color(red, green, blue)
By default field values are treated as mapping to an attribute on the object. If you need to customize how the field value is accessed and set you need to override `.field_to_native()` and/or `.field_from_native()`.
As an example, let's create a field that can be used represent the class name of the object being serialized:
class ClassNameField(serializers.Field):
def field_to_native(self, obj, field_name):
"""
Serialize the object's class name.
"""
return obj.__class__
[cite]: https://docs.djangoproject.com/en/dev/ref/forms/api/#django.forms.Form.cleaned_data
[FILE_UPLOAD_HANDLERS]: https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-FILE_UPLOAD_HANDLERS
[strftime]: http://docs.python.org/2/library/datetime.html#strftime-and-strptime-behavior

View File

@ -37,9 +37,6 @@ Declaring a serializer looks very similar to declaring a form:
"""
Given a dictionary of deserialized field values, either update
an existing model instance, or create a new model instance.
Note that if we don't define this method, then deserializing
data will simply return a dictionary of items.
"""
if instance is not None:
instance.title = attrs.get('title', instance.title)
@ -48,7 +45,9 @@ Declaring a serializer looks very similar to declaring a form:
return instance
return Comment(**attrs)
The first part of serializer class defines the fields that get serialized/deserialized. The `restore_object` method defines how fully fledged instances get created when deserializing data. The `restore_object` method is optional, and is only required if we want our serializer to support deserialization.
The first part of serializer class defines the fields that get serialized/deserialized. The `restore_object` method defines how fully fledged instances get created when deserializing data.
The `restore_object` method is optional, and is only required if we want our serializer to support deserialization into fully fledged object instances. If we don't define this method, then deserializing data will simply return a dictionary of items.
## Serializing objects
@ -88,23 +87,21 @@ By default, serializers must be passed values for all required fields or they wi
serializer = CommentSerializer(comment, data={'content': u'foo bar'}, partial=True) # Update `instance` with partial data
## Serializing querysets
To serialize a queryset instead of an object instance, you should pass the `many=True` flag when instantiating the serializer.
queryset = Comment.objects.all()
serializer = CommentSerializer(queryset, many=True)
serializer.data
# [{'email': u'leila@example.com', 'content': u'foo bar', 'created': datetime.datetime(2012, 8, 22, 16, 20, 9, 822774)}, {'email': u'jamie@example.com', 'content': u'baz', 'created': datetime.datetime(2013, 1, 12, 16, 12, 45, 104445)}]
## Validation
When deserializing data, you always need to call `is_valid()` before attempting to access the deserialized object. If any validation errors occur, the `.errors` property will contain a dictionary representing the resulting error messages.
When deserializing data, you always need to call `is_valid()` before attempting to access the deserialized object. If any validation errors occur, the `.errors` property will contain a dictionary representing the resulting error messages. For example:
serializer = CommentSerializer(data={'email': 'foobar', 'content': 'baz'})
serializer.is_valid()
# False
serializer.errors
# {'email': [u'Enter a valid e-mail address.'], 'created': [u'This field is required.']}
Each key in the dictionary will be the field name, and the values will be lists of strings of any error messages corresponding to that field. The `non_field_errors` key may also be present, and will list any general validation errors.
When deserializing a list of items, errors will be returned as a list of dictionaries representing each of the deserialized items.
### Field-level validation
#### Field-level validation
You can specify custom field-level validation by adding `.validate_<fieldname>` methods to your `Serializer` subclass. These are analagous to `.clean_<fieldname>` methods on Django forms, but accept slightly different arguments.
@ -127,7 +124,7 @@ Your `validate_<fieldname>` methods should either just return the `attrs` dictio
raise serializers.ValidationError("Blog post is not about Django")
return attrs
### Object-level validation
#### Object-level validation
To do any other validation that requires access to multiple fields, add a method called `.validate()` to your `Serializer` subclass. This method takes a single argument, which is the `attrs` dictionary. It should raise a `ValidationError` if necessary, or just return `attrs`. For example:
@ -148,26 +145,44 @@ To do any other validation that requires access to multiple fields, add a method
## Saving object state
Serializers also include a `.save()` method that you can override if you want to provide a method of persisting the state of a deserialized object. The default behavior of the method is to simply call `.save()` on the deserialized object instance.
To save the deserialized objects created by a serializer, call the `.save()` method:
if serializer.is_valid():
serializer.save()
The default behavior of the method is to simply call `.save()` on the deserialized object instance. You can override the default save behaviour by overriding the `.save_object(obj)` method on the serializer class.
The generic views provided by REST framework call the `.save()` method when updating or creating entities.
## Dealing with nested objects
The previous example is fine for dealing with objects that only have simple datatypes, but sometimes we also need to be able to represent more complex objects,
where some of the attributes of an object might not be simple datatypes such as strings, dates or integers.
The previous examples are fine for dealing with objects that only have simple datatypes, but sometimes we also need to be able to represent more complex objects, where some of the attributes of an object might not be simple datatypes such as strings, dates or integers.
The `Serializer` class is itself a type of `Field`, and can be used to represent relationships where one object type is nested inside another.
class UserSerializer(serializers.Serializer):
email = serializers.Field()
username = serializers.Field()
email = serializers.EmailField()
username = serializers.CharField(max_length=100)
class CommentSerializer(serializers.Serializer):
user = UserSerializer()
title = serializers.Field()
content = serializers.Field()
created = serializers.Field()
content = serializers.CharField(max_length=200)
created = serializers.DateTimeField()
If a nested representation may optionally accept the `None` value you should pass the `required=False` flag to the nested serializer.
class CommentSerializer(serializers.Serializer):
user = UserSerializer(required=False) # May be an anonymous user.
content = serializers.CharField(max_length=200)
created = serializers.DateTimeField()
Similarly if a nested representation should be a list of items, you should the `many=True` flag to the nested serialized.
class CommentSerializer(serializers.Serializer):
user = UserSerializer(required=False)
edits = EditItemSerializer(many=True) # A nested list of 'edit' items.
content = serializers.CharField(max_length=200)
created = serializers.DateTimeField()
---
@ -175,6 +190,96 @@ The `Serializer` class is itself a type of `Field`, and can be used to represent
---
## Dealing with multiple objects
The `Serializer` class can also handle serializing or deserializing lists of objects.
#### Serializing multiple objects
To serialize a queryset or list of objects instead of a single object instance, you should pass the `many=True` flag when instantiating the serializer. You can then pass a queryset or list of objects to be serialized.
queryset = Book.objects.all()
serializer = BookSerializer(queryset, many=True)
serializer.data
# [
# {'id': 0, 'title': 'The electric kool-aid acid test', 'author': 'Tom Wolfe'},
# {'id': 1, 'title': 'If this is a man', 'author': 'Primo Levi'},
# {'id': 2, 'title': 'The wind-up bird chronicle', 'author': 'Haruki Murakami'}
# ]
#### Deserializing multiple objects for creation
To deserialize a list of object data, and create multiple object instances in a single pass, you should also set the `many=True` flag, and pass a list of data to be deserialized.
This allows you to write views that create multiple items when a `POST` request is made.
For example:
data = [
{'title': 'The bell jar', 'author': 'Sylvia Plath'},
{'title': 'For whom the bell tolls', 'author': 'Ernest Hemingway'}
]
serializer = BookSerializer(data=data, many=True)
serializer.is_valid()
# True
serializer.save() # `.save()` will be called on each deserialized instance
#### Deserializing multiple objects for update
You can also deserialize a list of objects as part of a bulk update of multiple existing items.
In this case you need to supply both an existing list or queryset of items, as well as a list of data to update those items with.
This allows you to write views that update or create multiple items when a `PUT` request is made.
# Capitalizing the titles of the books
queryset = Book.objects.all()
data = [
{'id': 3, 'title': 'The Bell Jar', 'author': 'Sylvia Plath'},
{'id': 4, 'title': 'For Whom the Bell Tolls', 'author': 'Ernest Hemingway'}
]
serializer = BookSerializer(queryset, data=data, many=True)
serializer.is_valid()
# True
serialize.save() # `.save()` will be called on each updated or newly created instance.
Bulk updates will update any instances that already exist, and create new instances for data items that do not have a corresponding instance.
When performing a bulk update you may want any items that are not present in the incoming data to be deleted. To do so, pass `allow_delete=True` to the serializer.
serializer = BookSerializer(queryset, data=data, many=True, allow_delete=True)
serializer.is_valid()
# True
serializer.save() # `.save()` will be called on each updated or newly created instance.
# `.delete()` will be called on any other items in the `queryset`.
Passing `allow_delete=True` ensures that any update operations will completely overwrite the existing queryset, rather than simply updating any objects found in the incoming data.
#### How identity is determined when performing bulk updates
Performing a bulk update is slightly more complicated than performing a bulk creation, because the serializer needs a way of determining how the items in the incoming data should be matched against the existing object instances.
By default the serializer class will use the `id` key on the incoming data to determine the canonical identity of an object. If you need to change this behavior you should override the `get_identity` method on the `Serializer` class. For example:
class AccountSerializer(serializers.Serializer):
slug = serializers.CharField(max_length=100)
created = serializers.DateTimeField()
... # Various other fields
def get_identity(self, data):
"""
This hook is required for bulk update.
We need to override the default, to use the slug as the identity.
Note that the data has not yet been validated at this point,
so we need to deal gracefully with incorrect datatypes.
"""
try:
return data.get('slug', None)
except AttributeError:
return None
To map the incoming data items to their corresponding object instances, the `.get_identity()` method will be called both against the incoming data, and against the serialized representation of the existing objects.
## Including extra context
There are some cases where you need to provide extra context to the serializer in addition to the object being serialized. One common case is if you're using a serializer that includes hyperlinked relations, which requires the serializer to have access to the current request so that it can properly generate fully qualified URLs.
@ -187,47 +292,6 @@ You can provide arbitrary additional context by passing a `context` argument whe
The context dictionary can be used within any serializer field logic, such as a custom `.to_native()` method, by accessing the `self.context` attribute.
## Creating custom fields
If you want to create a custom field, you'll probably want to override either one or both of the `.to_native()` and `.from_native()` methods. These two methods are used to convert between the intial datatype, and a primative, serializable datatype. Primative datatypes may be any of a number, string, date/time/datetime or None. They may also be any list or dictionary like object that only contains other primative objects.
The `.to_native()` method is called to convert the initial datatype into a primative, serializable datatype. The `from_native()` method is called to restore a primative datatype into it's initial representation.
Let's look at an example of serializing a class that represents an RGB color value:
class Color(object):
"""
A color represented in the RGB colorspace.
"""
def __init__(self, red, green, blue):
assert(red >= 0 and green >= 0 and blue >= 0)
assert(red < 256 and green < 256 and blue < 256)
self.red, self.green, self.blue = red, green, blue
class ColourField(serializers.WritableField):
"""
Color objects are serialized into "rgb(#, #, #)" notation.
"""
def to_native(self, obj):
return "rgb(%d, %d, %d)" % (obj.red, obj.green, obj.blue)
def from_native(self, data):
data = data.strip('rgb(').rstrip(')')
red, green, blue = [int(col) for col in data.split(',')]
return Color(red, green, blue)
By default field values are treated as mapping to an attribute on the object. If you need to customize how the field value is accessed and set you need to override `.field_to_native()` and/or `.field_from_native()`.
As an example, let's create a field that can be used represent the class name of the object being serialized:
class ClassNameField(serializers.Field):
def field_to_native(self, obj, field_name):
"""
Serialize the object's class name.
"""
return obj.__class__
---
# ModelSerializers

View File

@ -110,13 +110,15 @@ class BaseSerializer(Field):
_dict_class = SortedDictWithMetadata
def __init__(self, instance=None, data=None, files=None,
context=None, partial=False, many=None, source=None):
context=None, partial=False, many=None, source=None,
allow_delete=False):
super(BaseSerializer, self).__init__(source=source)
self.opts = self._options_class(self.Meta)
self.parent = None
self.root = None
self.partial = partial
self.many = many
self.allow_delete = allow_delete
self.context = context or {}
@ -128,6 +130,13 @@ class BaseSerializer(Field):
self._data = None
self._files = None
self._errors = None
self._deleted = None
if many and instance is not None and not hasattr(instance, '__iter__'):
raise ValueError('instance should be a queryset or other iterable with many=True')
if allow_delete and not many:
raise ValueError('allow_delete should only be used for bulk updates, but you have not set many=True')
#####
# Methods to determine which fields to use when (de)serializing objects.
@ -331,6 +340,20 @@ class BaseSerializer(Field):
return [self.to_native(item) for item in obj]
return self.to_native(obj)
def get_identity(self, data):
"""
This hook is required for bulk update.
It is used to determine the canonical identity of a given object.
Note that the data has not been validated at this point, so we need
to make sure that we catch any cases of incorrect datatypes being
passed to this method.
"""
try:
return data.get('id', None)
except AttributeError:
return None
@property
def errors(self):
"""
@ -352,10 +375,33 @@ class BaseSerializer(Field):
if many:
ret = []
errors = []
for item in data:
ret.append(self.from_native(item, None))
errors.append(self._errors)
self._errors = any(errors) and errors or []
update = self.object is not None
if update:
# If this is a bulk update we need to map all the objects
# to a canonical identity so we can determine which
# individual object is being updated for each item in the
# incoming data
objects = self.object
identities = [self.get_identity(self.to_native(obj)) for obj in objects]
identity_to_objects = dict(zip(identities, objects))
if hasattr(data, '__iter__') and not isinstance(data, (dict, six.text_type)):
for item in data:
if update:
# Determine which object we're updating
identity = self.get_identity(item)
self.object = identity_to_objects.pop(identity, None)
ret.append(self.from_native(item, None))
errors.append(self._errors)
if update:
self._deleted = identity_to_objects.values()
self._errors = any(errors) and errors or []
else:
self._errors = {'non_field_errors': ['Expected a list of items']}
else:
ret = self.from_native(data, files)
@ -394,6 +440,9 @@ class BaseSerializer(Field):
def save_object(self, obj, **kwargs):
obj.save(**kwargs)
def delete_object(self, obj):
obj.delete()
def save(self, **kwargs):
"""
Save the deserialized object and return it.
@ -402,6 +451,10 @@ class BaseSerializer(Field):
[self.save_object(item, **kwargs) for item in self.object]
else:
self.save_object(self.object, **kwargs)
if self.allow_delete and self._deleted:
[self.delete_object(item) for item in self._deleted]
return self.object
@ -690,3 +743,13 @@ class HyperlinkedModelSerializer(ModelSerializer):
'many': to_many
}
return HyperlinkedRelatedField(**kwargs)
def get_identity(self, data):
"""
This hook is required for bulk update.
We need to override the default, to use the url as the identity.
"""
try:
return data.get('url', None)
except AttributeError:
return None

View File

@ -261,34 +261,6 @@ class ValidationTests(TestCase):
self.assertEqual(serializer.is_valid(), True)
self.assertEqual(serializer.errors, {})
def test_bad_type_data_is_false(self):
"""
Data of the wrong type is not valid.
"""
data = ['i am', 'a', 'list']
serializer = CommentSerializer(self.comment, data=data, many=True)
self.assertEqual(serializer.is_valid(), False)
self.assertTrue(isinstance(serializer.errors, list))
self.assertEqual(
serializer.errors,
[
{'non_field_errors': ['Invalid data']},
{'non_field_errors': ['Invalid data']},
{'non_field_errors': ['Invalid data']}
]
)
data = 'and i am a string'
serializer = CommentSerializer(self.comment, data=data)
self.assertEqual(serializer.is_valid(), False)
self.assertEqual(serializer.errors, {'non_field_errors': ['Invalid data']})
data = 42
serializer = CommentSerializer(self.comment, data=data)
self.assertEqual(serializer.is_valid(), False)
self.assertEqual(serializer.errors, {'non_field_errors': ['Invalid data']})
def test_cross_field_validation(self):
class CommentSerializerWithCrossFieldValidator(CommentSerializer):

View File

@ -0,0 +1,254 @@
"""
Tests to cover bulk create and update using serializers.
"""
from __future__ import unicode_literals
from django.test import TestCase
from rest_framework import serializers
class BulkCreateSerializerTests(TestCase):
"""
Creating multiple instances using serializers.
"""
def setUp(self):
class BookSerializer(serializers.Serializer):
id = serializers.IntegerField()
title = serializers.CharField(max_length=100)
author = serializers.CharField(max_length=100)
self.BookSerializer = BookSerializer
def test_bulk_create_success(self):
"""
Correct bulk update serialization should return the input data.
"""
data = [
{
'id': 0,
'title': 'The electric kool-aid acid test',
'author': 'Tom Wolfe'
}, {
'id': 1,
'title': 'If this is a man',
'author': 'Primo Levi'
}, {
'id': 2,
'title': 'The wind-up bird chronicle',
'author': 'Haruki Murakami'
}
]
serializer = self.BookSerializer(data=data, many=True)
self.assertEqual(serializer.is_valid(), True)
self.assertEqual(serializer.object, data)
def test_bulk_create_errors(self):
"""
Correct bulk update serialization should return the input data.
"""
data = [
{
'id': 0,
'title': 'The electric kool-aid acid test',
'author': 'Tom Wolfe'
}, {
'id': 1,
'title': 'If this is a man',
'author': 'Primo Levi'
}, {
'id': 'foo',
'title': 'The wind-up bird chronicle',
'author': 'Haruki Murakami'
}
]
expected_errors = [
{},
{},
{'id': ['Enter a whole number.']}
]
serializer = self.BookSerializer(data=data, many=True)
self.assertEqual(serializer.is_valid(), False)
self.assertEqual(serializer.errors, expected_errors)
def test_invalid_list_datatype(self):
"""
Data containing list of incorrect data type should return errors.
"""
data = ['foo', 'bar', 'baz']
serializer = self.BookSerializer(data=data, many=True)
self.assertEqual(serializer.is_valid(), False)
expected_errors = [
{'non_field_errors': ['Invalid data']},
{'non_field_errors': ['Invalid data']},
{'non_field_errors': ['Invalid data']}
]
self.assertEqual(serializer.errors, expected_errors)
def test_invalid_single_datatype(self):
"""
Data containing a single incorrect data type should return errors.
"""
data = 123
serializer = self.BookSerializer(data=data, many=True)
self.assertEqual(serializer.is_valid(), False)
expected_errors = {'non_field_errors': ['Expected a list of items']}
self.assertEqual(serializer.errors, expected_errors)
def test_invalid_single_object(self):
"""
Data containing only a single object, instead of a list of objects
should return errors.
"""
data = {
'id': 0,
'title': 'The electric kool-aid acid test',
'author': 'Tom Wolfe'
}
serializer = self.BookSerializer(data=data, many=True)
self.assertEqual(serializer.is_valid(), False)
expected_errors = {'non_field_errors': ['Expected a list of items']}
self.assertEqual(serializer.errors, expected_errors)
class BulkUpdateSerializerTests(TestCase):
"""
Updating multiple instances using serializers.
"""
def setUp(self):
class Book(object):
"""
A data type that can be persisted to a mock storage backend
with `.save()` and `.delete()`.
"""
object_map = {}
def __init__(self, id, title, author):
self.id = id
self.title = title
self.author = author
def save(self):
Book.object_map[self.id] = self
def delete(self):
del Book.object_map[self.id]
class BookSerializer(serializers.Serializer):
id = serializers.IntegerField()
title = serializers.CharField(max_length=100)
author = serializers.CharField(max_length=100)
def restore_object(self, attrs, instance=None):
if instance:
instance.id = attrs['id']
instance.title = attrs['title']
instance.author = attrs['author']
return instance
return Book(**attrs)
self.Book = Book
self.BookSerializer = BookSerializer
data = [
{
'id': 0,
'title': 'The electric kool-aid acid test',
'author': 'Tom Wolfe'
}, {
'id': 1,
'title': 'If this is a man',
'author': 'Primo Levi'
}, {
'id': 2,
'title': 'The wind-up bird chronicle',
'author': 'Haruki Murakami'
}
]
for item in data:
book = Book(item['id'], item['title'], item['author'])
book.save()
def books(self):
"""
Return all the objects in the mock storage backend.
"""
return self.Book.object_map.values()
def test_bulk_update_success(self):
"""
Correct bulk update serialization should return the input data.
"""
data = [
{
'id': 0,
'title': 'The electric kool-aid acid test',
'author': 'Tom Wolfe'
}, {
'id': 2,
'title': 'Kafka on the shore',
'author': 'Haruki Murakami'
}
]
serializer = self.BookSerializer(self.books(), data=data, many=True, allow_delete=True)
self.assertEqual(serializer.is_valid(), True)
self.assertEqual(serializer.data, data)
serializer.save()
new_data = self.BookSerializer(self.books(), many=True).data
self.assertEqual(data, new_data)
def test_bulk_update_and_create(self):
"""
Bulk update serialization may also include created items.
"""
data = [
{
'id': 0,
'title': 'The electric kool-aid acid test',
'author': 'Tom Wolfe'
}, {
'id': 3,
'title': 'Kafka on the shore',
'author': 'Haruki Murakami'
}
]
serializer = self.BookSerializer(self.books(), data=data, many=True, allow_delete=True)
self.assertEqual(serializer.is_valid(), True)
self.assertEqual(serializer.data, data)
serializer.save()
new_data = self.BookSerializer(self.books(), many=True).data
self.assertEqual(data, new_data)
def test_bulk_update_error(self):
"""
Incorrect bulk update serialization should return error data.
"""
data = [
{
'id': 0,
'title': 'The electric kool-aid acid test',
'author': 'Tom Wolfe'
}, {
'id': 'foo',
'title': 'Kafka on the shore',
'author': 'Haruki Murakami'
}
]
expected_errors = [
{},
{'id': ['Enter a whole number.']}
]
serializer = self.BookSerializer(self.books(), data=data, many=True, allow_delete=True)
self.assertEqual(serializer.is_valid(), False)
self.assertEqual(serializer.errors, expected_errors)