Merge pull request #303 from tomchristie/parser_refactor

Parser refactor

docs/api-guide/parsers.md

@@ -91,19 +91,27 @@ You will typically want to use both `FormParser` and `MultiPartParser` together
 
 # Custom parsers
 
-To implement a custom parser, you should override `BaseParser`, set the `.media_type` property, and implement the `.parse_stream(self, stream, parser_context)` method.
+To implement a custom parser, you should override `BaseParser`, set the `.media_type` property, and implement the `.parse(self, stream, parser_context)` method.
 
 The method should return the data that will be used to populate the `request.DATA` property.
 
-The arguments passed to `.parse_stream()` are:
+The arguments passed to `.parse()` are:
 
 ### stream
 
 A stream-like object representing the body of the request.
 
+### media_type
+
+Optional.  If provided, this is the media type of the incoming request.
+
+Depending on the request's `Content-Type:` header, this may be more specific than the renderer's `media_type` attribute, and may include media type parameters.  For example `"text/plain; charset=utf-8"`.
+
 ### parser_context
 
-If supplied, this argument will be a dictionary containing any additional context that may be required to parse the request content.  By default it includes the keys `'upload_handlers'` and `'meta'`, which contain the values of the `request.upload_handlers` and `request.meta` properties.
+Optional.  If supplied, this argument will be a dictionary containing any additional context that may be required to parse the request content.
+
+By default this will include the following keys: `view`, `request`, `args`, `kwargs`.
 
 ## Example
 
@@ -116,7 +124,7 @@ The following is an example plaintext parser that will populate the `request.DAT
 
     media_type = 'text/plain'
 
-    def parse_stream(self, stream, parser_context=None):
+    def parse(self, stream, media_type=None, parser_context=None):
         """
         Simply return a string representing the body of the request.
         """
@@ -124,7 +132,7 @@ The following is an example plaintext parser that will populate the `request.DAT
 
 ## Uploading file content
 
-If your custom parser needs to support file uploads, you may return a `DataAndFiles` object from the `.parse_stream()` method.  `DataAndFiles` should be instantiated with two arguments.  The first argument will be used to populate the `request.DATA` property, and the second argument will be used to populate the `request.FILES` property.
+If your custom parser needs to support file uploads, you may return a `DataAndFiles` object from the `.parse()` method.  `DataAndFiles` should be instantiated with two arguments.  The first argument will be used to populate the `request.DATA` property, and the second argument will be used to populate the `request.FILES` property.
 
 For example:
 
@@ -133,7 +141,7 @@ For example:
         A naive raw file upload parser.
         """
 
-        def parse_stream(self, stream, parser_context):
+        def parse(self, stream, media_type=None, parser_context=None):
             content = stream.read()
             name = 'example.dat'
             content_type = 'application/octet-stream'

docs/api-guide/renderers.md

@@ -162,11 +162,14 @@ The request data, as set by the `Response()` instantiation.
 
 ### `media_type=None`
 
-Optional. If provided, this is the accepted media type, as determined by the content negotiation stage.  Depending on the client's `Accept:` header, this may be more specific than the renderer's `media_type` attribute, and may include media type parameters.  For example `"application/json; nested=true"`.
+Optional. If provided, this is the accepted media type, as determined by the content negotiation stage.
+
+Depending on the client's `Accept:` header, this may be more specific than the renderer's `media_type` attribute, and may include media type parameters.  For example `"application/json; nested=true"`.
 
 ### `renderer_context=None`
 
 Optional. If provided, this is a dictionary of contextual information provided by the view.
+
 By default this will include the following keys: `view`, `request`, `response`, `args`, `kwargs`.
 
 ## Example

docs/tutorial/1-serialization.md

@@ -134,12 +134,15 @@ We've now got a few comment instances to play with.  Let's take a look at serial
 
 At this point we've translated the model instance into python native datatypes.  To finalise the serialization process we render the data into `json`.
 
-    stream = JSONRenderer().render(serializer.data)
+    content = JSONRenderer().render(serializer.data)
-    stream
+    content
     # '{"id": 1, "email": "leila@example.com", "content": "nothing to say", "created": "2012-08-22T16:20:09.822"}'
 
 Deserialization is similar.  First we parse a stream into python native datatypes... 
 
+    import StringIO
+
+    stream = StringIO.StringIO(content)
     data = JSONParser().parse(stream)
 
 ...then we restore those native datatypes into to a fully populated object instance.

rest_framework/parsers.py

@@ -1,14 +1,8 @@
 """
-Django supports parsing the content of an HTTP request, but only for form POST requests.
+Parsers are used to parse the content of incoming HTTP requests.
-That behavior is sufficient for dealing with standard HTML forms, but it doesn't map well
-to general HTTP requests.
 
-We need a method to be able to:
+They give us a generic way of being able to handle various media types
-
+on the request, such as form content or json encoded data.
-1.) Determine the parsed content on a request for methods other than POST (eg typically also PUT)
-
-2.) Determine the parsed content on a request for media types other than application/x-www-form-urlencoded
-   and multipart/form-data.  (eg also handle multipart/json)
 """
 
 from django.http import QueryDict
@@ -21,7 +15,6 @@ from xml.etree import ElementTree as ET
 from xml.parsers.expat import ExpatError
 import datetime
 import decimal
-from io import BytesIO
 
 
 class DataAndFiles(object):
@@ -33,29 +26,18 @@ class DataAndFiles(object):
 class BaseParser(object):
     """
     All parsers should extend `BaseParser`, specifying a `media_type`
-    attribute, and overriding the `.parse_stream()` method.
+    attribute, and overriding the `.parse()` method.
     """
 
     media_type = None
 
-    def parse(self, string_or_stream, parser_context=None):
+    def parse(self, stream, media_type=None, parser_context=None):
         """
-        The main entry point to parsers.  This is a light wrapper around
+        Given a stream to read from, return the parsed representation.
-        `parse_stream`, that instead handles both string and stream objects.
+        Should return parsed data, or a `DataAndFiles` object consisting of the
-        """
-        if isinstance(string_or_stream, basestring):
-            stream = BytesIO(string_or_stream)
-        else:
-            stream = string_or_stream
-        return self.parse_stream(stream, parser_context)
-
-    def parse_stream(self, stream, parser_context=None):
-        """
-        Given a stream to read from, return the deserialized output.
-        Should return parsed data, or a DataAndFiles object consisting of the
         parsed data and files.
         """
-        raise NotImplementedError(".parse_stream() must be overridden.")
+        raise NotImplementedError(".parse() must be overridden.")
 
 
 class JSONParser(BaseParser):
@@ -65,7 +47,7 @@ class JSONParser(BaseParser):
 
     media_type = 'application/json'
 
-    def parse_stream(self, stream, parser_context=None):
+    def parse(self, stream, media_type=None, parser_context=None):
         """
         Returns a 2-tuple of `(data, files)`.
 
@@ -85,7 +67,7 @@ class YAMLParser(BaseParser):
 
     media_type = 'application/yaml'
 
-    def parse_stream(self, stream, parser_context=None):
+    def parse(self, stream, media_type=None, parser_context=None):
         """
         Returns a 2-tuple of `(data, files)`.
 
@@ -105,7 +87,7 @@ class FormParser(BaseParser):
 
     media_type = 'application/x-www-form-urlencoded'
 
-    def parse_stream(self, stream, parser_context=None):
+    def parse(self, stream, media_type=None, parser_context=None):
         """
         Returns a 2-tuple of `(data, files)`.
 
@@ -123,7 +105,7 @@ class MultiPartParser(BaseParser):
 
     media_type = 'multipart/form-data'
 
-    def parse_stream(self, stream, parser_context=None):
+    def parse(self, stream, media_type=None, parser_context=None):
         """
         Returns a DataAndFiles object.
 
@@ -131,8 +113,10 @@ class MultiPartParser(BaseParser):
         `.files` will be a `QueryDict` containing all the form files.
         """
         parser_context = parser_context or {}
-        meta = parser_context['meta']
+        request = parser_context['request']
-        upload_handlers = parser_context['upload_handlers']
+        meta = request.META
+        upload_handlers = request.upload_handlers
+
         try:
             parser = DjangoMultiPartParser(meta, stream, upload_handlers)
             data, files = parser.parse()
@@ -148,7 +132,7 @@ class XMLParser(BaseParser):
 
     media_type = 'application/xml'
 
-    def parse_stream(self, stream, parser_context=None):
+    def parse(self, stream, media_type=None, parser_context=None):
         try:
             tree = ET.parse(stream)
         except (ExpatError, ETParseError, ValueError), exc:

rest_framework/renderers.py

@@ -1,9 +1,10 @@
 """
-Renderers are used to serialize a View's output into specific media types.
+Renderers are used to serialize a response into specific media types.
 
-Django REST framework also provides HTML and PlainText renderers that help self-document the API,
+They give us a generic way of being able to handle various media types
-by serializing the output along with documentation regarding the View, output status and headers,
+on the response, such as JSON encoded data or HTML output.
-and providing forms and links depending on the allowed methods, renderers and parsers on the View.
+
+REST framework also provides an HTML renderer the renders the browseable API.
 """
 import string
 from django import forms
@@ -23,8 +24,8 @@ from rest_framework import serializers, parsers
 
 class BaseRenderer(object):
     """
-    All renderers must extend this class, set the :attr:`media_type` attribute,
+    All renderers should extend this class, setting the `media_type`
-    and override the :meth:`render` method.
+    and `format` attributes, and override the `.render()` method.
     """
 
     media_type = None

rest_framework/request.py

@@ -88,17 +88,12 @@ class Request(object):
         self._stream = Empty
 
         if self.parser_context is None:
-            self.parser_context = self._default_parser_context(request)
+            self.parser_context = {}
+        self.parser_context['request'] = self
 
     def _default_negotiator(self):
         return api_settings.DEFAULT_CONTENT_NEGOTIATION()
 
-    def _default_parser_context(self, request):
-        return {
-            'upload_handlers': request.upload_handlers,
-            'meta': request.META,
-        }
-
     @property
     def method(self):
         """
@@ -265,15 +260,19 @@ class Request(object):
 
         May raise an `UnsupportedMediaType`, or `ParseError` exception.
         """
-        if self.stream is None or self.content_type is None:
+        stream = self.stream
+        media_type = self.content_type
+
+        if stream is None or media_type is None:
             return (None, None)
 
-        parser = self.negotiator.select_parser(self.parsers, self.content_type)
+        parser = self.negotiator.select_parser(self.parsers, media_type)
 
         if not parser:
-            raise exceptions.UnsupportedMediaType(self.content_type)
+            raise exceptions.UnsupportedMediaType(media_type)
+
+        parsed = parser.parse(stream, media_type, self.parser_context)
 
-        parsed = parser.parse(self.stream, self.parser_context)
         # Parser classes may return the raw data, or a
         # DataAndFiles object.  Unpack the result as required.
         try:

rest_framework/tests/request.py

@@ -27,7 +27,7 @@ factory = RequestFactory()
 class PlainTextParser(BaseParser):
     media_type = 'text/plain'
 
-    def parse_stream(self, stream, parser_context=None):
+    def parse(self, stream, media_type=None, parser_context=None):
         """
         Returns a 2-tuple of `(data, files)`.
 

rest_framework/views.py

@@ -158,12 +158,15 @@ class APIView(View):
 
     def get_parser_context(self, http_request):
         """
-        Returns a dict that is passed through to Parser.parse_stream(),
+        Returns a dict that is passed through to Parser.parse(),
         as the `parser_context` keyword argument.
         """
+        # Note: Additionally `request` will also be added to the context
+        #       by the Request object.
         return {
-            'upload_handlers': http_request.upload_handlers,
+            'view': self,
-            'meta': http_request.META,
+            'args': getattr(self, 'args', ()),
+            'kwargs': getattr(self, 'kwargs', {})
         }
 
     def get_renderer_context(self):
@@ -171,13 +174,13 @@ class APIView(View):
         Returns a dict that is passed through to Renderer.render(),
         as the `renderer_context` keyword argument.
         """
-        # Note: Additionally 'response' will also be set on the context,
+        # Note: Additionally 'response' will also be added to the context,
         #       by the Response object.
         return {
             'view': self,
-            'request': self.request,
+            'args': getattr(self, 'args', ()),
-            'args': self.args,
+            'kwargs': getattr(self, 'kwargs', {}),
-            'kwargs': self.kwargs
+            'request': getattr(self, 'request', None)
         }
 
     # API policy instantiation methods