Fix charset issues

docs/api-guide/renderers.md

@@ -67,7 +67,7 @@ If your API includes views that can serve both regular webpages and API response
 
 ## JSONRenderer
 
-Renders the request data into `JSON` enforcing ASCII encoding
+Renders the request data into `JSON`, using ASCII encoding.
 
 The client may additionally include an `'indent'` media type parameter, in which case the returned `JSON` will be indented.  For example `Accept: application/json; indent=4`.
 
@@ -75,9 +75,19 @@ The client may additionally include an `'indent'` media type parameter, in which
 
 **.format**: `'.json'`
 
+**.charset**: `iso-8859-1`
+
 ## UnicodeJSONRenderer
 
-Same as `JSONRenderer` but doesn't enforce ASCII encoding
+Renders the request data into `JSON`, using utf-8 encoding.
+
+The client may additionally include an `'indent'` media type parameter, in which case the returned `JSON` will be indented.  For example `Accept: application/json; indent=4`.
+
+**.media_type**: `application/json`
+
+**.format**: `'.json'`
+
+**.charset**: `utf-8`
 
 ## JSONPRenderer
 
@@ -91,6 +101,8 @@ The javascript callback function must be set by the client including a `callback
 
 **.format**: `'.jsonp'`
 
+**.charset**: `iso-8859-1`
+
 ## YAMLRenderer
 
 Renders the request data into `YAML`. 
@@ -101,6 +113,8 @@ Requires the `pyyaml` package to be installed.
 
 **.format**: `'.yaml'`
 
+**.charset**: `utf-8`
+
 ## XMLRenderer
 
 Renders REST framework's default style of `XML` response content.
@@ -113,6 +127,8 @@ If you are considering using `XML` for your API, you may want to consider implem
 
 **.format**: `'.xml'`
 
+**.charset**: `utf-8`
+
 ## TemplateHTMLRenderer
 
 Renders data to HTML, using Django's standard template rendering.
@@ -147,6 +163,8 @@ If you're building websites that use `TemplateHTMLRenderer` along with other ren
 
 **.format**: `'.html'`
 
+**.charset**: `utf-8`
+
 See also: `StaticHTMLRenderer`
 
 ## StaticHTMLRenderer
@@ -167,6 +185,8 @@ You can use `TemplateHTMLRenderer` either to return regular HTML pages using RES
 
 **.format**: `'.html'`
 
+**.charset**: `utf-8`
+
 See also: `TemplateHTMLRenderer`
 
 ## BrowsableAPIRenderer
@@ -177,12 +197,16 @@ Renders data into HTML for the Browsable API.  This renderer will determine whic
 
 **.format**: `'.api'`
 
+**.charset**: `utf-8`
+
 ---
 
 # Custom renderers
 
 To implement a custom renderer, you should override `BaseRenderer`, set the `.media_type` and `.format` properties, and implement the `.render(self, data, media_type=None, renderer_context=None)` method.
 
+The method should return a bytestring, which wil be used as the body of the HTTP response.
+
 The arguments passed to the `.render()` method are:
 
 ### `data`
@@ -209,14 +233,34 @@ The following is an example plaintext renderer that will return a response with
     from rest_framework import renderers
 
 
-    class PlainText(renderers.BaseRenderer):
+    class PlainTextRenderer(renderers.BaseRenderer):
         media_type = 'text/plain'
         format = 'txt'
         
         def render(self, data, media_type=None, renderer_context=None):
-            if isinstance(data, basestring):
+            return data.encode(self.charset)
-                return data
+
-            return smart_unicode(data)
+## Setting the character set
+
+By default renderer classes are assumed to be using the `UTF-8` encoding.  To use a different encoding, set the `charset` attribute on the renderer.
+
+    class PlainTextRenderer(renderers.BaseRenderer):
+        media_type = 'text/plain'
+        format = 'txt'
+        charset = 'iso-8859-1'
+
+        def render(self, data, media_type=None, renderer_context=None):
+            return data.encode(self.charset)
+
+If the renderer returns a raw bytestring, you should set a charset value of `None`, which will ensure the `Content-Type` header of the response will not have a `charset` value set.  Doing so will also ensure that the browsable API will not attempt to display the binary content as a string.
+
+    class JPEGRenderer(renderers.BaseRenderer):
+        media_type = 'image/jpeg'
+        format = 'jpg'
+        charset = None
+
+        def render(self, data, media_type=None, renderer_context=None):
+            return data
 
 ---
 
@@ -286,11 +330,11 @@ Templates will render with a `RequestContext` which includes the `status_code` a
 
 The following third party packages are also available.
 
-## MessagePack
+### MessagePack
 
 [MessagePack][messagepack] is a fast, efficient binary serialization format.  [Juan Riaza][juanriaza] maintains the [djangorestframework-msgpack][djangorestframework-msgpack] package which provides MessagePack renderer and parser support for REST framework.
 
-## CSV
+### CSV
 
 Comma-separated values are a plain-text tabular data format, that can be easily imported into spreadsheet applications.  [Mjumbe Poe][mjumbewu] maintains the [djangorestframework-csv][djangorestframework-csv] package which provides CSV renderer support for REST framework.
 

rest_framework/renderers.py

@@ -9,7 +9,6 @@ REST framework also provides an HTML renderer the renders the browsable API.
 from __future__ import unicode_literals
 
 import copy
-import string
 import json
 from django import forms
 from django.http.multipartparser import parse_header
@@ -36,7 +35,7 @@ class BaseRenderer(object):
 
     media_type = None
     format = None
-    charset = None
+    charset = 'utf-8'
 
     def render(self, data, accepted_media_type=None, renderer_context=None):
         raise NotImplemented('Renderer class requires .render() to be implemented')
@@ -51,6 +50,7 @@ class JSONRenderer(BaseRenderer):
     format = 'json'
     encoder_class = encoders.JSONEncoder
     ensure_ascii = True
+    charset = 'iso-8859-1'
 
     def render(self, data, accepted_media_type=None, renderer_context=None):
         """
@@ -74,7 +74,12 @@ class JSONRenderer(BaseRenderer):
             except (ValueError, TypeError):
                 indent = None
 
-        return json.dumps(data, cls=self.encoder_class, indent=indent, ensure_ascii=self.ensure_ascii)
+        ret = json.dumps(data, cls=self.encoder_class,
+            indent=indent, ensure_ascii=self.ensure_ascii)
+
+        if not self.ensure_ascii:
+            return bytes(ret.encode(self.charset))
+        return ret
 
 
 class UnicodeJSONRenderer(JSONRenderer):
@@ -332,7 +337,7 @@ class BrowsableAPIRenderer(BaseRenderer):
         renderer_context['indent'] = 4
         content = renderer.render(data, accepted_media_type, renderer_context)
 
-        if not all(char in string.printable for char in content):
+        if renderer.charset is None:
             return '[%d bytes of binary content]' % len(content)
 
         return content

rest_framework/response.py

@@ -55,7 +55,11 @@ class Response(SimpleTemplateResponse):
         else:
             content_type = media_type
         self['Content-Type'] = content_type
-        return renderer.render(self.data, media_type, context)
+
+        ret = renderer.render(self.data, media_type, context)
+        if isinstance(ret, six.text_type):
+            return bytes(ret.encode(self.charset))
+        return ret
 
     @property
     def status_text(self):

rest_framework/tests/renderers.py

@@ -1,4 +1,6 @@
 # -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
 from decimal import Decimal
 from django.core.cache import cache
 from django.test import TestCase
@@ -135,7 +137,7 @@ class RendererEndToEndTests(TestCase):
     def test_default_renderer_serializes_content(self):
         """If the Accept header is not set the default renderer should serialize the response."""
         resp = self.client.get('/')
-        self.assertEqual(resp['Content-Type'], RendererA.media_type)
+        self.assertEqual(resp['Content-Type'], RendererA.media_type + '; charset=utf-8')
         self.assertEqual(resp.content, RENDERER_A_SERIALIZER(DUMMYCONTENT))
         self.assertEqual(resp.status_code, DUMMYSTATUS)
 
@@ -143,13 +145,13 @@ class RendererEndToEndTests(TestCase):
         """No response must be included in HEAD requests."""
         resp = self.client.head('/')
         self.assertEqual(resp.status_code, DUMMYSTATUS)
-        self.assertEqual(resp['Content-Type'], RendererA.media_type)
+        self.assertEqual(resp['Content-Type'], RendererA.media_type + '; charset=utf-8')
         self.assertEqual(resp.content, six.b(''))
 
     def test_default_renderer_serializes_content_on_accept_any(self):
         """If the Accept header is set to */* the default renderer should serialize the response."""
         resp = self.client.get('/', HTTP_ACCEPT='*/*')
-        self.assertEqual(resp['Content-Type'], RendererA.media_type)
+        self.assertEqual(resp['Content-Type'], RendererA.media_type + '; charset=utf-8')
         self.assertEqual(resp.content, RENDERER_A_SERIALIZER(DUMMYCONTENT))
         self.assertEqual(resp.status_code, DUMMYSTATUS)
 
@@ -157,7 +159,7 @@ class RendererEndToEndTests(TestCase):
         """If the Accept header is set the specified renderer should serialize the response.
         (In this case we check that works for the default renderer)"""
         resp = self.client.get('/', HTTP_ACCEPT=RendererA.media_type)
-        self.assertEqual(resp['Content-Type'], RendererA.media_type)
+        self.assertEqual(resp['Content-Type'], RendererA.media_type + '; charset=utf-8')
         self.assertEqual(resp.content, RENDERER_A_SERIALIZER(DUMMYCONTENT))
         self.assertEqual(resp.status_code, DUMMYSTATUS)
 
@@ -165,7 +167,7 @@ class RendererEndToEndTests(TestCase):
         """If the Accept header is set the specified renderer should serialize the response.
         (In this case we check that works for a non-default renderer)"""
         resp = self.client.get('/', HTTP_ACCEPT=RendererB.media_type)
-        self.assertEqual(resp['Content-Type'], RendererB.media_type)
+        self.assertEqual(resp['Content-Type'], RendererB.media_type + '; charset=utf-8')
         self.assertEqual(resp.content, RENDERER_B_SERIALIZER(DUMMYCONTENT))
         self.assertEqual(resp.status_code, DUMMYSTATUS)
 
@@ -176,7 +178,7 @@ class RendererEndToEndTests(TestCase):
             RendererB.media_type
         )
         resp = self.client.get('/' + param)
-        self.assertEqual(resp['Content-Type'], RendererB.media_type)
+        self.assertEqual(resp['Content-Type'], RendererB.media_type + '; charset=utf-8')
         self.assertEqual(resp.content, RENDERER_B_SERIALIZER(DUMMYCONTENT))
         self.assertEqual(resp.status_code, DUMMYSTATUS)
 
@@ -193,7 +195,7 @@ class RendererEndToEndTests(TestCase):
             RendererB.format
         )
         resp = self.client.get('/' + param)
-        self.assertEqual(resp['Content-Type'], RendererB.media_type)
+        self.assertEqual(resp['Content-Type'], RendererB.media_type + '; charset=utf-8')
         self.assertEqual(resp.content, RENDERER_B_SERIALIZER(DUMMYCONTENT))
         self.assertEqual(resp.status_code, DUMMYSTATUS)
 
@@ -201,7 +203,7 @@ class RendererEndToEndTests(TestCase):
         """If a 'format' keyword arg is specified, the renderer with the matching
         format attribute should serialize the response."""
         resp = self.client.get('/something.formatb')
-        self.assertEqual(resp['Content-Type'], RendererB.media_type)
+        self.assertEqual(resp['Content-Type'], RendererB.media_type + '; charset=utf-8')
         self.assertEqual(resp.content, RENDERER_B_SERIALIZER(DUMMYCONTENT))
         self.assertEqual(resp.status_code, DUMMYSTATUS)
 
@@ -214,7 +216,7 @@ class RendererEndToEndTests(TestCase):
         )
         resp = self.client.get('/' + param,
                                HTTP_ACCEPT=RendererB.media_type)
-        self.assertEqual(resp['Content-Type'], RendererB.media_type)
+        self.assertEqual(resp['Content-Type'], RendererB.media_type + '; charset=utf-8')
         self.assertEqual(resp.content, RENDERER_B_SERIALIZER(DUMMYCONTENT))
         self.assertEqual(resp.status_code, DUMMYSTATUS)
 
@@ -270,7 +272,7 @@ class UnicodeJSONRendererTests(TestCase):
         obj = {'countries': ['United Kingdom', 'France', 'España']}
         renderer = UnicodeJSONRenderer()
         content = renderer.render(obj, 'application/json')
-        self.assertEqual(content, '{"countries": ["United Kingdom", "France", "España"]}')
+        self.assertEqual(content, '{"countries": ["United Kingdom", "France", "España"]}'.encode('utf-8'))
 
 
 class JSONPRendererTests(TestCase):
@@ -287,7 +289,7 @@ class JSONPRendererTests(TestCase):
         resp = self.client.get('/jsonp/jsonrenderer',
                                HTTP_ACCEPT='application/javascript')
         self.assertEqual(resp.status_code, status.HTTP_200_OK)
-        self.assertEqual(resp['Content-Type'], 'application/javascript')
+        self.assertEqual(resp['Content-Type'], 'application/javascript; charset=iso-8859-1')
         self.assertEqual(resp.content,
             ('callback(%s);' % _flat_repr).encode('ascii'))
 
@@ -298,7 +300,7 @@ class JSONPRendererTests(TestCase):
         resp = self.client.get('/jsonp/nojsonrenderer',
                                HTTP_ACCEPT='application/javascript')
         self.assertEqual(resp.status_code, status.HTTP_200_OK)
-        self.assertEqual(resp['Content-Type'], 'application/javascript')
+        self.assertEqual(resp['Content-Type'], 'application/javascript; charset=iso-8859-1')
         self.assertEqual(resp.content,
             ('callback(%s);' % _flat_repr).encode('ascii'))
 
@@ -310,7 +312,7 @@ class JSONPRendererTests(TestCase):
         resp = self.client.get('/jsonp/nojsonrenderer?callback=' + callback_func,
                                HTTP_ACCEPT='application/javascript')
         self.assertEqual(resp.status_code, status.HTTP_200_OK)
-        self.assertEqual(resp['Content-Type'], 'application/javascript')
+        self.assertEqual(resp['Content-Type'], 'application/javascript; charset=iso-8859-1')
         self.assertEqual(resp.content,
             ('%s(%s);' % (callback_func, _flat_repr)).encode('ascii'))
 

rest_framework/tests/response.py

@@ -101,7 +101,7 @@ class RendererIntegrationTests(TestCase):
     def test_default_renderer_serializes_content(self):
         """If the Accept header is not set the default renderer should serialize the response."""
         resp = self.client.get('/')
-        self.assertEqual(resp['Content-Type'], RendererA.media_type)
+        self.assertEqual(resp['Content-Type'], RendererA.media_type + '; charset=utf-8')
         self.assertEqual(resp.content, RENDERER_A_SERIALIZER(DUMMYCONTENT))
         self.assertEqual(resp.status_code, DUMMYSTATUS)
 
@@ -109,13 +109,13 @@ class RendererIntegrationTests(TestCase):
         """No response must be included in HEAD requests."""
         resp = self.client.head('/')
         self.assertEqual(resp.status_code, DUMMYSTATUS)
-        self.assertEqual(resp['Content-Type'], RendererA.media_type)
+        self.assertEqual(resp['Content-Type'], RendererA.media_type + '; charset=utf-8')
         self.assertEqual(resp.content, six.b(''))
 
     def test_default_renderer_serializes_content_on_accept_any(self):
         """If the Accept header is set to */* the default renderer should serialize the response."""
         resp = self.client.get('/', HTTP_ACCEPT='*/*')
-        self.assertEqual(resp['Content-Type'], RendererA.media_type)
+        self.assertEqual(resp['Content-Type'], RendererA.media_type + '; charset=utf-8')
         self.assertEqual(resp.content, RENDERER_A_SERIALIZER(DUMMYCONTENT))
         self.assertEqual(resp.status_code, DUMMYSTATUS)
 
@@ -123,7 +123,7 @@ class RendererIntegrationTests(TestCase):
         """If the Accept header is set the specified renderer should serialize the response.
         (In this case we check that works for the default renderer)"""
         resp = self.client.get('/', HTTP_ACCEPT=RendererA.media_type)
-        self.assertEqual(resp['Content-Type'], RendererA.media_type)
+        self.assertEqual(resp['Content-Type'], RendererA.media_type + '; charset=utf-8')
         self.assertEqual(resp.content, RENDERER_A_SERIALIZER(DUMMYCONTENT))
         self.assertEqual(resp.status_code, DUMMYSTATUS)
 
@@ -131,7 +131,7 @@ class RendererIntegrationTests(TestCase):
         """If the Accept header is set the specified renderer should serialize the response.
         (In this case we check that works for a non-default renderer)"""
         resp = self.client.get('/', HTTP_ACCEPT=RendererB.media_type)
-        self.assertEqual(resp['Content-Type'], RendererB.media_type)
+        self.assertEqual(resp['Content-Type'], RendererB.media_type + '; charset=utf-8')
         self.assertEqual(resp.content, RENDERER_B_SERIALIZER(DUMMYCONTENT))
         self.assertEqual(resp.status_code, DUMMYSTATUS)
 
@@ -142,7 +142,7 @@ class RendererIntegrationTests(TestCase):
             RendererB.media_type
         )
         resp = self.client.get('/' + param)
-        self.assertEqual(resp['Content-Type'], RendererB.media_type)
+        self.assertEqual(resp['Content-Type'], RendererB.media_type + '; charset=utf-8')
         self.assertEqual(resp.content, RENDERER_B_SERIALIZER(DUMMYCONTENT))
         self.assertEqual(resp.status_code, DUMMYSTATUS)
 
@@ -150,7 +150,7 @@ class RendererIntegrationTests(TestCase):
         """If a 'format' query is specified, the renderer with the matching
         format attribute should serialize the response."""
         resp = self.client.get('/?format=%s' % RendererB.format)
-        self.assertEqual(resp['Content-Type'], RendererB.media_type)
+        self.assertEqual(resp['Content-Type'], RendererB.media_type + '; charset=utf-8')
         self.assertEqual(resp.content, RENDERER_B_SERIALIZER(DUMMYCONTENT))
         self.assertEqual(resp.status_code, DUMMYSTATUS)
 
@@ -158,7 +158,7 @@ class RendererIntegrationTests(TestCase):
         """If a 'format' keyword arg is specified, the renderer with the matching
         format attribute should serialize the response."""
         resp = self.client.get('/something.formatb')
-        self.assertEqual(resp['Content-Type'], RendererB.media_type)
+        self.assertEqual(resp['Content-Type'], RendererB.media_type + '; charset=utf-8')
         self.assertEqual(resp.content, RENDERER_B_SERIALIZER(DUMMYCONTENT))
         self.assertEqual(resp.status_code, DUMMYSTATUS)
 
@@ -167,7 +167,7 @@ class RendererIntegrationTests(TestCase):
         the renderer with the matching format attribute should serialize the response."""
         resp = self.client.get('/?format=%s' % RendererB.format,
                                HTTP_ACCEPT=RendererB.media_type)
-        self.assertEqual(resp['Content-Type'], RendererB.media_type)
+        self.assertEqual(resp['Content-Type'], RendererB.media_type + '; charset=utf-8')
         self.assertEqual(resp.content, RENDERER_B_SERIALIZER(DUMMYCONTENT))
         self.assertEqual(resp.status_code, DUMMYSTATUS)
 
@@ -204,7 +204,8 @@ class Issue807Testts(TestCase):
         """
         headers = {"HTTP_ACCEPT": RendererA.media_type}
         resp = self.client.get('/', **headers)
-        self.assertEqual(RendererA.media_type, resp['Content-Type'])
+        expected = "{0}; charset={1}".format(RendererA.media_type, 'utf-8')
+        self.assertEqual(expected, resp['Content-Type'])
 
     def test_if_there_is_charset_specified_on_renderer_it_gets_appended(self):
         """