More docs, some API permutation

This commit is contained in:
Andrew Godwin 2015-07-12 21:25:53 -05:00
parent aa921b1659
commit 8492dcde48
7 changed files with 278 additions and 34 deletions

View File

@ -99,5 +99,7 @@ class RedisChannelBackend(BaseChannelBackend):
-1,
)
# TODO: send_group efficient implementation using Lua
def __str__(self):
return "%s(host=%s, port=%s)" % (self.__class__.__name__, self.host, self.port)

View File

@ -1,8 +1,6 @@
import random
import string
from django.utils import six
from channels import channel_backends, DEFAULT_CHANNEL_BACKEND
@ -53,22 +51,6 @@ class Channel(object):
from channels.adapters import view_producer
return view_producer(self.name)
@classmethod
def consumer(self, *channels, alias=DEFAULT_CHANNEL_BACKEND):
"""
Decorator that registers a function as a consumer.
"""
# Upconvert if you just pass in a string
if isinstance(channels, six.string_types):
channels = [channels]
# Get the channel
channel_backend = channel_backends[alias]
# Return a function that'll register whatever it wraps
def inner(func):
channel_backend.registry.add_consumer(func, channels)
return func
return inner
class Group(object):
"""

24
channels/decorators.py Normal file
View File

@ -0,0 +1,24 @@
import functools
from django.utils import six
from channels import channel_backends, DEFAULT_CHANNEL_BACKEND
def consumer(self, *channels, alias=DEFAULT_CHANNEL_BACKEND):
"""
Decorator that registers a function as a consumer.
"""
# Upconvert if you just pass in a string
if isinstance(channels, six.string_types):
channels = [channels]
# Get the channel
channel_backend = channel_backends[alias]
# Return a function that'll register whatever it wraps
def inner(func):
channel_backend.registry.add_consumer(func, channels)
return func
return inner
# TODO: Sessions, auth

View File

@ -114,6 +114,7 @@ class WebsocketTwistedInterface(object):
"""
while True:
channels = self.factory.send_channels()
# TODO: Send keepalives
# Quit if reactor is stopping
if not reactor.running:
return

View File

@ -1,5 +1,24 @@
Channel Backend Requirements
============================
Backends
========
Multiple choices of backend are available, to fill different tradeoffs of
complexity, throughput and scalability. You can also write your own backend if
you wish; the API is very simple and documented below.
In-memory
---------
Database
--------
Redis
-----
Writing Custom Backends
-----------------------
Backend Requirements
^^^^^^^^^^^^^^^^^^^^
While the channel backends are pluggable, there's a minimum standard they
must meet in terms of the way they perform.
@ -23,12 +42,14 @@ In particular, a channel backend MUST:
* Preserve the ordering of messages inside a channel
* Never deliver a message more than once
* Never deliver a message more than once (design for at-most-once delivery)
* Never block on sending of a message (dropping the message/erroring is preferable to blocking)
* Be able to store messages of at least 5MB in size
* Allow for channel and group names of up to 200 printable ASCII characters
* Expire messages only after the expiry period provided is up (a backend may
keep them longer if it wishes, but should expire them at some reasonable
point to ensure users do not come to rely on permanent messages)
@ -41,6 +62,10 @@ In addition, it SHOULD:
* Provide a ``send_group()`` method which sends a message to every channel
in a group.
* Make ``send_group()`` perform better than ``O(n)``, where ``n`` is the
number of members in the group; preferably send the messages to all
members in a single call to your backing datastore or protocol.
* Try and preserve a rough global ordering, so that one busy channel does not
drown out an old message in another channel if a worker is listening on both.

View File

@ -21,9 +21,10 @@ handling and handles every HTTP request directly. Make a new project, a new
app, and put this in a ``consumers.py`` file in the app::
from channels import Channel
from channels.decorators import consumer
from django.http import HttpResponse
@Channel.consumer("django.wsgi.request")
@consumer("django.wsgi.request")
def http_consumer(response_channel, path, **kwargs):
response = HttpResponse("Hello world! You asked for %s" % path)
Channel(response_channel).send(response.channel_encode())
@ -46,7 +47,7 @@ do any time. Let's try some WebSockets, and make a basic chat server!
Delete that consumer from above - we'll need the normal Django view layer to
serve templates later - and make this WebSocket consumer instead::
@Channel.consumer("django.websocket.connect")
@consumer("django.websocket.connect")
def ws_connect(channel, send_channel, **kwargs):
Group("chat").add(send_channel)
@ -70,14 +71,14 @@ so we can hook that up to re-add the channel (it's safe to add the channel to
a group it's already in - similarly, it's safe to discard a channel from a
group it's not in)::
@Channel.consumer("django.websocket.keepalive")
@consumer("django.websocket.keepalive")
def ws_keepalive(channel, send_channel, **kwargs):
Group("chat").add(send_channel)
Of course, this is exactly the same code as the ``connect`` handler, so let's
just combine them::
@Channel.consumer("django.websocket.connect", "django.websocket.keepalive")
@consumer("django.websocket.connect", "django.websocket.keepalive")
def ws_add(channel, send_channel, **kwargs):
Group("chat").add(send_channel)
@ -85,7 +86,7 @@ And, even though channels will expire out, let's add an explicit ``disconnect``
handler to clean up as people disconnect (most channels will cleanly disconnect
and get this called)::
@Channel.consumer("django.websocket.disconnect")
@consumer("django.websocket.disconnect")
def ws_disconnect(channel, send_channel, **kwargs):
Group("chat").discard(send_channel)
@ -94,15 +95,18 @@ Now, that's taken care of adding and removing WebSocket send channels for the
we're not going to store a history of messages or anything and just replay
any message sent in to all connected clients. Here's all the code::
@Channel.consumer("django.websocket.connect", "django.websocket.keepalive")
from channels import Channel
from channels.decorators import consumer
@consumer("django.websocket.connect", "django.websocket.keepalive")
def ws_add(channel, send_channel, **kwargs):
Group("chat").add(send_channel)
@Channel.consumer("django.websocket.receive")
@consumer("django.websocket.receive")
def ws_message(channel, send_channel, content, **kwargs):
Group("chat").send(content=content)
@Channel.consumer("django.websocket.disconnect")
@consumer("django.websocket.disconnect")
def ws_disconnect(channel, send_channel, **kwargs):
Group("chat").discard(send_channel)
@ -125,10 +129,10 @@ though, and so you'll need to configure a channel backend to allow the
channels to run over the network. By default, when you're using Django out of
the box, the channel backend is set to an in-memory one that only works in
process; this is enough to serve normal WSGI style requests (``runserver`` is
just running a WSGI interface and a worker in two threads), but now we want
just running a WSGI interface and a worker in two separate threads), but now we want
WebSocket support we'll need a separate process to keep things clean.
For simplicity, we'll configure the database backend - this uses two tables
For simplicity, we'll use the database channel backend - this uses two tables
in the database to do message handling, and isn't particularly fast but
requires no extra dependencies. Put this in your ``settings.py`` file::
@ -149,7 +153,8 @@ and their performance considerations if you wish.
The second thing, once we have a networked channel backend set up, is to make
sure we're running the WebSocket interface server. Even in development, we need
to do this; ``runserver`` will take care of normal Web requests and running
a worker for us, but WebSockets require an in-process async solution.
a worker for us, but WebSockets isn't compatible with WSGI and needs to run
separately.
The easiest way to do this is to use the ``runwsserver`` management command
that ships with Django; just make sure you've installed the latest release
@ -179,5 +184,210 @@ receive the message and show an alert.
Feel free to put some calls to ``print`` in your handler functions too, if you
like, so you can understand when they're called. If you run three or four
copies of ``runworker``, too, you will probably be able to see the tasks running
copies of ``runworker`` you'll probably be able to see the tasks running
on different workers.
Authentication
--------------
Now, of course, a WebSocket solution is somewhat limited in scope without the
ability to live with the rest of your website - in particular, we want to make
sure we know what user we're talking to, in case we have things like private
chat channels (we don't want a solution where clients just ask for the right
channels, as anyone could change the code and just put in private channel names)
It can also save you having to manually make clients ask for what they want to
see; if I see you open a WebSocket to my "updates" endpoint, and I know which
user ID, I can just auto-add that channel to all the relevant groups (mentions
of that user, for example).
Handily, as WebSockets start off using the HTTP protocol, they have a lot of
familiar features, including a path, GET parameters, and cookies. Notably,
the cookies allow us to perform authentication using the same methods the
normal Django middleware does. Middleware only runs on requests to views,
however, and not on raw consumer calls; it's tailored to work with single
HTTP requests, and so we need a different solution to authenticate WebSockets.
In addition, we don't want the interface servers storing data or trying to run
authentication; they're meant to be simple, lean, fast processes without much
state, and so we'll need to do our authentication inside our consumer functions.
Fortunately, because Channels has standardised WebSocket event
:doc:`message-standards`, it ships with decorators that help you with
authentication, as well as using Django's session framework (which authentication
relies on).
All we need to do is add the ``websocket_auth`` decorator to our views,
and we'll get extra ``session`` and ``user`` keyword arguments we can use;
let's make one where users can only chat to people with the same first letter
of their username::
from channels import Channel
from channels.decorators import consumer, websocket_auth
@consumer("django.websocket.connect", "django.websocket.keepalive")
@websocket_auth
def ws_add(channel, send_channel, user, **kwargs):
Group("chat-%s" % user.username[0]).add(send_channel)
@consumer("django.websocket.receive")
@websocket_auth
def ws_message(channel, send_channel, content, user, **kwargs):
Group("chat-%s" % user.username[0]).send(content=content)
@consumer("django.websocket.disconnect")
@websocket_auth
def ws_disconnect(channel, send_channel, user, **kwargs):
Group("chat-%s" % user.username[0]).discard(send_channel)
(Note that we always end consumers with ``**kwargs``; this is to save us
from writing out all variables we might get sent and to allow forwards-compatibility
with any additions to the message formats in the future)
Persisting Data
---------------
Doing chatrooms by username first letter is a nice simple example, but it's
skirting around the real design pattern - persistent state for connections.
A user may open our chat site and select the chatroom to join themselves, so we
should let them send this request in the initial WebSocket connection,
check they're allowed to access it, and then remember which room a socket is
connected to when they send a message in so we know which group to send it to.
The ``send_channel`` is our unique pointer to the open WebSocket - as you've
seen, we do all our operations on it - but it's not something we can annotate
with data; it's just a simple string, and even if we hack around and set
attributes on it that's not going to carry over to other workers.
Instead, the solution is to persist information keyed by the send channel in
some other data store - sound familiar? This is what Django's session framework
does for HTTP requests, only there it uses cookies as the lookup key rather
than the ``send_channel``.
Now, as you saw above, you can use the ``websocket_auth`` decorator to get
both a ``user`` and a ``session`` variable in your message arguments - and,
indeed, there is a ``websocket_session`` decorator that will just give you
the ``session`` attribute.
However, that session is based on cookies, and so follows the user round the
site - it's great for information that should persist across all WebSocket and
HTTP connections, but not great for information that is specific to a single
WebSocket (such as "which chatroom should this socket be connected to"). For
this reason, Channels also provides a ``websocker_channel_session`` decorator,
which adds a ``channel_session`` attribute to the message; this works just like
the normal ``session`` attribute, and persists to the same storage, but varies
per-channel rather than per-cookie.
Let's use it now to build a chat server that expects you to pass a chatroom
name in the path of your WebSocket request (we'll ignore auth for now)::
from channels import Channel
from channels.decorators import consumer, websocket_channel_session
@consumer("django.websocket.connect")
@websocket_channel_session
def ws_connect(channel, send_channel, path, channel_session, **kwargs):
# Work out room name from path (ignore slashes)
room = path.strip("/")
# Save room in session and add us to the group
channel_session['room'] = room
Group("chat-%s" % room).add(send_channel)
@consumer("django.websocket.keepalive")
@websocket_channel_session
def ws_add(channel, send_channel, channel_session, **kwargs):
Group("chat-%s" % channel_session['room']).add(send_channel)
@consumer("django.websocket.receive")
@websocket_channel_session
def ws_message(channel, send_channel, content, channel_session, **kwargs):
Group("chat-%s" % channel_session['room']).send(content=content)
@consumer("django.websocket.disconnect")
@websocket_channel_session
def ws_disconnect(channel, send_channel, channel_session, **kwargs):
Group("chat-%s" % channel_session['room']).discard(send_channel)
If you play around with it from the console (or start building a simple
JavaScript chat client that appends received messages to a div), you'll see
that you can now request which chat room you want in the initial request. We
could easily add in the auth decorator here too and do an initial check in
``connect`` that the user had permission to join that chatroom.
Models
------
So far, we've just been taking incoming messages and rebroadcasting them to
other clients connected to the same group, but this isn't that great; really,
we want to persist messages to a datastore, and we'd probably like to be
able to inject messages into chatrooms from things other than WebSocket client
connections (perhaps a built-in bot, or server status messages).
Thankfully, we can just use Django's ORM to handle persistence of messages and
easily integrate the send into the save flow of the model, rather than the
message receive - that way, any new message saved will be broadcast to all
the appropriate clients, no matter where it's saved from.
We'll even take some performance considerations into account - We'll make our
own custom channel for new chat messages and move the model save and the chat
broadcast into that, meaning the sending process/consumer can move on
immediately and not spend time waiting for the database save and the
(slow on some backends) ``Group.send()`` call.
Let's see what that looks like, assuming we
have a ChatMessage model with ``message`` and ``room`` fields::
from channels import Channel
from channels.decorators import consumer, websocket_channel_session
from .models import ChatMessage
@consumer("chat-messages")
def msg_consumer(room, message):
# Save to model
ChatMessage.objects.create(room=room, message=message)
# Broadcast to listening sockets
Group("chat-%s" % room).send(message)
@consumer("django.websocket.connect")
@websocket_channel_session
def ws_connect(channel, send_channel, path, channel_session, **kwargs):
# Work out room name from path (ignore slashes)
room = path.strip("/")
# Save room in session and add us to the group
channel_session['room'] = room
Group("chat-%s" % room).add(send_channel)
@consumer("django.websocket.keepalive")
@websocket_channel_session
def ws_add(channel, send_channel, channel_session, **kwargs):
Group("chat-%s" % channel_session['room']).add(send_channel)
@consumer("django.websocket.receive")
@websocket_channel_session
def ws_message(channel, send_channel, content, channel_session, **kwargs):
# Stick the message onto the processing queue
Channel("chat-messages").send(room=channel_session['room'], message=content)
@consumer("django.websocket.disconnect")
@websocket_channel_session
def ws_disconnect(channel, send_channel, channel_session, **kwargs):
Group("chat-%s" % channel_session['room']).discard(send_channel)
Note that we could add messages onto the ``chat-messages`` channel from anywhere;
inside a View, inside another model's ``post_save`` signal, inside a management
command run via ``cron``. If we wanted to write a bot, too, we could put its
listening logic inside the ``chat-messages`` consumer, as every message would
pass through it.
Next Steps
----------
That covers the basics of using Channels; you've seen not only how to use basic
channels, but also seen how they integrate with WebSockets, how to use groups
to manage logical sets of channels, and how Django's session and authentication
systems easily integrate with WebSockets.
We recommend you read through the rest of the reference documentation to see
all of what Channels has to offer; in particular, you may want to look at
our :doc:`deployment` and :doc:`scaling` resources to get an idea of how to
design and run apps in production environments.

View File

@ -26,4 +26,4 @@ Contents:
integration-changes
message-standards
scaling
backend-requirements
backends