From a54932ee9c086f658881b4d1141646d82fdc98db Mon Sep 17 00:00:00 2001 From: Daniele Varrazzo Date: Sun, 4 Apr 2010 03:10:18 +0100 Subject: [PATCH] Added documentation for the green features. --- doc/src/advanced.rst | 69 ++++++++++++++++++++++++++++++++++++++++++ doc/src/connection.rst | 10 +++--- doc/src/extensions.rst | 18 ++++++++++- doc/src/extras.rst | 8 +++++ psycopg/green.h | 15 +++++---- 5 files changed, 109 insertions(+), 11 deletions(-) diff --git a/doc/src/advanced.rst b/doc/src/advanced.rst index b8ed5375..13def38f 100644 --- a/doc/src/advanced.rst +++ b/doc/src/advanced.rst @@ -362,6 +362,75 @@ this will be probably implemented in a future release. + +.. index:: + single: Greenlet, Coroutine, eventlet, gevent, Wait callback + +.. _green-support: + +Support to coroutine libraries +------------------------------ + +.. versionadded:: 2.2.0 + +Psycopg can be used together with coroutine_\-based libraries, and participate +to cooperative multithread. + +Coroutine-based libraries (such as Eventlet_ or gevent_) can usually patch the +Python standard library in order to enable a coroutine switch in presence of +blocking I/O: the process is usually referred as making the system *green*, in +reference to greenlet_, the basic Python micro-thread library. + +Because Psycopg is a C extension module, it is not possible for coroutine +libraries to patch it: Psycopg instead enables cooperative multithreading by +allowing the registration of a *wait callback* using the +`psycopg2.extensions.set_wait_callback()` function. When a wait callback is +registered, Psycopg will use `libpq non-blocking calls`__ instead of the regular +blocking ones, and will delegate to the callback the responsibility to wait +for available data. + +This way of working is less flexible of complete asynchronous I/O, but has the +advantage of maintaining a complete |DBAPI| semantics: from the point of view +of the end user, all Psycopg functions and objects will work transparently +in the coroutine environment (the calling coroutine will be blocked while +other coroutines can be scheduled to run), allowing non modified code and +third party libraries (such as SQLAlchemy_) to be used in coroutine-based +programs. + +Notice that, while I/O correctly yields control to other coroutines, each +connection has a lock allowing a single cursor at time to communicate with the +backend: such lock is not *green*, so blocking against it would block the +entire program waiting for data, not the single coroutine. Therefore, +programmers are advised to either avoid to share connections between coroutines +or to use a library-friendly lock to synchronize shares connections, e.g. for +pooling. + +Coroutine libraries authors should provide a callback implementation (and +probably register it) to make Psycopg as green as they want. An example +callback (using `!select()` to block) is provided as +`psycopg2.extras.wait_select()`: it boils down to something similar to:: + + def wait_select(conn): + while 1: + state = conn.poll() + if state == extensions.POLL_OK: + break + elif state == extensions.POLL_READ: + select.select([conn.fileno()], [], []) + elif state == extensions.POLL_WRITE: + select.select([], [conn.fileno()], []) + else: + raise OperationalError("bad state from poll: %s" % state) + +.. _coroutine: http://en.wikipedia.org/wiki/Coroutine +.. _greenlet: http://pypi.python.org/pypi/greenlet +.. _Eventlet: http://eventlet.net/ +.. _gevent: http://www.gevent.org/ +.. _SQLAlchemy: http://www.sqlalchemy.org/ +.. __: http://www.postgresql.org/docs/8.4/static/libpq-async.html + + + .. testcode:: :hide: diff --git a/doc/src/connection.rst b/doc/src/connection.rst index 5d7077a3..048bbcd6 100644 --- a/doc/src/connection.rst +++ b/doc/src/connection.rst @@ -321,7 +321,7 @@ The ``connection`` class .. versionadded:: 2.2.0 - .. seealso:: :ref:`Asynchronous support `. + .. seealso:: :ref:`async-support` and :ref:`green-support`. .. attribute:: async @@ -338,9 +338,11 @@ The ``connection`` class Return one of the constants defined in :ref:`poll-constants`. If it returns `~psycopg2.extensions.POLL_OK` then the connection has been estabilished or the query results are available on the client. - Otherwise wait until the file descriptor returned by - `~connection.fileno()` is ready to read or to write, as explained in - :ref:`async-support`. + Otherwise wait until the file descriptor returned by `fileno()` is + ready to read or to write, as explained in :ref:`async-support`. + `poll()` should be also used by the function installed by + `~psycopg2.extensions.set_wait_callback()` as explained in + :ref:`green-support`. `poll()` is also used to receive asynchronous notifications from the database: see :ref:`async-notify` from further details. diff --git a/doc/src/extensions.rst b/doc/src/extensions.rst index 2d43c29e..552dfdcd 100644 --- a/doc/src/extensions.rst +++ b/doc/src/extensions.rst @@ -106,6 +106,14 @@ functionalities defined by the |DBAPI|_. Close the object and remove it from the database. +.. autofunction:: set_wait_callback(f) + + .. versionadded:: 2.2.0 + +.. autofunction:: get_wait_callback() + + .. versionadded:: 2.2.0 + .. _sql-adaptation-objects: @@ -456,7 +464,9 @@ Poll constants .. versionadded:: 2.2.0 These values can be returned by `connection.poll()` during asynchronous -connection and communication. See :ref:`async-support`. +connection and communication. They match the values in the libpq enum +`!PostgresPollingStatusType`. See :ref:`async-support` and +:ref:`green-support`. .. data:: POLL_OK @@ -481,6 +491,12 @@ connection and communication. See :ref:`async-support`. select.select([], [conn.fileno()], []) +.. data:: POLL_ERROR + + There was a problem during connection polling. This value should actually + never be returned: in case of poll error usually an exception containing + the relevant details is raised. + Additional database types diff --git a/doc/src/extras.rst b/doc/src/extras.rst index 9eeddccf..9751d19c 100644 --- a/doc/src/extras.rst +++ b/doc/src/extras.rst @@ -155,3 +155,11 @@ Fractional time zones .. versionadded:: 2.0.9 +.. index:: + pair: Example; Coroutine; + +Coroutine support +----------------- + +.. autofunction:: wait_select(conn) + diff --git a/psycopg/green.h b/psycopg/green.h index 16e83046..175f0552 100644 --- a/psycopg/green.h +++ b/psycopg/green.h @@ -34,23 +34,26 @@ extern "C" { #endif #define psyco_set_wait_callback_doc \ -"set_wait_callback(f) -- Register a callback function to block waiting for data.\n" \ +"Register a callback function to block waiting for data.\n" \ "\n" \ "The callback should have signature :samp:`fun({conn})` and\n" \ "is called to wait for data available whenever a blocking function from the\n" \ -"libpq is called. Use `!register_wait_function(None)` to revert to the\n" \ -"original behaviour (using blocking libpq functions).\n" \ +"libpq is called. Use `!set_wait_callback(None)` to revert to the\n" \ +"original behaviour (i.e. using blocking libpq functions).\n" \ "\n" \ "The function is an hook to allow coroutine-based libraries (such as\n" \ -"eventlet_ or gevent_) to switch when Psycopg is blocked, allowing\n" \ +"Eventlet_ or gevent_) to switch when Psycopg is blocked, allowing\n" \ "other coroutines to run concurrently.\n" \ "\n" \ "See `~psycopg2.extras.wait_select()` for an example of a wait callback\n" \ -"implementation.\n" +"implementation.\n" \ +"\n" \ +".. _Eventlet: http://eventlet.net/\n" \ +".. _gevent: http://www.gevent.org/\n" HIDDEN PyObject *psyco_set_wait_callback(PyObject *self, PyObject *obj); #define psyco_get_wait_callback_doc \ -"get_wait_callback() -- Return the currently registered wait callback.\n" \ +"Return the currently registered wait callback.\n" \ "\n" \ "Return `None` if no callback is currently registered.\n" HIDDEN PyObject *psyco_get_wait_callback(PyObject *self, PyObject *obj);