psycopg2/doc/src/advanced.rst

More advanced topics
====================

.. sectionauthor:: Daniele Varrazzo <daniele.varrazzo@gmail.com>

.. testsetup:: *

    import re

    cur.execute("CREATE TABLE atable (apoint point)")
    conn.commit()

.. index::
    double: Subclassing; Cursor
    double: Subclassing; Connection

.. _subclassing-connection:
.. _subclassing-cursor:

Connection and cursor factories
-------------------------------

Psycopg exposes two new-style classes that can be sub-classed and expanded to
adapt them to the needs of the programmer: `psycopg2.extensions.cursor`
and `psycopg2.extensions.connection`.  The `connection` class is
usually sub-classed only to provide an easy way to create customized cursors
but other uses are possible. `cursor` is much more interesting, because
it is the class where query building, execution and result type-casting into
Python variables happens.

.. index::
    single: Example; Cursor subclass

An example of cursor subclass performing logging is::

    import psycopg2
    import psycopg2.extensions
    import logging

    class LoggingCursor(psycopg2.extensions.cursor):
        def execute(self, sql, args=None):
            logger = logging.getLogger('sql_debug')
            logger.info(self.mogrify(sql, args))

            try:
                psycopg2.extensions.cursor.execute(self, sql, args)
            except Exception, exc:
                logger.error("%s: %s" % (exc.__class__.__name__, exc))
                raise

    conn = psycopg2.connect(DSN)
    cur = conn.cursor(cursor_factory=LoggingCursor)
    cur.execute("INSERT INTO mytable VALUES (%s, %s, %s);",
                 (10, 20, 30))



.. index::
    single: Objects; Creating new adapters
    single: Adaptation; Creating new adapters
    single: Data types; Creating new adapters

.. _adapting-new-types:

Adapting new Python types to SQL syntax
---------------------------------------

Any Python class or type can be adapted to an SQL string.  Adaptation mechanism
is similar to the Object Adaptation proposed in the :pep:`246` and is exposed
by the `psycopg2.extensions.adapt()` function.

The `~cursor.execute()` method adapts its arguments to the
`~psycopg2.extensions.ISQLQuote` protocol.  Objects that conform to this
protocol expose a `!getquoted()` method returning the SQL representation
of the object as a string.

The easiest way to adapt an object to an SQL string is to register an adapter
function via the `~psycopg2.extensions.register_adapter()` function.  The
adapter function must take the value to be adapted as argument and return a
conform object.  A convenient object is the `~psycopg2.extensions.AsIs`
wrapper, whose `!getquoted()` result is simply the `!str()`\ ing
conversion of the wrapped object.

.. index::
    single: Example; Types adaptation

Example: mapping of a `!Point` class into the |point|_ PostgreSQL
geometric type:

.. doctest::

    >>> from psycopg2.extensions import adapt, register_adapter, AsIs

    >>> class Point(object):
    ...    def __init__(self, x, y):
    ...        self.x = x
    ...        self.y = y

    >>> def adapt_point(point):
    ...     return AsIs("'(%s, %s)'" % (adapt(point.x), adapt(point.y)))

    >>> register_adapter(Point, adapt_point)

    >>> cur.execute("INSERT INTO atable (apoint) VALUES (%s)",
    ...             (Point(1.23, 4.56),))


.. |point| replace:: :sql:`point`
.. _point: http://www.postgresql.org/docs/8.4/static/datatype-geometric.html#AEN6084

The above function call results in the SQL command::

    INSERT INTO atable (apoint) VALUES ((1.23, 4.56));



.. index:: Type casting

.. _type-casting-from-sql-to-python:

Type casting of SQL types into Python objects
---------------------------------------------

PostgreSQL objects read from the database can be adapted to Python objects
through an user-defined adapting function.  An adapter function takes two
arguments: the object string representation as returned by PostgreSQL and the
cursor currently being read, and should return a new Python object.  For
example, the following function parses the PostgreSQL :sql:`point`
representation into the previously defined `!Point` class:

    >>> def cast_point(value, cur):
    ...    if value is None:
    ...        return None
    ...
    ...    # Convert from (f1, f2) syntax using a regular expression.
    ...    m = re.match(r"\(([^)]+),([^)]+)\)", value)
    ...    if m:
    ...        return Point(float(m.group(1)), float(m.group(2)))
    ...    else:
    ...        raise InterfaceError("bad point representation: %r" % value)
                

In order to create a mapping from a PostgreSQL type (either standard or
user-defined), its OID must be known. It can be retrieved either by the second
column of the `cursor.description`:

    >>> cur.execute("SELECT NULL::point")
    >>> point_oid = cur.description[0][1]
    >>> point_oid
    600

or by querying the system catalog for the type name and namespace (the
namespace for system objects is :sql:`pg_catalog`):

    >>> cur.execute("""
    ...    SELECT pg_type.oid
    ...      FROM pg_type JOIN pg_namespace
    ...             ON typnamespace = pg_namespace.oid
    ...     WHERE typname = %(typename)s
    ...       AND nspname = %(namespace)s""",
    ...    {'typename': 'point', 'namespace': 'pg_catalog'})
    >>> point_oid = cur.fetchone()[0]
    >>> point_oid
    600

After you know the object OID, you can create and register the new type:

    >>> POINT = psycopg2.extensions.new_type((point_oid,), "POINT", cast_point)
    >>> psycopg2.extensions.register_type(POINT)

The `~psycopg2.extensions.new_type()` function binds the object OIDs
(more than one can be specified) to the adapter function.
`~psycopg2.extensions.register_type()` completes the spell.  Conversion
is automatically performed when a column whose type is a registered OID is
read:

    >>> cur.execute("SELECT '(10.2,20.3)'::point")
    >>> point = cur.fetchone()[0]
    >>> print type(point), point.x, point.y
    <class 'Point'> 10.2 20.3



.. index::
    pair: Asynchronous; Notifications
    pair: LISTEN; SQL command
    pair: NOTIFY; SQL command

.. _async-notify:

Asynchronous notifications
--------------------------

Psycopg allows asynchronous interaction with other database sessions using the
facilities offered by PostgreSQL commands |LISTEN|_ and |NOTIFY|_. Please
refer to the PostgreSQL documentation for examples of how to use this form of
communications.

Notifications received are made available in the `connection.notifies`
list. Notifications can be sent from Python code simply using a :sql:`NOTIFY`
command in an `~cursor.execute()` call.

Because of the way sessions interact with notifications (see |NOTIFY|_
documentation), you should keep the connection in :ref:`autocommit
<autocommit>` mode while sending and receiveng notification.

.. |LISTEN| replace:: :sql:`LISTEN`
.. _LISTEN: http://www.postgresql.org/docs/8.4/static/sql-listen.html
.. |NOTIFY| replace:: :sql:`NOTIFY`
.. _NOTIFY: http://www.postgresql.org/docs/8.4/static/sql-notify.html

.. index::
    single: Example; Asynchronous notification

Example::

    import sys
    import select
    import psycopg2
    import psycopg2.extensions

    conn = psycopg2.connect(DSN)
    conn.set_isolation_level(psycopg2.extensions.ISOLATION_LEVEL_AUTOCOMMIT)

    curs = conn.cursor()
    curs.execute("LISTEN test;")

    print "Waiting for 'NOTIFY test'"
    while 1:
        if select.select([curs],[],[],5)==([],[],[]):
            print "Timeout"
        else:
            if curs.isready():
                print "Got NOTIFY:", curs.connection.notifies.pop()

Running the script and executing the command :sql:`NOTIFY test` in a separate
:program:`psql` shell, the output may look similar to::

    Waiting for 'NOTIFY test'
    Timeout
    Timeout
    Got NOTIFY: (6535, 'test')
    Timeout
    ...



.. index::
    double: Asynchronous; Connection

.. _async-support:

Asynchronous support
--------------------

.. versionadded:: 2.2.0

Psycopg can issue asynchronous queries to a Postgresql database. An asynchronous
communication style is estabilished passing the parameter *async*\=1 to the
`~psycopg2.connect()` function: the returned connection will work in
asynchronous mode.

In asynchronous mode, a Psycopg connection will rely on the caller to poll for
the socket file descriptor ready to accept data or a query result ready to be
read from the server. The caller can use the method `~cursor.fileno()` to get
the connection file descriptor and `~cursor.poll()` to make communication
proceed. An application can use a loop like the one below to transfer data
between the client and the server::

    def wait(conn_or_cur):
        while 1:
            state = conn_or_cur.poll()
            if state == psycopg2.extensions.POLL_OK:
                break
            elif state == psycopg2.extensions.POLL_WRITE:
                select.select([], [conn_or_cur.fileno()], [])
            elif state == psycopg2.extensions.POLL_READ:
                select.select([conn_or_cur.fileno()], [], [])
            else:
                raise psycopg2.OperationalError("poll() returned %s" % state)

After `!poll()` has returned `~psycopg2.extensions.POLL_OK`, the results are
available in the cursor for regular reading::

    curs.execute("SELECT * FROM foo;")
    wait(curs)
    for record in curs:
        # use it...

The same loop should also be used to accomplish a connection with the server:
the connection is usable only after `connection.poll()` has returned `!POLL_OK`.
The `!connection` has a `~connection.fileno()` method too, so it is possible to
use the same interface for the wait loop::

    conn = psycopg2.connect(database='test', async=1)
    wait(conn)
    # Now you can have a cursor.
    curs = conn.cursor()

Notice that there are a few other requirements to be met in order to have a
completely non-blocking connection attempt: see the libpq documentation for
|PQconnectStart|_.

.. |PQconnectStart| replace:: `!PQconnectStart()`
.. _PQconnectStart: http://www.postgresql.org/docs/8.4/static/libpq-connect.html#AEN33199

When an asynchronous query is being executed, `connection.executing()` returns
`True`. Two cursors can't execute concurrent queries on the same asynchronous
connection.

There are several limitations in using asynchronous connections: the connection
is always in :ref:`autocommit <autocommit>` mode and it is not possible to
change it using `~connection.set_isolation_level()`. So transaction are not
started at each query and is not possible to use methods `~connection.commit()`
and `~connection.rollback()`: you can manually control transactions using
`~cursor.execute()` to send commands :sql:`BEGIN`, :sql:`COMMIT` and
:sql:`ROLLBACK`.

With asynchronous connections it is also not possible to use
`~connection.set_client_encoding()`, `~cursor.executemany()`, :ref:`large
objects <large-objects>`, :ref:`named cursors <server-side-cursors>`.

:ref:`COPY commands <copy>` are not supported either in asynchronous mode, but
this will be probably implemented in a future release.



.. testcode::
    :hide:

    conn.rollback()
    cur.execute("DROP TABLE atable")
    conn.commit()
    cur.close()
    conn.close()