Commit Graph

422 Commits

Author SHA1 Message Date
Alexander Kukushkin
6cff5a3e08 Smart replication feedback
This commit makes psycopg2 responsible for sending the status update
(feedback) messages to the server regardless of whether a synchronous or
asynchronous connection is used.

Feedback is sent every *status_update* (default value is 10) seconds,
which could be configured by passing a corresponding parameter to the
`start_replication()` or `start_replication_expert()` methods.
The actual feedback message is sent by the
`pq_read_replication_message()` when the *status_update* timeout is
reached.

The default behavior of the `send_feedback()` method is changed.
It doesn't send a feedback message on every call anymore but just
updates internal structures. There is still a way to *force* sending
a message if *force* or *reply* parameters are set.

The new approach has certain advantages:
1. The client can simply call the `send_feedback()` for every
   processed message and the library will take care of not overwhelming
   the server. Actually, in the synchronous mode it is even mandatory
   to confirm every processed message.
2. The library tracks internally the pointer of the last received
   message which is not keepalive. If the client confirmed the last
   message and after that server sends only keepalives with increasing
   *wal_end*, the library can safely move forward *flush* position to
   the *wal_end* and later automatically report it to the server.

Reporting of the *wal_end* received from keepalive messages is very
important. Not doing so casing:
1. Excessive disk usage, because the replication slot prevents from
   WAL being cleaned up.
2. The smart and fast shutdown of the server could last indefinitely
   because walsender waits until the client report *flush* position
   equal to the *wal_end*.

This implementation is only extending the existing API and therefore
should not break any of the existing code.
2019-05-06 10:27:44 +02:00
Daniele Varrazzo
4058f363d6 More robust RealDictRow population
In the presence of repeated columns the mapping would have remained into
into the dictionary. Now it is removed.

Fix #884
2019-04-06 21:17:28 +01:00
Daniele Varrazzo
cc815e8e8d RealDictRow inherits from OrderedDict
Now its state is unmodified, so apart from special-casing creation
and initial population can work unmodified, and all the desired
properties just work (modifiability, picklability...)

Close #886.
2019-04-06 19:43:31 +01:00
Daniele Varrazzo
21d16b6f67 Fixed RealDictCursor.pop()
Addresses #886, but there might be something else broken there.
2019-04-06 19:43:09 +01:00
Daniele Varrazzo
b4b470c29a Allow 0 as key in the pool
Close #848
2019-03-17 18:24:50 +00:00
Daniele Varrazzo
dfb301b42b Use PY2, PY3 for conditional code instead of sys.version_info 2019-03-16 19:54:40 +00:00
Daniele Varrazzo
f4a2630f1a Added PY2, PY3 to compat module 2019-03-16 19:54:40 +00:00
Jon Dufresne
8685120485 Remove unnecessary except ImportError for Python datetime types
The Python datetime module is available on all supported platforms. The
types are always available and never raise an ImportError.
2019-03-16 16:35:51 +00:00
Jon Dufresne
3f890f8bbe Use True/False instead of 1/0 to represent bool values
Slightly more modern, readable, and Pythonic.
2019-03-13 11:13:36 +00:00
Jon Dufresne
03bb44dd2c Convert while 1: statements to while True:
A slightly more readable and modern syntax.
2019-03-13 11:13:05 +00:00
David Fischer
147ff65e4a Fix typo 2019-02-26 11:33:48 +01:00
Daniele Varrazzo
599432552a Copyright year updated
ag -l Copyright | xargs sed -i \
        "s/\(.*copyright (C) [0-9]\+\)\(-[0-9]\+\)\?\(.*\)/\1-$(date +%Y)\3/I"
2019-02-17 01:36:36 +00:00
Daniele Varrazzo
30c1befa64 SQLSTATE error classes implemented in C
The module is only used to export them to Python.
2019-02-10 03:45:14 +00:00
Daniele Varrazzo
35ec7ad9c1 Use a proper LRU cache for namedtuples
Previous one didn't refresh by last use. Use the stdlib version for py3
and one of our own for py2.

Max size set to 512, which should be fine for everyone (tweaking is
still possible by monkeypatching, as the tests do, but I don't want to
make an interface of it).
2019-02-02 19:29:20 +00:00
Daniele Varrazzo
805527fcd6 Added caching of types generated by NamedTupleCursor
see #838
2019-02-02 14:22:25 +00:00
Daniele Varrazzo
ddbe495d70 Added BYTESARRAY typecaster 2019-01-18 15:47:01 +00:00
Daniele Varrazzo
f713dc9fc1 Preliminary test for a BYTES adapter.
Allow returning unparsed bytes from databases with mixed encodings. See
issue #519.
2019-01-18 13:17:02 +00:00
Daniele Varrazzo
f3695e36c7 Merge remote-tracking branch 'eternalflow/execute-values-returning-clause-support' 2018-12-27 14:53:12 +01:00
Daniele Varrazzo
bde9fc6dea Docs wordsmith for execute_values() fetch param 2018-12-27 14:42:58 +01:00
Daniele Varrazzo
6fdac46137 Some harmless adjustments in execute_values() code
No optional result, and create the list only if requred.
2018-12-27 14:39:13 +01:00
Jon Dufresne
a1fdaebc30 Dropped deprecated PersistentConnectionPool
This class was deprecated in
27cd6c4880 (Dec 2, 2012), which was first
included in release 2.5. Enough time has passed for library uses to find
an alternative solution.

This class was untested.
2018-12-04 06:32:00 -08:00
Ivan Kotelnikov
b40ac15efc rename param name to fetch 2018-11-24 20:51:18 +02:00
Ivan Kotelnikov
9a8d7d5a6a add RETURNING clause support in execute_values function 2018-11-21 16:38:13 +02:00
Daniele Varrazzo
8fb0f694f7 Don't barf on Composite passed to execute_values()
Close #794
2018-10-23 01:57:48 +01:00
Daniele Varrazzo
05f9e231a0 Full flake8 3.5 cleanup 2018-10-23 00:39:14 +01:00
Daniele Varrazzo
1bb3d5cfe2 Dropped duplicate classes in errors module
Also using a more compact class registration using a decorator
2018-10-23 00:31:57 +01:00
Daniele Varrazzo
0236c68da9 Merge branch 'connection-info' 2018-10-15 01:24:00 +01:00
Daniele Varrazzo
b205764fdd Merge branch 'master' into errors-module 2018-10-15 00:58:32 +01:00
Daniele Varrazzo
e7227ce87b Added errors.lookup() function 2018-10-15 00:56:51 +01:00
Daniele Varrazzo
61df7bdd8d Errors module content updated to Postgres 11 2018-10-14 23:07:11 +01:00
Daniele Varrazzo
7a5edff6c6 errorcodes map update to PostgreSQL 11 2018-10-14 22:57:48 +01:00
Daniele Varrazzo
44bd2927c5 Use the connection.info properties instead of the legacy methods 2018-10-13 03:28:42 +01:00
Daniele Varrazzo
9ddf59959f Adding ConnectionInfo object documentation
I'm still fought whether docs should be in the C module or in the .rst.
I'd prefer the first because DRY, but writing multiline strings in C
really sucks.
2018-10-12 03:25:06 +01:00
Daniele Varrazzo
9148157697 Merge branch 'master' into https 2018-10-10 22:07:33 +01:00
Daniele Varrazzo
4aa02b7855 sql.Identifier can wrap a sequence of strings to represent qualified names
Close #732.
2018-10-04 12:46:10 +01:00
Daniele Varrazzo
8bd7ad7bb5 Faster check for empty range in str() 2018-10-04 11:46:07 +01:00
Michel Albert
d317977205 Implement __str__ for range types 2018-09-30 10:15:55 +02:00
Jon Dufresne
b07e34e0b8 Prefer https:// URLs when available 2018-09-22 19:02:33 -07:00
Daniel Hahler
6af55ee51c Improve doc for extras.wait_select 2018-09-05 16:45:33 +01:00
Daniele Varrazzo
782fa39647 Generating the whole errors file from script 2018-08-17 02:00:40 +01:00
Daniele Varrazzo
ce1ac3aea9 Read exceptions to raise from a Python module 2018-08-17 02:00:01 +01:00
Daniele Varrazzo
0bb7d0db48 DictCursor and RealDictCursor rows maintain columns order
Close #177.
2018-05-21 03:14:08 +01:00
Daniele Varrazzo
7bdaf0affd Restored methods iter*() on dict cursors rows 2018-05-21 02:51:37 +01:00
Daniele Varrazzo
e8a831dda2 Merge branch 'master' into drop-2to3 2018-05-20 23:56:29 +01:00
Daniele Varrazzo
0161d54dbb Merge branch 'master' into encrypt-pass 2018-05-20 17:57:34 +01:00
Daniele Varrazzo
bc84b6233e Allow non-ascii chars in namedtuple fields
They can be valid chars in Python 3. Or maybe not? In which case Python
will throw an exception, but that's fine.

Fix regression introduced fixing #211
2018-05-18 12:15:50 +01:00
Ashesh Vashi
1bec2bdc43
Merge branch 'master' into master 2018-05-08 15:17:59 +05:30
Daniele Varrazzo
ddb87b7727 Convert fields names into valid Python identifiers in NamedTupleCursor
Close #211.
2018-01-29 03:30:46 +00:00
Jon Dufresne
8ad2098b74 Drop 2to3 build step; make all code compatible with all Pythons
Make all library code compatible with both Python 2 and Python 3. Helps
move to modern Python idioms. Can now write for Python 3 (with
workarounds for Python 2) instead of the other way around.

In the future, when it is eventually time to drop Python 2, the library
will be in a better position to remove workarounds

Added a very small comparability module compat.py where required. It
includes definitions for:

- text_type -- A type. str on Python 3. unicode on Python 2.
- string_types -- A tuple. Contains only str on Python 3. Contains str &
                  unicode on Python 2.
2017-12-11 20:26:58 -08:00
Jon Dufresne
f35465231f Drop the Python 2 style interface from DictRow
Now standardizes on the Python 3 interface for all uses. Makes behavior
of DictRow between Pythons more consistent and predictable.
2017-12-11 20:02:16 -08:00