Commit Graph

3003 Commits

Author SHA1 Message Date
Daniele Varrazzo
668d507c34 Merge branch 'feature/smart-feedback' 2019-05-12 23:55:11 +09:00
Daniele Varrazzo
b79895186c Added news entry about smart replication feedback 2019-05-12 23:48:51 +09:00
Alexander Kukushkin
90755e6f13 Address code-review 2019-05-07 14:18:09 +02:00
Alexander Kukushkin
5eec11f232 Improve docs 2019-05-06 15:42:37 +02:00
Alexander Kukushkin
f827e49f55 Change the default value of keepalive_interval parameter to None
The previous default value was 10 seconds, what might cause silent
overwrite of the *status_interval* specified in the `start_replication()`
2019-05-06 15:26:21 +02:00
Alexander Kukushkin
6cff5a3e08 Smart replication feedback
This commit makes psycopg2 responsible for sending the status update
(feedback) messages to the server regardless of whether a synchronous or
asynchronous connection is used.

Feedback is sent every *status_update* (default value is 10) seconds,
which could be configured by passing a corresponding parameter to the
`start_replication()` or `start_replication_expert()` methods.
The actual feedback message is sent by the
`pq_read_replication_message()` when the *status_update* timeout is
reached.

The default behavior of the `send_feedback()` method is changed.
It doesn't send a feedback message on every call anymore but just
updates internal structures. There is still a way to *force* sending
a message if *force* or *reply* parameters are set.

The new approach has certain advantages:
1. The client can simply call the `send_feedback()` for every
   processed message and the library will take care of not overwhelming
   the server. Actually, in the synchronous mode it is even mandatory
   to confirm every processed message.
2. The library tracks internally the pointer of the last received
   message which is not keepalive. If the client confirmed the last
   message and after that server sends only keepalives with increasing
   *wal_end*, the library can safely move forward *flush* position to
   the *wal_end* and later automatically report it to the server.

Reporting of the *wal_end* received from keepalive messages is very
important. Not doing so casing:
1. Excessive disk usage, because the replication slot prevents from
   WAL being cleaned up.
2. The smart and fast shutdown of the server could last indefinitely
   because walsender waits until the client report *flush* position
   equal to the *wal_end*.

This implementation is only extending the existing API and therefore
should not break any of the existing code.
2019-05-06 10:27:44 +02:00
Daniele Varrazzo
f96982bdfd Merge remote-tracking branch 'origin/appveyor-py' 2019-05-01 16:30:09 +01:00
Daniele Varrazzo
ed7d8ea28c Appveyor: added package_name to options 2019-04-22 22:39:59 +01:00
Daniele Varrazzo
014097c1af Dropped command line config from appveyor scrips
Only use env vars, they were unused.

Use consistently a config object with properties instead of functions
(the one returning a binary are especially dangerous if parens are
forgotten).

Also add helpers to call the target python more succinctly.
2019-04-22 12:41:44 +01:00
Daniele Varrazzo
1b2c1d620f Run tests more quiet/faster building wheels
We are mostly interested it installed alright.
2019-04-22 11:24:29 +01:00
Daniele Varrazzo
285c64d101 Better dir names in appveyor build 2019-04-22 11:24:29 +01:00
Daniele Varrazzo
637a990e09 Added support for wheel building and uploading
To be used by the psycopg/psycopg2-wheels project.
2019-04-22 11:24:29 +01:00
Daniele Varrazzo
9eec303cf7 Configure postgres to run appveyor tests on ssl 2019-04-22 11:24:29 +01:00
Daniele Varrazzo
b1078b1b92 Setup build environment only before building 2019-04-22 11:24:29 +01:00
Daniele Varrazzo
1178501aaf appveyor: added logging level configuration 2019-04-22 11:24:29 +01:00
Daniele Varrazzo
5c72203180 Using pathlib to manipulate paths 2019-04-22 11:24:29 +01:00
Daniele Varrazzo
591476621c Dropped problematic init step
It was performed before repos cloned so no resource available (including
the script!)
2019-04-22 11:24:29 +01:00
Daniele Varrazzo
fda738c90d All together now, let's make this real 2019-04-22 02:54:56 +01:00
Daniele Varrazzo
5858b0b9b4 Test packages from Python 2019-04-22 02:54:56 +01:00
Daniele Varrazzo
169ce22228 Build psycopg from Python 2019-04-22 02:54:56 +01:00
Daniele Varrazzo
c875197432 Build libpq from Python 2019-04-22 02:54:56 +01:00
Daniele Varrazzo
73f6a0cd95 Build openssl from Python 2019-04-22 02:54:56 +01:00
Daniele Varrazzo
37ce131d2c Added setenv to log env vars changes 2019-04-22 02:54:56 +01:00
Daniele Varrazzo
00fc2820a0 Added script to implement appveyor functionality in Python
Only the init step for the moment.
2019-04-22 02:54:56 +01:00
Daniele Varrazzo
325aadbf2c Check return code of pthread_mutex_init
Close #901
2019-04-21 11:53:34 +01:00
Daniele Varrazzo
39b1994c26 Bump to next dev version number 2019-04-21 11:53:13 +01:00
Daniele Varrazzo
324cded166 Bumping to version 2.8.2 2019-04-14 15:11:31 +01:00
Daniele Varrazzo
6b740df704 Added FAQ entry about the change in binary packages in 2.8 2019-04-14 15:08:07 +01:00
Daniele Varrazzo
4821a6294e Merge branch 'openssl-1.1-windows' 2019-04-14 14:49:41 +01:00
Daniele Varrazzo
26b61e809f Mention building openssl 1.1 on windows in news file
Close #836.
2019-04-11 01:42:54 +01:00
Daniele Varrazzo
14bfc54344 Mention fix to repeated fields in RealDictCursor in news file 2019-04-11 00:54:55 +01:00
Daniele Varrazzo
72fe91c02e Build and test with openssl 1.1 on windows
Porting to openssl 1.1 by Matthew Brett, from the psycopg2-wheel repos.
2019-04-11 00:27:49 +01:00
Daniele Varrazzo
5e01c47818 Highlight in docs that the context manager doesn't close the connection
Code as in #889 is not robust, but the behaviour is actually
counter-intuitive.
2019-04-07 11:19:56 +01:00
Daniele Varrazzo
4058f363d6 More robust RealDictRow population
In the presence of repeated columns the mapping would have remained into
into the dictionary. Now it is removed.

Fix #884
2019-04-06 21:17:28 +01:00
Daniele Varrazzo
63e4bc961e Version bumped to next dev release 2019-04-06 21:16:42 +01:00
Daniele Varrazzo
3357477fde Fixed debug title level in news 2019-04-06 20:49:00 +01:00
Daniele Varrazzo
d2dce4dd17 Fixed link to RealDictRow from last news 2019-04-06 20:22:03 +01:00
Daniele Varrazzo
15d684134f Bumping to version 2.8.1 2019-04-06 20:12:10 +01:00
Daniele Varrazzo
5c4b8a3d1f Merge branch 'fix-886' 2019-04-06 20:09:47 +01:00
Daniele Varrazzo
cc815e8e8d RealDictRow inherits from OrderedDict
Now its state is unmodified, so apart from special-casing creation
and initial population can work unmodified, and all the desired
properties just work (modifiability, picklability...)

Close #886.
2019-04-06 19:43:31 +01:00
Daniele Varrazzo
21d16b6f67 Fixed RealDictCursor.pop()
Addresses #886, but there might be something else broken there.
2019-04-06 19:43:09 +01:00
Daniele Varrazzo
7b31b39fed Merge branch 'fix-887' 2019-04-06 10:51:03 +01:00
Daniele Varrazzo
46106e1b78 Test notifies are received ok polling an async cur 2019-04-05 19:04:48 +01:00
Daniele Varrazzo
755a128ffb Handle ok poll() without a cursor having executed queries
Close #887
2019-04-05 18:04:55 +01:00
Daniele Varrazzo
34d45aea87 Bump to next dev release 2019-04-05 18:02:30 +01:00
Daniele Varrazzo
c64d2448e8 Bumping to 2.8 for release 2019-04-03 10:44:07 +01:00
Daniele Varrazzo
544e157649 Fixed replication cursor docs warning indentation level 2019-04-01 10:54:01 +01:00
Daniele Varrazzo
ed74189acd Replication objects docs header level bumped up
Now it appears in the ToCs.
2019-04-01 10:40:44 +01:00
Daniele Varrazzo
813ca30953 Fixed notice about new wal_end attribute being on the cursor 2019-04-01 10:40:06 +01:00
Daniele Varrazzo
b76ff2fc33 Merge branch 'keepalive-save-wal-end'
Close #800
2019-03-30 21:29:39 +00:00