Fix & improve PostgreSQL backup/restore scripts (#1571)

* Fix & imporve postgres backup/restore scripts

* Update PostgreSQL backup/restore docs

* Fix postgres Dockerfile regression

* Extend error messages in PostgreSQL maintenance scipts

docs/docker-postgres-backups.rst

@@ -1,40 +1,91 @@
-============================
+PostgreSQL Backups with Docker
-Database Backups with Docker
+==============================
-============================
 
-The database has to be running to create/restore a backup. These examples show local examples. If you want to use it on a remote server, use ``-f production.yml`` instead.
+Prerequisites:
 
-Running Backups
+#. the project was generated with ``use_docker`` set to ``y``.
-================
 
-Run the app with `docker-compose -f local.yml up`.
+For brevity it is assumed that will be running the below commands against local environment, however, this is by no means mandatory so feel free switching to ``production.yml`` when needed.
+
+Note that the application stack should not necessarily be running when applying any of the instructions below, unless explicitly stated otherwise. For instance, suppose the stack has been down for quite some time or have never even been up yet -- rather than starting it beforehand use a single ``$ docker-compose -f local.yml run --rm <command>`` with the desired command. By contrast, should you already have your application up and running do not bother waiting for ``run`` instruction to finish (they usually take a bit longer due to bootstrapping phase), just use ``$ docker-compose -f local.yml exec <command>`` instead; note that any ``exec`` command fails unless all of the required containers are running. From now on, we will be using ``run``-style examples for general-case compatibility.
+
+
+Creating a Backup
+-----------------
 
 To create a backup, run::
 
-    docker-compose -f local.yml run --rm postgres backup
+    $ docker-compose -f local.yml run --rm postgres backup
+
+Assuming your project's database is named ``my_project`` here is what you will see: ::
+
+    Backing up the 'my_project' database...
+    SUCCESS: 'my_project' database backup 'backup_2018_03_13T09_05_07.sql.gz' has been created and placed in '/backups'.
+
+Keep in mind that ``/backups`` is the ``postgres`` container directory.
 
 
-To list backups, run::
+Viewing the Existing Backups
+----------------------------
 
-    docker-compose -f local.yml run --rm postgres list-backups
+To list existing backups, ::
+
+    $ docker-compose -f local.yml run --rm postgres backups
+
+These are the sample contents of ``/backups``: ::
+
+    These are the backups you have got:
+    total 24K
+    -rw-r--r-- 1 root root 5.2K Mar 13 09:05 backup_2018_03_13T09_05_07.sql.gz
+    -rw-r--r-- 1 root root 5.2K Mar 12 21:13 backup_2018_03_12T21_13_03.sql.gz
+    -rw-r--r-- 1 root root 5.2K Mar 12 21:12 backup_2018_03_12T21_12_58.sql.gz
 
 
-To restore a backup, run::
+Copying Backups Locally
+-----------------------
 
-    docker-compose -f local.yml run --rm postgres restore filename.sql
+If you want to copy backups from your ``postgres`` container locally, ``docker cp`` command_ will help you on that.
 
-Where <containerId> is the ID of the Postgres container. To get it, run::
+For example, given ``9c5c3f055843`` is the container ID copying all the backups over to a local directory is as simple as ::
 
-    docker ps
+    $ docker cp 9c5c3f055843:/backups ./backups
 
-To copy the files from the running Postgres container to the host system::
+With a single backup file copied to ``.`` that would be ::
 
-    docker cp <containerId>:/backups /host/path/target
+    $ docker cp 9c5c3f055843:/backups/backup_2018_03_13T09_05_07.sql.gz .
 
-Restoring From Backups
+.. _`command`: https://docs.docker.com/engine/reference/commandline/cp/
-======================
 
-To restore the production database to a local PostgreSQL database::
 
-    createdb NAME_OF_DATABASE
+Restoring from the Existing Backup
-    psql NAME_OF_DATABASE < NAME_OF_BACKUP_FILE
+----------------------------------
+
+To restore from one of the backups you have already got (take the ``backup_2018_03_13T09_05_07.sql.gz`` for example), ::
+
+    $ docker-compose -f local.yml run --rm postgres restore backup_2018_03_13T09_05_07.sql.gz
+
+You will see something like ::
+
+    Restoring the 'my_project' database from the '/backups/backup_2018_03_13T09_05_07.sql.gz' backup...
+    INFO: Dropping all connections to the database...
+     pg_terminate_backend
+    ----------------------
+    (0 rows)
+
+    INFO: Dropping the database...
+    INFO: Creating a new database...
+    INFO: Applying the backup to the new database...
+    SET
+    SET
+    SET
+    SET
+    SET
+     set_config
+    ------------
+
+    (1 row)
+
+    SET
+    # ...
+    ALTER TABLE
+    SUCCESS: The 'my_project' database has been restored from the '/backups/backup_2018_03_13T09_05_07.sql.gz' backup.

{{cookiecutter.project_slug}}/compose/production/postgres/Dockerfile

@@ -1,10 +1,6 @@
 FROM postgres:{{ cookiecutter.postgresql_version }}
 
-COPY ./compose/production/postgres/backup.sh /usr/local/bin/backup
+COPY ./compose/production/postgres/maintenance /usr/local/bin/maintenance
-RUN chmod +x /usr/local/bin/backup
+RUN chmod +x /usr/local/bin/maintenance/*
-
+RUN mv /usr/local/bin/maintenance/* /usr/local/bin \
-COPY ./compose/production/postgres/restore.sh /usr/local/bin/restore
+    && rmdir /usr/local/bin/maintenance
-RUN chmod +x /usr/local/bin/restore
-
-COPY ./compose/production/postgres/list-backups.sh /usr/local/bin/list-backups
-RUN chmod +x /usr/local/bin/list-backups

{{cookiecutter.project_slug}}/compose/production/postgres/backup.sh

@@ -1,25 +0,0 @@
-#!/usr/bin/env bash
-
-set -o errexit
-set -o pipefail
-set -o nounset
-
-
-# we might run into trouble when using the default `postgres` user, e.g. when dropping the postgres
-# database in restore.sh. Check that something else is used here
-if [ "$POSTGRES_USER" == "postgres" ]
-then
-    echo "creating a backup as the postgres user is not supported, make sure to set the POSTGRES_USER environment variable"
-    exit 1
-fi
-
-# export the postgres password so that subsequent commands don't ask for it
-export PGPASSWORD=$POSTGRES_PASSWORD
-
-echo "creating backup"
-echo "---------------"
-
-FILENAME=backup_$(date +'%Y_%m_%dT%H_%M_%S').sql.gz
-pg_dump -h postgres -U $POSTGRES_USER | gzip > /backups/$FILENAME
-
-echo "successfully created backup $FILENAME"

{{cookiecutter.project_slug}}/compose/production/postgres/list-backups.sh

@@ -1,10 +0,0 @@
-#!/usr/bin/env bash
-
-set -o errexit
-set -o pipefail
-set -o nounset
-
-
-echo "listing available backups"
-echo "-------------------------"
-ls /backups/

{{cookiecutter.project_slug}}/compose/production/postgres/maintenance/_sourced/constants.sh

@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+
+
+BACKUP_DIR_PATH='/backups'
+BACKUP_FILE_PREFIX='backup'

{{cookiecutter.project_slug}}/compose/production/postgres/maintenance/_sourced/countdown.sh

@@ -0,0 +1,12 @@
+#!/usr/bin/env bash
+
+
+countdown() {
+    declare desc="A simple countdown. Source: https://superuser.com/a/611582"
+    local seconds="${1}"
+    local d=$(($(date +%s) + "${seconds}"))
+    while [ "$d" -ge `date +%s` ]; do
+        echo -ne "$(date -u --date @$(($d - `date +%s`)) +%H:%M:%S)\r";
+        sleep 0.1
+    done
+}

{{cookiecutter.project_slug}}/compose/production/postgres/maintenance/_sourced/messages.sh

@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+
+
+message_newline() {
+    echo
+}
+
+message_debug()
+{
+    echo -e "DEBUG: ${@}"
+}
+
+message_welcome()
+{
+    echo -e "\e[1m${@}\e[0m"
+}
+
+message_warning()
+{
+    echo -e "\e[33mWARNING\e[0m: ${@}"
+}
+
+message_error()
+{
+    echo -e "\e[31mERROR\e[0m: ${@}"
+}
+
+message_info()
+{
+    echo -e "\e[37mINFO\e[0m: ${@}"
+}
+
+message_suggestion()
+{
+    echo -e "\e[33mSUGGESTION\e[0m: ${@}"
+}
+
+message_success()
+{
+    echo -e "\e[32mSUCCESS\e[0m: ${@}"
+}

{{cookiecutter.project_slug}}/compose/production/postgres/maintenance/_sourced/yes_no.sh

@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+
+
+yes_no() {
+    declare desc="Prompt for confirmation. \$\"\{1\}\": confirmation message."
+    local arg1="${1}"
+
+    local response=
+    read -r -p "${arg1} (y/[n])? " response
+    if [[ "${response}" =~ ^[Yy]$ ]]
+    then
+        exit 0
+    else
+        exit 1
+    fi
+}

{{cookiecutter.project_slug}}/compose/production/postgres/maintenance/backup

@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+
+
+### Create a database backup.
+###
+### Usage:
+###     $ docker-compose -f <environment>.yml (exec |run --rm) postgres backup
+
+
+set -o errexit
+set -o pipefail
+set -o nounset
+
+
+working_dir="$(dirname ${0})"
+source "${working_dir}/_sourced/constants.sh"
+source "${working_dir}/_sourced/messages.sh"
+
+
+message_welcome "Backing up the '${POSTGRES_DB}' database..."
+
+
+if [[ "${POSTGRES_USER}" == "postgres" ]]; then
+    message_error "Backing up as 'postgres' user is not supported. Assign 'POSTGRES_USER' env with another one and try again."
+    exit 1
+fi
+
+export PGPASSWORD="${POSTGRES_PASSWORD}"
+
+backup_filename="${BACKUP_FILE_PREFIX}_$(date +'%Y_%m_%dT%H_%M_%S').sql.gz"
+pg_dump \
+    --host=postgres \
+    --dbname="${POSTGRES_DB}" \
+    --username="${POSTGRES_USER}" \
+    | gzip > "${BACKUP_DIR_PATH}/${backup_filename}"
+
+
+message_success "'${POSTGRES_DB}' database backup '${backup_filename}' has been created and placed in '${BACKUP_DIR_PATH}'."

{{cookiecutter.project_slug}}/compose/production/postgres/maintenance/backups

@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+
+
+### View backups.
+###
+### Usage:
+###     $ docker-compose -f <environment>.yml (exec |run --rm) postgres backups
+
+
+set -o errexit
+set -o pipefail
+set -o nounset
+
+
+working_dir="$(dirname ${0})"
+source "${working_dir}/_sourced/constants.sh"
+source "${working_dir}/_sourced/messages.sh"
+
+
+message_welcome "These are the backups you have got:"
+
+ls -lht "${BACKUP_DIR_PATH}"

{{cookiecutter.project_slug}}/compose/production/postgres/maintenance/restore

@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+
+
+### Restore database from a backup.
+###
+### Parameters:
+###     <1> filename of an existing backup.
+###
+### Usage:
+###     $ docker-compose -f <environment>.yml (exec |run --rm) postgres restore <1>
+
+
+set -o errexit
+set -o pipefail
+set -o nounset
+
+
+working_dir="$(dirname ${0})"
+source "${working_dir}/_sourced/constants.sh"
+source "${working_dir}/_sourced/messages.sh"
+
+
+if [[ -z ${1+x} ]]; then
+    message_error "Backup filename is not specified yet it is a required parameter. Make sure you provide one and try again."
+    exit 1
+fi
+backup_filename="${BACKUP_DIR_PATH}/${1}"
+if [[ ! -f "${backup_filename}" ]]; then
+    message_error "No backup with the specified filename found. Check out the 'backups' maintenance script output to see if there is one and try again."
+    exit 1
+fi
+
+message_welcome "Restoring the '${POSTGRES_DB}' database from the '${backup_filename}' backup..."
+
+if [[ "${POSTGRES_USER}" == "postgres" ]]; then
+    message_error "Restoring as 'postgres' user is not supported. Assign 'POSTGRES_USER' env with another one and try again."
+    exit 1
+fi
+
+export PGPASSWORD="${POSTGRES_PASSWORD}"
+
+message_info "Dropping all connections to the database..."
+# Source: http://dba.stackexchange.com/a/11895
+drop_postgres_connections_sql='UPDATE pg_database'
+drop_postgres_connections_sql+=" SET datallowconn = 'false'"
+drop_postgres_connections_sql+=" WHERE datname = '${POSTGRES_DB}';"
+drop_postgres_connections_sql+='SELECT pg_terminate_backend(pid)'
+drop_postgres_connections_sql+=' FROM pg_stat_activity'
+drop_postgres_connections_sql+=" WHERE datname = '${POSTGRES_DB}';"
+psql \
+    --host=localhost \
+    --username=postgres \
+    --dbname=postgres \
+    --command="${drop_postgres_connections_sql}"
+
+message_info "Dropping the database..."
+dropdb \
+    --host=postgres \
+    --username="${POSTGRES_USER}" \
+    "${POSTGRES_DB}"
+
+message_info "Creating a new database..."
+createdb \
+    --host=postgres \
+    --username="${POSTGRES_USER}" \
+    --owner="${POSTGRES_USER}" \
+    "${POSTGRES_DB}"
+
+message_info "Applying the backup to the new database..."
+gunzip -c "${backup_filename}" \
+    | psql \
+        --host=postgres \
+        --username="${POSTGRES_USER}" \
+        "${POSTGRES_DB}"
+
+message_success "The '${POSTGRES_DB}' database has been restored from the '${backup_filename}' backup."

{{cookiecutter.project_slug}}/compose/production/postgres/restore.sh

@@ -1,58 +0,0 @@
-#!/usr/bin/env bash
-
-set -o errexit
-set -o pipefail
-set -o nounset
-
-
-# we might run into trouble when using the default `postgres` user, e.g. when dropping the postgres
-# database in restore.sh. Check that something else is used here
-if [ "$POSTGRES_USER" == "postgres" ]
-then
-    echo "restoring as the postgres user is not supported, make sure to set the POSTGRES_USER environment variable"
-    exit 1
-fi
-
-# export the postgres password so that subsequent commands don't ask for it
-export PGPASSWORD=$POSTGRES_PASSWORD
-
-# check that we have an argument for a filename candidate
-if [[ $# -eq 0 ]] ; then
-    echo 'usage:'
-    echo '    docker-compose -f production.yml run postgres restore <backup-file>'
-    echo ''
-    echo 'to get a list of available backups, run:'
-    echo '    docker-compose -f production.yml run postgres list-backups'
-    exit 1
-fi
-
-# set the backupfile variable
-BACKUPFILE=/backups/$1
-
-# check that the file exists
-if ! [ -f $BACKUPFILE ]; then
-    echo "backup file not found"
-    echo 'to get a list of available backups, run:'
-    echo '    docker-compose -f production.yml run postgres list-backups'
-    exit 1
-fi
-
-echo "beginning restore from $1"
-echo "-------------------------"
-
-# delete the db
-# deleting the db can fail. Spit out a comment if this happens but continue since the db
-# is created in the next step
-echo "deleting old database $POSTGRES_USER"
-if dropdb -h postgres -U $POSTGRES_USER $POSTGRES_USER
-then echo "deleted $POSTGRES_USER database"
-else echo "database $POSTGRES_USER does not exist, continue"
-fi
-
-# create a new database
-echo "creating new database $POSTGRES_USER"
-createdb -h postgres -U $POSTGRES_USER $POSTGRES_USER -O $POSTGRES_USER
-
-# restore the database
-echo "restoring database $POSTGRES_USER"
-gunzip -c $BACKUPFILE | psql -h postgres -U $POSTGRES_USER