pg_dumpall dumped every database, and so_telegraf dominated the result: it
is the only database that grows with grid size and metric volume, while
everything else in the cluster is small and mostly static.
Nothing is lost by skipping it. The data is transient metrics on a 14-day
retention window, roles are globals so the per-minion telegraf logins are
still dumped, and the database itself is rebuilt after a restore without
operator action -- init-db.sh recreates it (run on every highstate by
postgres_bootstrap_soc_db, not just on a fresh volume), telegraf_users.sls
re-provisions the roles and schema, and Telegraf recreates its tables on
first write.
The dump pipeline returned gzip's exit status, so a pg_dumpall that
died mid-stream still produced a valid .gz holding a truncated dump,
written straight to the final filename. The idempotency check then
blocked retries for the day and the corrupt file counted toward
retention, evicting a good backup each day until none remained.
- set -o pipefail so a failed pg_dumpall fails the pipeline
- dump to a .tmp file and atomically rename only after success, so
the final filename appears only for a complete backup
- gzip -t integrity check before publishing
- trap-based cleanup of the temp file; sweep stale temps at startup
- run retention only after a successful backup, with a glob
restricted to finished backups
- log timestamped OK/ERROR outcomes to /opt/so/log/postgres/backup.log
The so-postgres-backup script and its cron were living under
salt/backup/config_backup.sls, which meant the backup script and cron
were deployed independently of whether postgres was enabled/disabled.
- Relocate salt/backup/tools/sbin/so-postgres-backup to
salt/postgres/tools/sbin/so-postgres-backup so the existing
postgres_sbin file.recurse in postgres/config.sls picks it up with
everything else — no separate file.managed needed.
- Remove postgres_backup_script and so_postgres_backup from
salt/backup/config_backup.sls.
- Add cron.present for so_postgres_backup to salt/postgres/enabled.sls
and the matching cron.absent to salt/postgres/disabled.sls so the
cron follows the container's lifecycle.
Mike Reeves