Retention drops every child once they all age out, partman refuses to
drop the last one, and with infinite_time_partitions off it will not
premake forward from a child that far in the past. The set is left with
one stale partition and no current one, so metrics land right back in
the default.
Set infinite_time_partitions on telegraf parents, in both the repair
script and the retention subcommand, and stop blaming the launcher when
pg_cron is not loaded at all.
so-telegraf-partition-repair cleared the backlog but left the cause in place:
pg_cron's launcher is still dead, so the grid re-stalls as soon as it walks off
the premade window. Operators on the preview release need something they can run
once, before they soup, that leaves Telegraf collecting again.
Replace it with so-telegraf-repair, which fixes both halves. The running release
already creates the pg_cron extension and registers telegraf-partman-maintenance
in so_telegraf; only the launcher is missing, because so_telegraf did not exist
when the postmaster started. Restarting so-postgres is therefore enough to get
the existing job firing, so this touches no configuration and duplicates none of
the postgres state's SQL -- group_role still migrates the job to the postgres
database on the next soup. It also reconciles premake to 7 and prefers
so_admin.telegraf_maintenance() when that state has already landed.
Exit status separates healthy (0) from needs-repair (1) from does-not-apply (2),
which is what soup now gates on. postupgrade_changes runs after the highstate,
so the database is already converted by then and the backlog is the only thing
left to detect. Truncating is destructive and most grids were never affected --
fresh installs in particular, since they have no Telegraf history at all -- so
soup asks first and skips silently rather than clearing defaults on every host.
Truncating default partitions is destructive and premaking them is pg_cron's
job, so neither belongs in a state that runs on every checkin. Replace the two
telegraf_users states with so-telegraf-partition-repair, a standalone tool that
reports partition health and clears the backlog, and call it once from soup.
The script depends only on pg_partman, so it also runs against a grid that has
not yet picked up the new postgres state. It no-ops when nothing is stranded,
refuses to discard rows non-interactively without --yes, and reports when the
pg_cron job has never fired, which is the underlying cause rather than a
symptom the truncate addresses.
Hourly self-healing stays with so_admin.telegraf_maintenance() via pg_cron, so
a grid that never soups still recovers, just gradually and without discarding
in-retention metrics.
pg_cron's launcher connects to cron.database_name at postmaster start and is
registered BGW_NEVER_RESTART. On a host upgraded onto an existing /nsm/postgres
volume, init-db.sh never runs, so so_telegraf does not exist when PostgreSQL
starts -- the launcher dies and never retries. Salt then creates the database,
the extension, and the schedule, all of which succeed, but no worker is left to
fire the job. partman.run_maintenance_proc() therefore never runs: partitions
stop being premade after create_parent's initial window and every metric lands
in <parent>_default. Retention never fires either.
That state is self-perpetuating. Once the default partition holds rows for a day
with no child, PostgreSQL cannot create that child at all -- attaching it would
violate the default partition's constraint -- so maintenance aborts on the first
parent it reaches. Fixing the scheduler alone does not recover a stalled grid.
Point cron.database_name at the always-present postgres database and register the
job with cron.schedule_in_database targeting so_telegraf, so the launcher no
longer depends on database creation order. group_role drops any registration left
behind in so_telegraf, and both halves are guarded on the live GUC so applying
postgres.telegraf_users before the postgresql.conf change has restarted the
container skips instead of failing.
Maintenance now runs so_admin.telegraf_maintenance(), which drains stranded rows
out of any default partition before calling partman: expired rows are deleted,
the rest are repartitioned. It runs from the state on every highstate as well as
hourly from pg_cron, so a grid whose worker is dead still recovers on its own.
The routines live in a postgres-owned schema so_telegraf has no rights on, since
pg_cron executes them as postgres.
Existing grids are recovered by a marker-guarded repair state that truncates the
non-empty defaults once per host. The backlog is mostly past retention already
and moving tens of GB just to delete most of it is not worth the WAL.
Also raise premake from 3 to 7, reconciled onto existing parents in the retention
subcommand, so an outage has a week of headroom before anything reaches a
default partition, and add a check subcommand reporting partition age, default
occupancy and last job status.
PostgreSQL's log_min_error_statement defaults to 'error', so whenever a
CREATE/ALTER ROLE ... PASSWORD statement errored, the full statement text --
including the plaintext password -- was written to /opt/so/log/postgres/postgres.log.
The role-provisioning paths (init-db.sh for so_postgres, so-telegraf-postgres
for per-minion telegraf roles) both dispatch such DDL, the latter on every
state.apply.
- init-db.sh / so-telegraf-postgres: SET log_min_error_statement = panic before
the password-bearing DDL so an error no longer emits the STATEMENT line. The
ERROR message itself (no password) still logs, preserving debuggability.
- logrotate: add a postgres stanza (daily, keep 14, copytruncate, compress) so
postgres.log is rotated like every other service and leaked content can't
persist indefinitely. copytruncate is required because the container holds the
log open via redirected stderr.
- soup: scrub any already-logged PASSWORD lines from postgres.log during
post_to_3.2.0, rewriting in place to preserve the inode postgres is writing to.
psql does not substitute :var references inside dollar-quoted strings,
so the DO blocks in the user and retention subcommands were receiving
literal colons and failing (silently for user, via hide_output: True).
Rewrite the conditional CREATE/ALTER ROLE with SELECT format(...) \\gexec
and guard the retention UPDATE with \\gset + \\if.
Mike Reeves