pg_cron's launcher connects to cron.database_name at postmaster start and is
registered BGW_NEVER_RESTART. On a host upgraded onto an existing /nsm/postgres
volume, init-db.sh never runs, so so_telegraf does not exist when PostgreSQL
starts -- the launcher dies and never retries. Salt then creates the database,
the extension, and the schedule, all of which succeed, but no worker is left to
fire the job. partman.run_maintenance_proc() therefore never runs: partitions
stop being premade after create_parent's initial window and every metric lands
in <parent>_default. Retention never fires either.
That state is self-perpetuating. Once the default partition holds rows for a day
with no child, PostgreSQL cannot create that child at all -- attaching it would
violate the default partition's constraint -- so maintenance aborts on the first
parent it reaches. Fixing the scheduler alone does not recover a stalled grid.
Point cron.database_name at the always-present postgres database and register the
job with cron.schedule_in_database targeting so_telegraf, so the launcher no
longer depends on database creation order. group_role drops any registration left
behind in so_telegraf, and both halves are guarded on the live GUC so applying
postgres.telegraf_users before the postgresql.conf change has restarted the
container skips instead of failing.
Maintenance now runs so_admin.telegraf_maintenance(), which drains stranded rows
out of any default partition before calling partman: expired rows are deleted,
the rest are repartitioned. It runs from the state on every highstate as well as
hourly from pg_cron, so a grid whose worker is dead still recovers on its own.
The routines live in a postgres-owned schema so_telegraf has no rights on, since
pg_cron executes them as postgres.
Existing grids are recovered by a marker-guarded repair state that truncates the
non-empty defaults once per host. The backlog is mostly past retention already
and moving tens of GB just to delete most of it is not worth the WAL.
Also raise premake from 3 to 7, reconciled onto existing parents in the retention
subcommand, so an outage has a week of headroom before anything reaches a
default partition, and add a check subcommand reporting partition age, default
occupancy and last job status.
Two defects surfaced testing BRANCH=asasoup soup on a manager+heavynode grid:
1. The tiered-pass actions JSON was built from a multi-line heredoc. salt parses
'pillar=<value>' kwargs with a non-DOTALL regex, so the embedded newlines made
salt-run treat the whole token as a positional saltenv -- 'No matching salt
environment for environment pillar={...}' -- and the highstate never ran, leaving
the heavynode on the old version. Emit the actions JSON on a single line (matching
how so-push-drainer's json.dumps payload already works).
2. orch.push_batch's salt.state step reports 'No minions returned' (a failure) for a
tier whose compound target matches nothing, so any grid lacking a role (no receiver,
small grids) always logged a warning and returned rc=1 even when every present node
converged. Pre-check each tier with 'salt -C <tgt> --preview-target' and include only
tiers that match >=1 minion; exit cleanly if none match.
Verified live: heavynode highstated 3.2.0 -> 3.3.0 (427 states, 0 failed) and a
follow-up run skips the empty receiver/remainder tiers with rc=0.
The per-minion highstate schedule moved from 15 minutes to 120 minutes
(salt:schedule:highstate_interval_minutes), so after soup upgrades the manager,
non-manager minions could otherwise sit on the old version for up to ~2.5 hours
(interval + splay) before their scheduled highstate applies the new code.
Add so-grid-highstate, a detached best-effort driver soup fires at the end of an
upgrade. It uses the existing orch.push_batch runner to highstate the grid in
role tiers (searchnodes/heavynodes -> receivers -> the rest), skips single-node
grids, and when Salt itself was upgraded first runs an untiered pass and waits
for minions to reconnect on the new salt-minion before the tiered pass.
soup gains a push_grid_highstate() helper (guarded, launched via setsid nohup so
an SSH drop can't kill it) called on both the hotfix and full-upgrade paths,
wires the previously-dead -b flag through as --batch (now accepting N or N%),
and updates the distributed-deployment message to reflect the active push.
salt.schedule.highstate_interval_hours could not express a sub-hour
cadence, so an operator who disables salt.auto_apply had no way back to
the legacy 15-minute highstate. Rename the setting to
highstate_interval_minutes (default 120, behavior unchanged) and enforce
a 15-minute floor in SOC.
Non-manager splay is now a quarter of the interval clamped to [5, 30]
minutes, so a short interval no longer gets jitter larger than itself; at
the 120-minute default it stays 1800s. The so-salt-minion-check restart
threshold keeps its interval-plus-one-hour grace, now in minute math.
Previously salt/vars/elasticsearch.map.jinja read the elasticsearch heap
size via a raw pillar access (INIT.PILLAR.elasticsearch.esheap). If the
esheap key was missing from a minion pillar, building GLOBALS raised
'dict object has no attribute esheap', cascading into every state that
imports GLOBALS (elasticsearch.enabled, logstash.config, telegraf.config,
etc.) and blocking highstate entirely.
Add an esheap default ('600m', matching the es_heapsize() floor) to
elasticsearch/defaults.yaml and fall back to it via dict .get() so a
missing pillar value degrades gracefully. The per-node pillar value still
wins when present.
Mike Reeves
Jason Ertel
Jason Ertel
Josh Patterson
Mike Reeves
coreyogburn
Corey Ogburn
Josh Brower
Jorge Reyes