FIX: retry the push orchestration's refresh_pillar step

Holding dispatch in the drainer cannot help an orchestration that is
already running when salt-master restarts: its first action raises, and
the next action's refresh_pillar is published seconds after the restart,
before minions have rejoined the publisher, so it fails with "No minions
responded".

refresh_pillar now retries (3 attempts, 10s apart; a failed attempt takes
about 15s to time out, so the later attempts land after the reconnect
window). It also runs ahead of highstate actions, which now require it,
so they get the same reachability check. Retrying the highstate step
itself is not an option because it would re-run a fleetwide highstate
whenever any state failed.

Run straight after a salt-master restart on a standalone 3.5.0 install,
the orchestration previously failed with "No minions responded"; it now
succeeds on the second attempt. A target that is genuinely offline takes
about 65s to fail instead of 15s.
This commit is contained in:
Josh Patterson committed 2026-11-18 10:27:34 -05:00
1 parent 65816ae717
commit aba2f5ea43
1 file changed
+13 -6
+13 -6
View File
@@ -4,8 +4,19 @@
{% set BATCH_WAIT = AUTOAPPLY.batch_wait %}
{# queue must be a top-level salt.state arg (kwarg is ignored); an int is max_queue and still fails on conflict #}
{# refresh_pillar doubles as the reachability check: minions take up to ~22s to rejoin a restarted master's
publisher, and a job published before then reaches nobody. Each failed attempt takes ~15s to time out. #}
{% for action in actions %}
refresh_pillar_{{ loop.index }}:
salt.function:
- name: saltutil.refresh_pillar
- tgt: '{{ action.tgt }}'
- tgt_type: {{ action.get('tgt_type', 'compound') }}
- retry:
attempts: 3
interval: 10
{% if action.get('highstate') %}
apply_highstate_{{ loop.index }}:
salt.state:
@@ -15,13 +26,9 @@ apply_highstate_{{ loop.index }}:
- batch: {{ action.get('batch', BATCH) }}
- batch_wait: {{ action.get('batch_wait', BATCH_WAIT) }}
- queue: True
- require:
- salt: refresh_pillar_{{ loop.index }}
{% else %}
refresh_pillar_{{ loop.index }}:
salt.function:
- name: saltutil.refresh_pillar
- tgt: '{{ action.tgt }}'
- tgt_type: {{ action.get('tgt_type', 'compound') }}
apply_{{ action.state | replace('.', '_') }}_{{ loop.index }}:
salt.state:
- tgt: '{{ action.tgt }}'