mirror of
https://github.com/Security-Onion-Solutions/securityonion.git
synced 2026-11-18 18:16:02 +01:00
FIX: retry the push orchestration's refresh_pillar step
Holding dispatch in the drainer cannot help an orchestration that is already running when salt-master restarts: its first action raises, and the next action's refresh_pillar is published seconds after the restart, before minions have rejoined the publisher, so it fails with "No minions responded". refresh_pillar now retries (3 attempts, 10s apart; a failed attempt takes about 15s to time out, so the later attempts land after the reconnect window). It also runs ahead of highstate actions, which now require it, so they get the same reachability check. Retrying the highstate step itself is not an option because it would re-run a fleetwide highstate whenever any state failed. Run straight after a salt-master restart on a standalone 3.5.0 install, the orchestration previously failed with "No minions responded"; it now succeeds on the second attempt. A target that is genuinely offline takes about 65s to fail instead of 15s.
This commit is contained in:
1 file changed
+13
-6
@@ -4,8 +4,19 @@
|
||||
{% set BATCH_WAIT = AUTOAPPLY.batch_wait %}
|
||||
|
||||
{# queue must be a top-level salt.state arg (kwarg is ignored); an int is max_queue and still fails on conflict #}
|
||||
{# refresh_pillar doubles as the reachability check: minions take up to ~22s to rejoin a restarted master's
|
||||
publisher, and a job published before then reaches nobody. Each failed attempt takes ~15s to time out. #}
|
||||
|
||||
{% for action in actions %}
|
||||
refresh_pillar_{{ loop.index }}:
|
||||
salt.function:
|
||||
- name: saltutil.refresh_pillar
|
||||
- tgt: '{{ action.tgt }}'
|
||||
- tgt_type: {{ action.get('tgt_type', 'compound') }}
|
||||
- retry:
|
||||
attempts: 3
|
||||
interval: 10
|
||||
|
||||
{% if action.get('highstate') %}
|
||||
apply_highstate_{{ loop.index }}:
|
||||
salt.state:
|
||||
@@ -15,13 +26,9 @@ apply_highstate_{{ loop.index }}:
|
||||
- batch: {{ action.get('batch', BATCH) }}
|
||||
- batch_wait: {{ action.get('batch_wait', BATCH_WAIT) }}
|
||||
- queue: True
|
||||
- require:
|
||||
- salt: refresh_pillar_{{ loop.index }}
|
||||
{% else %}
|
||||
refresh_pillar_{{ loop.index }}:
|
||||
salt.function:
|
||||
- name: saltutil.refresh_pillar
|
||||
- tgt: '{{ action.tgt }}'
|
||||
- tgt_type: {{ action.get('tgt_type', 'compound') }}
|
||||
|
||||
apply_{{ action.state | replace('.', '_') }}_{{ loop.index }}:
|
||||
salt.state:
|
||||
- tgt: '{{ action.tgt }}'
|
||||
|
||||
Reference in new issue
Block a user