Two refinements to the --salt-upgraded path, surfaced testing a salt downgrade+soup
on a manager+heavynode grid:
- The post-upgrade settle loop settled on any stable reachable count >0, so when a
target was briefly down for its salt-minion restart it could settle on the
not-yet-restarted subset (observed: 'settled at 1' with 2 accepted) and release the
tiered pass before nodes reconnected. Capture the reachable count just before the
pass and wait for it to recover to that count (up >= pre-upgrade target) and hold
steady, with an initial grace so the delayed restart dip is observed rather than
skipped. Still compares against the pre-upgrade reachable set, not accepted keys, so
an intentionally powered-off node never stalls past the backstop.
- The salt-upgrade pass returns non-zero by design (targets restart salt-minion
mid-run), but it logged the generic 'nodes it missed will converge on the scheduled
highstate' warning, which reads like a real failure. Mark that dispatch as
expect_restart so it logs a benign, explanatory line instead.
Verified live: with the heavynode's salt-minion bounced during the settle window, the
loop logged 'fleet recovered to 2 minions up (>= pre-upgrade 2)' and only then ran the
tiered pass (heavynode highstate 427 succeeded, 0 failed).
Josh Patterson
6abf382ea8