Postgres: run pg_partman maintenance over the local socket and detect failed runs

Merge pull request #16378 from Security-Onion-Solutions/reyesj2/fix-16369
surface ilm policy errors in so-elasticsearch-ilm-policy-load output
2026-11-10 22:50:04 +01:00 · 2026-11-10 11:38:23 -05:00 · 2026-11-05 14:05:29 -06:00 · 2026-11-05 11:11:45 -06:00 · 2026-11-04 15:56:12 -05:00 · 2026-11-04 15:51:16 -05:00
59 changed files with 695 additions and 274 deletions
@@ -14,6 +14,7 @@ body:
        - 3.2.0
        - 3.3.0
        - 3.4.0
+        - 3.5.0
        - Other (please provide detail below)
    validations:
      required: true
@@ -1,17 +1,17 @@
-### 3.3.0-20260911 ISO image released on 2026/09/11
+### 3.4.0-20261102 ISO image released on 2026/11/02


 ### Download and Verify

-3.3.0-20260911 ISO image:  
-https://download.securityonion.net/file/securityonion/securityonion-3.3.0-20260911.iso
+3.4.0-20261102 ISO image:  
+https://download.securityonion.net/file/securityonion/securityonion-3.4.0-20261102.iso
 
-MD5: 12B18433D3A2198A185892FF79CF638F  
-SHA1: 2B3C2E1FA7A78ED1F956E7EDCC12E32593C14EEE  
-SHA256: 0938C73B76CE30EC9E4394D312C79EA7CAC721B6818541697279A6221F7D870D  
+MD5: FB0380AE996E175D6CC7E7A35D8CC178  
+SHA1: D8960F1DA352710BCA209773BF0B3AE8791D0D1B  
+SHA256: DFC4C87668836852A328EC7A0B755C8C1511FF0E43D423DEE23047A95F676D99  

 Signature for ISO image:  
-https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.3.0-20260911.iso.sig
+https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.4.0-20261102.iso.sig

 Signing key:  
 https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/3/main/KEYS  
@@ -25,22 +25,22 @@ wget https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/3/

 Download the signature file for the ISO:  
 ```
-wget https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.3.0-20260911.iso.sig
+wget https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.4.0-20261102.iso.sig
 ```

 Download the ISO image:  
 ```
-wget https://download.securityonion.net/file/securityonion/securityonion-3.3.0-20260911.iso
+wget https://download.securityonion.net/file/securityonion/securityonion-3.4.0-20261102.iso
 ```

 Verify the downloaded ISO image using the signature file:  
 ```
-gpg --verify securityonion-3.3.0-20260911.iso.sig securityonion-3.3.0-20260911.iso
+gpg --verify securityonion-3.4.0-20261102.iso.sig securityonion-3.4.0-20261102.iso
 ```

 The output should show "Good signature" and the Primary key fingerprint should match what's shown below:
 ```
-gpg: Signature made Fri 11 Sep 2026 11:23:56 AM EDT using RSA key ID FE507013
+gpg: Signature made Sat 31 Oct 2026 09:27:50 AM EDT using RSA key ID FE507013
 gpg: Good signature from "Security Onion Solutions, LLC <info@securityonionsolutions.com>"
 gpg: WARNING: This key is not certified with a trusted signature!
 gpg:          There is no indication that the signature belongs to the owner.
@@ -1 +1 @@
-3.4.0
+3.5.0
@@ -0,0 +1,14 @@
+-----BEGIN CERTIFICATE-----
+MIICKzCCAbGgAwIBAgIKe3G2gla4EnycqDAKBggqhkjOPQQDAzBaMQswCQYDVQQG
+EwJVUzETMBEGA1UECxMKZW1TaWduIFBLSTEUMBIGA1UEChMLZU11ZGhyYSBJbmMx
+IDAeBgNVBAMTF2VtU2lnbiBFQ0MgUm9vdCBDQSAtIEMzMB4XDTE4MDIxODE4MzAw
+MFoXDTQzMDIxODE4MzAwMFowWjELMAkGA1UEBhMCVVMxEzARBgNVBAsTCmVtU2ln
+biBQS0kxFDASBgNVBAoTC2VNdWRocmEgSW5jMSAwHgYDVQQDExdlbVNpZ24gRUND
+IFJvb3QgQ0EgLSBDMzB2MBAGByqGSM49AgEGBSuBBAAiA2IABP2lYa57JhAd6bci
+MK4G9IGzsUJxlTm801Ljr6/58pc1kjZGDoeVjbk5Wum739D+yAdBPLtVb4Ojavti
+sIGJAnB9SMVK4+kiVCJNk7tCDK93nCOmfddhEc5lx/h//vXyqaNCMEAwHQYDVR0O
+BBYEFPtaSNCAIEDyqOkAB2kZd6fmw/TPMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMB
+Af8EBTADAQH/MAoGCCqGSM49BAMDA2gAMGUCMQC02C8Cif22TGK6Q04ThHK1rt0c
+3ta13FaPWEBaLd4gTCKDypOofu4SQMfWh0/434UCMBwUZOR8loMRnLDRWmFLpg9J
+0wD8ofzkpf9/rdcw0Md3f76BB1UwUCAU9Vc4CqgxUQ==
+-----END CERTIFICATE-----
@@ -0,0 +1,15 @@
+-----BEGIN CERTIFICATE-----
+MIICTjCCAdOgAwIBAgIKPPYHqWhwDtqLhDAKBggqhkjOPQQDAzBrMQswCQYDVQQG
+EwJJTjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNo
+bm9sb2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0g
+RzMwHhcNMTgwMjE4MTgzMDAwWhcNNDMwMjE4MTgzMDAwWjBrMQswCQYDVQQGEwJJ
+TjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNobm9s
+b2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0gRzMw
+djAQBgcqhkjOPQIBBgUrgQQAIgNiAAQjpQy4LRL1KPOxst3iAhKAnjlfSU2fySU0
+WXTsuwYc58Byr+iuL+FBVIcUqEqy6HyC5ltqtdyzdc6LBtCGI79G1Y4PPwT01xyS
+fvalY8L1X44uT6EYGQIrMgqCZH0Wk9GjQjBAMB0GA1UdDgQWBBR8XQKEE9TMipuB
+zhccLikenEhjQjAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAKBggq
+hkjOPQQDAwNpADBmAjEAvvNhzwIQHWSVB7gYboiFBS+DCBeQyh+KTOgNG3qxrdWB
+CUfvO6wIBHxcmbHtRwfSAjEAnbpV/KlK6O3t5nYBQnvI+GDZjVGLVTv7jHvrZQnD
+JbNR6iC8hZVdyR+EhCVBCyj
+-----END CERTIFICATE-----
@@ -0,0 +1,21 @@
+-----BEGIN CERTIFICATE-----
+MIIDczCCAlugAwIBAgILAK7PALrEzzL4Q7IwDQYJKoZIhvcNAQELBQAwVjELMAkG
+A1UEBhMCVVMxEzARBgNVBAsTCmVtU2lnbiBQS0kxFDASBgNVBAoTC2VNdWRocmEg
+SW5jMRwwGgYDVQQDExNlbVNpZ24gUm9vdCBDQSAtIEMxMB4XDTE4MDIxODE4MzAw
+MFoXDTQzMDIxODE4MzAwMFowVjELMAkGA1UEBhMCVVMxEzARBgNVBAsTCmVtU2ln
+biBQS0kxFDASBgNVBAoTC2VNdWRocmEgSW5jMRwwGgYDVQQDExNlbVNpZ24gUm9v
+dCBDQSAtIEMxMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAz+upufGZ
+BczYKCFK83M0UYRWEPWgTywS4/oTmifQz/l5GnRfHXk5/Fv4cI7gklL35CX5VIPZ
+HdPIWoU/Xse2B+4+wM6ar6xWQio5JXDWv7V7Nq2s9nPczdcdioOl+yuQFTdrHCZH
+3DspVpNqs8FqOp099cGXOFgFixwR4+S0uF2FHYP+eF8LRWgYSKVGczQ7/g/IdrvH
+GPMF0Ybzhe3nudkyrVWIzqa2kbBPrH4VI5b2P/AgNBbeCsbEBEV5f6f9vtKppa+c
+xSMq9zwhbL2vj07FOrLzNBL834AaSaTUqZX3noleoomslMuoaJuvimUnzYnu3Yy1
+aylwQ6BpC+S5DwIDAQABo0IwQDAdBgNVHQ4EFgQU/qHgcB4qAzlSWkK+XJGFehiq
+TbUwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQEL
+BQADggEBAMJKVvoVIXsoounlHfv4LcQ5lkFMOycsxGwYFYDGrK9HWS8mC+M2sO87
+/kOXSTKZEhVb3xEp/6tT+LvBeA+snFOvV71ojD1pM/CjoCNjO2RnIkSt1XHLVip4
+kqNPEjE2NuLe/gDEo2APJ62gsIq1NnpSob0n9CAnYuhNlCQT5AoE6TyrLshDCUrG
+YQTlSTR+08TI9Q/Aqum6VF7zYytPT1DU/rl7mYw9wC68AivTxEDkigcxHpvOJpkT
+xHqmiIMERnHXhuBUDDIlhJu58tBf5E7oke3VIAb3ADMmpDqw8NQBmIMMMAVSKeo
+WXzhriKi4gp6D/piq1JM4fHfyr6DDUI=
+-----END CERTIFICATE-----
@@ -0,0 +1,32 @@
+-----BEGIN CERTIFICATE-----
+MIIFcjCCA1qgAwIBAgIKLwq3aw3LSq8nWDANBgkqhkiG9w0BAQwFADBWMQswCQYD
+VQQGEwJVUzETMBEGA1UECxMKZW1TaWduIFBLSTEUMBIGA1UEChMLZU11ZGhyYSBJ
+bmMxHDAaBgNVBAMTE2VtU2lnbiBSb290IENBIC0gQzIwHhcNMTgwMjE4MTgzMDAw
+WhcNNDMwMjE4MTgzMDAwWjBWMQswCQYDVQQGEwJVUzETMBEGA1UECxMKZW1TaWdu
+IFBLSTEUMBIGA1UEChMLZU11ZGhyYSBJbmMxHDAaBgNVBAMTE2VtU2lnbiBSb290
+IENBIC0gQzIwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCMfX1lA+Tb
+mh9YInmRgOW97IVx4LUJf2DRZfs837Jrml+py64aVnYgWO4t6C78fgjfS7jX+c4T
+inIzEquWcI+zi0fd4Sc8NDf7JONp27VWX0qwUYqzLDRCt+s7zpLcfx1ky0zVIJj6
+L06uPyK3kIr9+YAsrVj+39utm6e2MBQsRNstSI3fCQYAGvoQTQ8fULauTqNWaYAk
+NYFe6HUHHQPp2u1Ua00odMXiD5oRFxLcDnGAcE1I/9E9mLCdkggXijYUmico7+Xw
+ZeFoPhva6eIJ5p03Lt3Du5W3EcHR0cJmmY1pyeA36JaXKWRNM9IRjYMVNCcp4jhB
+2tIYiZ+LVk8bwQ9/1c23txmv3u97taZlV22NF4ttS1qq3J+MOp0oGULBzpKfRx0q
+GVqbPukQNGAjOLIN8KDNQNzbR1iAl2d8H+MSoicBo4Aid8TjLWcNv48oCWL53ZrF
+BMTDjaIA6frG1t4IpbnHadA7qCJJe2qpJN6n2eQKAUn6UiQDHPsSqNBlcUhQ4Y/0
+Y0mU5rghm2OB9rXQS1Fb1JRCfJMNnJIm5AUB2+2RWzq5Tgz7SbSho8NsZk0UbQnF
+xciqQ9uoVTAsK14Sk9oG8Q3zfsM08cdPoRb0WlIZklR6mKD7L8nH/zfGu8PIJv94
+GGB9RZ9U4A69r3ePmy8MvrzfNxHKtH6svwIDAQABo0IwQDAdBgNVHQ4EFgQUs/eK
+pNYPiABZ6FEXT9V+7IYigZ0wDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB
+Af8wDQYJKoZIhvcNAQEMBQADggIBADQlpiWM0cv2nZ0H5jVsBq0x2q62Q0LwqATs
+CFvyub7gxNCytRuoA8stmPOEu/lg8Igxj4FIjoyhIrWUVxyiLU7No4P+WjEUOwUT
+xIpkEOtvGUQ9fiOlcGHtIZDNBlZq7WpktXAxeV55RPPsor26p2FNAMRFfZQh0sLX
+hKgk8iulSSggqx8ezgPye63FaiYEi4c/dzRj3HOCnsZiwZZU02df5YpNFjxSwZvE
+41cjGpsrpWMfQFI2s53RbeXp47lSAxYE4NzjBFMe+EwFuEveBCJBEAH5rvYu3pi2
+orsJ424TqWEQV1tCsCkQz+Yq/Okal7yHAkKDeOXcP7oN4A+TdXc2pdqxuVCnBO0R
+mWz2JpGSSeJjiTk/OPwRsPNWtwG/KXL04o2ta3jiPpJuICVtWDAc9R3auBEgJl5r
+ShRmBdszG0LmzsHuZPCFSYC15RBDCOBsa8bDRJ8pBFU2Wi/CVXCACEuavgoveA4F
+a5bt38o0PWxsBP+MpocCdVtDMqzQhxy9IohKuXWAGresoIvKDg3xFk6rBOrjfVwJ
+elwi/xAisojHPJVQv9W1zVIoHp+EQg/4MQC21NbIX2RoioB+V3hK439b/w7deU8x
+2M8cl1OG0nPfbnARl5GPM7vJgi470jto4SeMg6HMAW3Egb56tQcNLwI9U8mZnNvR
+gUMrkAgL
+-----END CERTIFICATE-----
@@ -0,0 +1,22 @@
+-----BEGIN CERTIFICATE-----
+MIIDlDCCAnygAwIBAgIKMfXkYgxsWO3W2DANBgkqhkiG9w0BAQsFADBnMQswCQYD
+VQQGEwJJTjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBU
+ZWNobm9sb2dpZXMgTGltaXRlZDEcMBoGA1UEAxMTZW1TaWduIFJvb3QgQ0EgLSBH
+MTAeFw0xODAyMTgxODMwMDBaFw00MzAyMTgxODMwMDBaMGcxCzAJBgNVBAYTAklO
+MRMwEQYDVQQLEwplbVNpZ24gUEtJMSUwIwYDVQQKExxlTXVkaHJhIFRlY2hub2xv
+Z2llcyBMaW1pdGVkMRwwGgYDVQQDExNlbVNpZ24gUm9vdCBDQSAtIEcxMIIBIjAN
+BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAk0u76WaK7p1b1TST0Bsew+eeuGQz
+f2N4aLTNLnF115sgxk0pvLZoYIr3IZpWNVrzdr3YzZr/k1ZLpVkGoZM0Kd0WNHVO
+8oG0x5ZOrRkVUkr+PHB1cM2vK6sVmjM8qrOLqs1D/fXqcP/tzxE7lM5OMhbTI0Aq
+d7OvPAEsbO2ZLIvZTmmYsvePQbAyeGHWDV/D+qJAkh1cF+ZwPjXnorfCYuKrpDhM
+tTk1b+oDafo6VGiFbdbyL0NVHpENDtjVaqSW0RM8LHhQ6DqS0hdW5TUaQBw+jSzt
+Od9C4INBdN+jzcKGYEho42kLVACL5HZpIQ15TjQIXhTCzLG3rdd8cIrHhQIDAQAB
+o0IwQDAdBgNVHQ4EFgQU++8Nhp6w492pufEhF38+/PB3KxowDgYDVR0PAQH/BAQD
+AgEGMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAFn/8oz1h31x
+PaOfG1vR2vjTnGs2vZupYeveFix0PZ7mddrXuqe8QhfnPZHr5X3dPpzxz5KsbEjM
+wiI/aTvFthUvozXGaCocV685743QNcMYDHsAVhzNixl03r4PEuDQqqE/AjSxcM6d
+GNYIAwlG7mDgfrbESQRRfXBgvKqy/3lyeqYdPV8q+Mri/Tm3R7nrft8EI6/6nAYH
+6ftjk4BAtcZsCjEozgyfz7MjNYBBjWzEN3uBL4ChQEKF6dk4jeihU80Bv2noWgby
+RQuQ+q7hv53yrlc8pa6yVvSLZUDp/TGBLPQ5Cdjua6e0ph0VpZj3AYHYhX3zUVxx
+iN66zB+Afko=
+-----END CERTIFICATE-----
@@ -0,0 +1,32 @@
+-----BEGIN CERTIFICATE-----
+MIIFlTCCA32gAwIBAgILAIZNvw/jXtd9jtgwDQYJKoZIhvcNAQEMBQAwZzELMAkG
+A1UEBhMCSU4xEzARBgNVBAsTCmVtU2lnbiBQS0kxJTAjBgNVBAoTHGVNdWRocmEg
+VGVjaG5vbG9naWVzIExpbWl0ZWQxHDAaBgNVBAMTE2VtU2lnbiBSb290IENBIC0g
+RzIwHhcNMTgwMjE4MTgzMDAwWhcNNDMwMjE4MTgzMDAwWjBnMQswCQYDVQQGEwJJ
+TjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNobm9s
+b2dpZXMgTGltaXRlZDEcMBoGA1UEAxMTZW1TaWduIFJvb3QgQ0EgLSBHMjCCAiIw
+DQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMNwGIWW2kHfHK+sXTNwxF07K+IV
+ySTuyFM2r1v002wUfcdT+zs5OM5QbMYFFnedXQI6gCFLsjKrcaej48Zt37OyEb3i
+aPs7CsP4kAyTwzKH9aZe6gXYHrJq40/ZVMNcQVI2PcIp40B/SAN2gUZ+ZaUtIOvV
+jEx26/ebNaXRIsthlkOG/caB+QRwDw1tl7338Zlv0M2oTBUy4B3e7dGP5pgXH71M
+jqHPCoNo+xv9f0NTBT+hUDa8h8wUtcGQq9CDeJTpjWcD2bP2AMdVG6oVpMAUeUzo
+cCyglvtFdUMjggxBbw4qhau1HXPG8Ot9hwL7ZMi8tkTzrvUIxxb8G9LF/7kKeCE7
+tGZaVzDTnXuifl3msR4ErHsQ4P7lVu2AIjIAhrAXoedDidb7pMcf7TABdrYUT1Jo
+G/AiK+J9jO6GTjeADD4LMDSBZhHMuBK/PJ/g0kGBt+/C1L+/HURzQhJkMlRnM6Rv
+XoCtfKopSlns5trZmTi971Wjbn88QXP61lGpBCUPwCjs7rpOYvSUJtI+lcbF+37q
+kIqOXYkVT3cupDSpw+H89kFtj5GKY+Xny4LxY+3IvDIRiyd6ky1DPj713DI0yqve
+EpsIr3A0PdwuyUI7CS1jg0NnGFT6Xxyr0xB+VDt83FJYW8v16k2pbaQ4kVxA3aXd
+X9dZYyVR1S59KM75AgMBAAGjQjBAMB0GA1UdDgQWBBTt7E1FYRgo57MjKBEcTaUn
+DV7s9DAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0B
+AQwFAAOCAgEACFC/ilQg8KTCVBxFJW/sazomkS0kNYbEIZg4B3obqwsJ7SX98z8Z
+gfzBpz0nYClwwJjWbFN1R2zY8pCEot6/dgmA8Vbq0GxhwPM5YN/SZquNyRIxO3cU
+dlAcwf+vSezdVCf9wOzvSAF3q0a5ljvbdbNJNpfScQVp7UUd5sBsZk8jXO1KQ/go
+/Vf/GDPnrIFmxpAIGE3sgnO8lAv9FzUaAeuv7HWe47xN9J7+bQzF93yHuIXACPTL
+pQHhg2zMv5C7BAbuDHfbj1Cu294Z832yhSfBcziWGskOvl3es2EcHytbS9c9P+0z
+Mpka7zGC1FHrvLb/FoduH86TeZt0QjZ6pcplNzoaxDnDvzTJ6CC2Eny+qH/APFCu
+VUv5/wjwF+HPm8Pup2ARj9cEp92+0qcerfHacNq5hMeGZdbA/dzdUR/5z5zXdxAk
+nl8mcfGb0eMNSTXQmmB/i4AecNnr72uYjzlaXUGYN7Nrb6XouG0pnh0/BBtWWp0U
+ShIPpWEAqs7RJBj6+1ZUYXZ4ObrCw962DxhN2p19Hxw9LtuUUcLqqTPrFXYvwO4t
+ouj7KJnAkaTUfXGdEaFVtFig1EA30WzJY2X1vAQ7hVnniCjgaXAGqjsU6sklNM9n
+xDx5rFCCCEtj9Kh8UHjGK2QqgP5kwgttjOApQMaCoezMfK4KD7WpOXU=
+-----END CERTIFICATE-----
@@ -0,0 +1,32 @@
+-----BEGIN CERTIFICATE-----
+MIIFdjCCA16gAwIBAgIQCIYhdXKnxAzDuy7l1y1ubDANBgkqhkiG9w0BAQwFADBV
+MQswCQYDVQQGEwJJTjElMCMGA1UECgwcZU11ZGhyYSBUZWNobm9sb2dpZXMgTGlt
+aXRlZDEfMB0GA1UEAwwWZW1TaWduIFJvb3QgQ1MgQ0EgLSBHMjAeFw0yNDAyMDgw
+ODE1MzlaFw00OTAyMDcwODE1MzlaMFUxCzAJBgNVBAYTAklOMSUwIwYDVQQKDBxl
+TXVkaHJhIFRlY2hub2xvZ2llcyBMaW1pdGVkMR8wHQYDVQQDDBZlbVNpZ24gUm9v
+dCBDUyBDQSAtIEcyMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA4CVO
+xuBjdLJXpkrjosA+7ygTGaYzufUJrhQOeIZEuCqeCPI8hDqIu+MIqEqOlgDFetVM
+3Cor3CGBlGJzfzzlI31vyElvQq88jjCfo12Ghqcf+NAZDFduJk5+NgC0z3lfdiB6
+NNzxbfVkA+X6oDNNRioVUZsVFuIzT9Qt0Sh4KwQPRFBc/wI2yGpDuKHNsLehcECx
+Uw00m33xGYh9UkeL3reOmAaLYWZiO+Pd2vxPtrkXI4wyN4BMlESJ73uP0tyX9lq4
+M4yN2yh3i+ETI8/jzPjxHD7QFaVkCoS+uj2YmBa3mytlqlBgyOoKueNeotX8hfzB
+FNm962abpV4mN0SoC6hjkYv8IayseRPFv9VI0meQXg9OLRoNvBOqG5kw937FH0Li
+q3CIzKyvdyeYRT0LJdWA2iuB9l+4zzIfCwGyT4M4VDmJuPBhrTmjXKae6AUtm1eO
+q2f8gftA6J6AlbD0gRd/t8CrFssFNm+LYAEd36kWRBuolbaOfbSbEww84xNAgj4U
+SPwYjtNDsCkQjtgSOaDtDOmolN+Mbk528Y+LHMlbd4T3Xmp+VIWp5+70GhBDOA91
+hhtims+hHkXjRI2U7WrJLcTlcF8ykiymRbFKhwOBvTmtZNbHmDI7scU4wWUAsmW6
+tWaeeoM50KBl8ai28A6yBGYMO//03Ubvqiqst80CAwEAAaNCMEAwHQYDVR0OBBYE
+FOYcxqontIvyvYVkX6/0unKm0AF1MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8E
+BTADAQH/MA0GCSqGSIb3DQEBDAUAA4ICAQCv5Asord7fdQMT30emwb+oLuDUxdmJ
+DQHnzxIXoNvymDTbNHF9LqjeXc+X+XcjIV8PTp4o2I9wyVppHmcaVgRjN/G9FxeN
+5tCQ97v5sdx4eAdPljK1xqg7Vhfmzcp+7rSjJG7Sn9gMlZx81kL3Oa2qZj+W5Fi6
+tN4vaR9OLEyvKKHDG/eFxS/qghD1d/Clkb7UGGjBcWJlaZTWVALMKG+QCxiMoELi
+oeKaoBbZoXooxfF0mWdkIl9RWNU4r9O/yt66LBE9rj49dX19L+rGg71/4bTi5ua+
+Wz60XaOc2CdHJaBSols5hWEx/LBjS7UzfU3uJfnpQXVDi0US5sFzIHIlS6kjh7h
+YuF4+/9ZAl4Bx3w2TNN7pVm7I39pgYl7XN6KZRwfkAv4aRswPqhWbQfr3+qfcC93
+cgTyFy2RG5YE/OfmpSs6VtYKawqsdEODz/p2nhZfSDrVDnEPvERqahttY5+gaWNc
+2fbibgKMfuejTZaJ3yttPslVSko5SDs0xKU423swu34sb/fdISqtOu4NzPCZiHSG
+ladaoITmSA6UWFk4fdqaLBI+xSxY5Et3S1APp5r3j/l1KrWPNbb1/+ydHN+YLbh8
+hCBIecKdk6VbnnFRlulm0d2RSgpfCRrWLomN8JHCxCMVv/hBQXz3AXGM1ZlfYTyl
+E1BpDby0G2eEkw==
+-----END CERTIFICATE-----
@@ -10,3 +10,4 @@ include:
  - ca.server
 {% endif %}
  - ca.trustca
+  - ca.trust
@@ -0,0 +1,12 @@
+# Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
+# or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at 
+# https://securityonion.net/license; you may not use this file except in compliance with the
+# Elastic License 2.0.
+
+{# Containers mount the extracted directory rather than single files: update-ca-trust replaces
+   files by rename, so a file bind mount would keep pointing at the old inode. #}
+{% set CATRUST = {
+    'dir': '/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro',
+    'java': '/etc/pki/ca-trust/extracted/java/cacerts',
+    'pem': '/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'
+} %}
@@ -0,0 +1,37 @@
+# Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
+# or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at 
+# https://securityonion.net/license; you may not use this file except in compliance with the
+# Elastic License 2.0.
+
+ca_trust_anchor:
+  file.managed:
+    - name: /etc/pki/ca-trust/source/anchors/so-intca.crt
+    - source: salt://ca/files/ca.crt
+    - show_changes: False
+    - makedirs: True
+
+ca_trust_legacy_anchor:
+  file.absent:
+    - name: /etc/pki/ca-trust/source/anchors/ca.crt
+
+ca_trust_legacy_bundles:
+  file.absent:
+    - name: /opt/so/conf/ca
+
+# clean removes blocklist entries that are no longer shipped
+ca_trust_blocklist:
+  file.recurse:
+    - name: /etc/pki/ca-trust/source/blocklist
+    - source: salt://ca/files/blocklist
+    - clean: True
+    - makedirs: True
+    - file_mode: '0644'
+    - dir_mode: '0755'
+
+ca_trust_extract:
+  cmd.run:
+    - name: update-ca-trust extract
+    - onchanges:
+      - file: ca_trust_anchor
+      - file: ca_trust_legacy_anchor
+      - file: ca_trust_blocklist
@@ -17,6 +17,7 @@ import time
 EXPECTED_CONTAINERS_FILE = "/opt/so/conf/so-status/so-status.conf"
 HIGHSTATE_COMPLETE_FILE = "/opt/so/log/salt/lasthighstate"
 UPTIME_FILE = "/proc/uptime"
+POSTGRES_MAINTENANCE_STATUS_FILE = "/opt/so/log/postgres/maintenance.status"

 def showUsage(options, args):
  print('Usage: {} [OPTIONS]'.format(sys.argv[0]))
@@ -149,6 +150,20 @@ def check_container_status(options, console):
  return code, container_list


+def check_postgres_maintenance(code, container_list):
+  try:
+    with open(POSTGRES_MAINTENANCE_STATUS_FILE, "r") as status_file:
+      status = status_file.readline().strip()
+  except OSError:
+    return code
+  if status.startswith("failed"):
+    reason = status[len("failed"):].lstrip(":").strip()
+    container_list.append({ "Name": "so-postgres-maintenance", "Status": "failed", "Details": reason })
+    if code == 0:
+      code = 1
+  return code
+
+
 def check_status(options, console):
  container_list = []
  code = check_installation_status(options, console)
@@ -160,6 +175,7 @@ def check_status(options, console):
  # on the Grid until a highstate completes.
  if code == 0 or code == 2:
    container_code, container_list = check_container_status(options, console)
+    container_code = check_postgres_maintenance(container_code, container_list)
    if code == 0:
      code = container_code
  output(options, console, code, container_list)
@@ -438,6 +438,10 @@ install_networkminer:
  pkg.latest:
    - name: securityonion-networkminer

+install_so_crates:
+  pkg.latest:
+    - name: securityonion-so-crates
+
 {% else %}

 desktop_packages_os_fail:
@@ -1,18 +1,10 @@
- {% from 'vars/globals.map.jinja' import GLOBALS %}
+{% from 'vars/globals.map.jinja' import GLOBALS %}

 {# we only want this state to run it is CentOS #}
 {% if GLOBALS.os == 'OEL' %}

-trusted_ca:
-  file.managed:
-    - name: /etc/pki/ca-trust/source/anchors/ca.crt
-    - source: salt://ca/files/ca.crt
-
-update_ca_certs:
-  cmd.run:
-    - name: update-ca-trust
-    - onchanges:
-      - file: trusted_ca
+include:
+  - ca.trust

 {% else %}

@@ -6,9 +6,11 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}

 include:
+  - ca.trust
  - elastalert.config
  - elastalert.sostatus

@@ -28,6 +30,7 @@ so-elastalert:
        - ipv4_address: {{ DOCKERMERGED.containers['so-elastalert'].ip }}
    - detach: True
    - binds:
+      - {{ CATRUST.dir }}
      - /opt/so/rules/elastalert:/opt/elastalert/rules/:ro
      - /opt/so/log/elastalert:/var/log/elastalert:rw
      - /opt/so/conf/elastalert/modules/:/opt/elastalert/modules/:ro
@@ -46,8 +49,9 @@ so-elastalert:
      - {{ XTRAHOST }}
        {% endfor %}
      {% endif %}
-      {% if DOCKERMERGED.containers['so-elastalert'].extra_env %}
    - environment:
+      - REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt
+      {% if DOCKERMERGED.containers['so-elastalert'].extra_env %}
        {% for XTRAENV in DOCKERMERGED.containers['so-elastalert'].extra_env %}
      - {{ XTRAENV }}
        {% endfor %}
@@ -65,6 +69,7 @@ so-elastalert:
      - file: elastacustmodulesdir
      - file: elastaconf
    - watch:
+      - cmd: ca_trust_extract
      - file: elastaconf
    - onlyif:
      - "so-elasticsearch-query / | jq -r '.version.number[0:1]' | grep -q 9" {# only run this state if elasticsearch is version 9 #}
@@ -6,9 +6,11 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}

 include:
+  - ca.trust
  - elastic-fleet-package-registry.config
  - elastic-fleet-package-registry.sostatus

@@ -34,8 +36,9 @@ so-elastic-fleet-package-registry:
      {% for BINDING in DOCKERMERGED.containers['so-elastic-fleet-package-registry'].port_bindings %}
      - {{ BINDING }}
      {% endfor %}
-    {% if DOCKERMERGED.containers['so-elastic-fleet-package-registry'].custom_bind_mounts %}
    - binds:
+      - {{ CATRUST.pem }}:/etc/ssl/certs/ca-certificates.crt:ro
+    {% if DOCKERMERGED.containers['so-elastic-fleet-package-registry'].custom_bind_mounts %}
        {% for BIND in DOCKERMERGED.containers['so-elastic-fleet-package-registry'].custom_bind_mounts %}
      - {{ BIND }}
        {% endfor %}
@@ -52,6 +55,8 @@ so-elastic-fleet-package-registry:
      - {{ ULIMIT.name }}={{ ULIMIT.soft }}:{{ ULIMIT.hard }}
    {%   endfor %}
    {% endif %}
+    - watch:
+      - cmd: ca_trust_extract

 wait_for_so-elastic-fleet-package-registry:
  http.wait_for_successful_query:
@@ -6,6 +6,7 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}

 include:
@@ -37,6 +38,7 @@ so-elastic-agent:
      - {{ BINDING }}
      {% endfor %}
    - binds:
+      - {{ CATRUST.dir }}
      - /opt/so/conf/elastic-agent/elastic-agent.yml:/usr/share/elastic-agent/elastic-agent.yml:ro
      - /opt/so/log/elasticagent:/usr/share/elastic-agent/logs
      - /etc/pki/tls/certs/intca.crt:/etc/pki/tls/certs/intca.crt:ro 
@@ -65,6 +67,7 @@ so-elastic-agent:
      - file: create-elastic-agent-config
      - file: trusttheca
    - watch:
+      - cmd: ca_trust_extract
      - file: create-elastic-agent-config
      - file: trusttheca

@@ -6,6 +6,7 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'elasticfleet/map.jinja' import ELASTICFLEETMERGED %}

@@ -63,6 +64,7 @@ so-elastic-fleet:
      - {{ BINDING }}
      {% endfor %}
    - binds:
+      - {{ CATRUST.dir }}
      - /etc/pki/elasticfleet-server.crt:/etc/pki/elasticfleet-server.crt:ro
      - /etc/pki/elasticfleet-server.key:/etc/pki/elasticfleet-server.key:ro
      - /etc/pki/tls/certs/intca.crt:/etc/pki/tls/certs/intca.crt:ro
@@ -95,6 +97,7 @@ so-elastic-fleet:
    {%   endfor %}
    {% endif %}
    - watch:
+      - cmd: ca_trust_extract
      - file: trusttheca
      - x509: etc_elasticfleet_key
      - x509: etc_elasticfleet_crt
@@ -1,49 +0,0 @@
-# Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
-# or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at 
-# https://securityonion.net/license; you may not use this file except in compliance with the
-# Elastic License 2.0.
-
-{% from 'allowed_states.map.jinja' import allowed_states %}
-{% if sls.split('.')[0] in allowed_states or sls in allowed_states %}
-{%   from 'vars/globals.map.jinja' import GLOBALS %}
-
-# Move our new CA over so Elastic and Logstash can use SSL with the internal CA
-catrustdir:
-  file.directory:
-    - name: /opt/so/conf/ca
-    - user: 939
-    - group: 939
-    - makedirs: True
-
-{%   if GLOBALS.is_manager %}
-# We have to add the Manager CA to the CA list
-catrustscript:
-  cmd.script:
-    - source: salt://elasticsearch/tools/sbin_jinja/so-catrust
-    - template: jinja
-    - cwd: /opt/so
-    - defaults:
-        GLOBALS: {{ GLOBALS }}
-{%   endif %}
-
-elasticsearch_cacerts:
-  file.managed:
-    - name: /opt/so/conf/ca/cacerts
-    - source: salt://elasticsearch/cacerts
-    - user: 939
-    - group: 939
-
-elasticsearch_capems:
-  file.managed:
-    - name: /opt/so/conf/ca/tls-ca-bundle.pem
-    - source: salt://elasticsearch/tls-ca-bundle.pem
-    - user: 939
-    - group: 939
-
-{% else %}
-
-{{sls}}_state_not_allowed:
-  test.fail_without_changes:
-    - name: {{sls}}_state_not_allowed
-
-{% endif %}
@@ -107,6 +107,12 @@ so-elasticsearch-system-indices-patch:
      - file: so-elasticsearch-system-indices-patch-script
 {%  endif %}

+so-elasticsearch-ilm-policy-reload:
+  file.absent:
+    - name: /opt/so/state/esilm.txt
+    - onchanges:
+      - file: so-elasticsearch-ilm-policy-load-script
+
 # heavynodes will only load ILM policies for SO managed indices. (Indicies defined in elasticsearch/defaults.yaml)
 so-elasticsearch-ilm-policy-load:
  cmd.run:
@@ -115,8 +121,9 @@ so-elasticsearch-ilm-policy-load:
    - require:
      - docker_container: so-elasticsearch
      - file: so-elasticsearch-ilm-policy-load-script
-    - onchanges:
-      - file: so-elasticsearch-ilm-policy-load-script
+      - http: wait_for_so-elasticsearch
+      - file: so-elasticsearch-ilm-policy-reload
+    - unless: test -f /opt/so/state/esilm.txt

 so-elasticsearch-templates-reload:
  file.absent:
@@ -6,6 +6,7 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'elasticsearch/config.map.jinja' import ELASTICSEARCH_NODES %}
 {%   from 'elasticsearch/config.map.jinja' import ELASTICSEARCH_SEED_HOSTS %}
@@ -13,7 +14,6 @@

 include:
  - ca
-  - elasticsearch.ca
  - elasticsearch.ssl
  - elasticsearch.config
  - elasticsearch.sostatus
@@ -63,15 +63,14 @@ so-elasticsearch:
      - {{ BINDING }}
      {% endfor %}
    - binds:
+      - {{ CATRUST.dir }}
      - /opt/so/conf/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
      - /opt/so/conf/elasticsearch/log4j2.properties:/usr/share/elasticsearch/config/log4j2.properties:ro
      - /nsm/elasticsearch:/usr/share/elasticsearch/data:rw
      - /opt/so/log/elasticsearch:/var/log/elasticsearch:rw
-      - /opt/so/conf/ca/cacerts:/usr/share/elasticsearch/jdk/lib/security/cacerts:ro
      - /etc/pki/tls/certs/intca.crt:/usr/share/elasticsearch/config/ca.crt:ro
      - /etc/pki/elasticsearch.crt:/usr/share/elasticsearch/config/elasticsearch.crt:ro
      - /etc/pki/elasticsearch.key:/usr/share/elasticsearch/config/elasticsearch.key:ro
-      - /etc/pki/elasticsearch.p12:/usr/share/elasticsearch/config/elasticsearch.p12:ro
      - /opt/so/conf/elasticsearch/users_roles:/usr/share/elasticsearch/config/users_roles:ro
      - /opt/so/conf/elasticsearch/users:/usr/share/elasticsearch/config/users:ro
      {% if ELASTICSEARCHMERGED.config.path.get('repo', False) %}
@@ -85,21 +84,19 @@ so-elasticsearch:
        {% endfor %}
      {% endif %}
    - watch:
+      - cmd: ca_trust_extract
      - file: trusttheca
      - x509: elasticsearch_crt
      - x509: elasticsearch_key
-      - file: elasticsearch_cacerts
      - file: esyml
    - require:
      - file: trusttheca
      - x509: elasticsearch_crt
      - x509: elasticsearch_key
-      - file: elasticsearch_cacerts
      - file: esyml
      - file: eslog4jfile
      - file: nsmesdir
      - file: eslogdir
-      - file: elasticp12perms
      - cmd: auth_users_roles_inode
      - cmd: auth_users_inode

@@ -38,10 +38,6 @@ elasticsearch_crt:
    - retry:
        attempts: 5
        interval: 30
-  cmd.run:
-    - name: "/usr/bin/openssl pkcs12 -inkey /etc/pki/elasticsearch.key -in /etc/pki/elasticsearch.crt -export -out /etc/pki/elasticsearch.p12 -nodes -passout pass:"
-    - onchanges:
-      - x509: /etc/pki/elasticsearch.key

 elastickeyperms:
  file.managed:
@@ -50,13 +46,6 @@ elastickeyperms:
    - mode: 640
    - group: 930
    
-elasticp12perms:
-  file.managed:
-    - replace: False
-    - name: /etc/pki/elasticsearch.p12
-    - mode: 640
-    - group: 930
-
 {% else %}

 {{sls}}_state_not_allowed:
@@ -1,26 +0,0 @@
-#!/bin/bash
-#
-# Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
-# or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at 
-# https://securityonion.net/license; you may not use this file except in compliance with the
-# Elastic License 2.0.
-
-. /usr/sbin/so-common
-get_elastic_agent_vars
-
-# Exit on errors, since all lines must succeed
-set -e
-
-# Check to see if we have extracted the ca cert.
-if [ ! -f /opt/so/saltstack/local/salt/elasticsearch/cacerts ]; then
-    docker run -v /etc/pki/ca.crt:/etc/ssl/ca.crt --name so-elasticsearchca --user root --entrypoint jdk/bin/keytool {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-elasticsearch:$ELASTIC_AGENT_TARBALL_VERSION -keystore /usr/share/elasticsearch/jdk/lib/security/cacerts -alias SOSCA -import -file /etc/ssl/ca.crt -storepass changeit -noprompt
-    # Make sure symbolic links are followed when copying from container
-    docker cp -L so-elasticsearchca:/usr/share/elasticsearch/jdk/lib/security/cacerts /opt/so/saltstack/local/salt/elasticsearch/cacerts
-    docker cp -L so-elasticsearchca:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem /opt/so/saltstack/local/salt/elasticsearch/tls-ca-bundle.pem
-    docker rm so-elasticsearchca
-    echo "" >> /opt/so/saltstack/local/salt/elasticsearch/tls-ca-bundle.pem
-    echo "sosca" >> /opt/so/saltstack/local/salt/elasticsearch/tls-ca-bundle.pem
-    cat /etc/pki/ca.crt >> /opt/so/saltstack/local/salt/elasticsearch/tls-ca-bundle.pem
-else
-    exit 0
-fi
@@ -7,11 +7,17 @@
 . /usr/sbin/so-common

 MAX_JOBS=${MAX_ILM_JOBS:-10}
+ILM_STATEFILE=/opt/so/state/esilm.txt
+rm -f "$ILM_STATEFILE"

+check_elasticsearch_responsive
+
+ILM_TMP_DIR=$(mktemp -d)
+trap 'rm -rf "$ILM_TMP_DIR"' EXIT
 # Lock used to serialize block writes so concurrent jobs never interleave their output.
-ILM_OUTPUT_LOCK=$(mktemp)
-ILM_FAIL_FILE=$(mktemp)
-trap 'rm -f "$ILM_OUTPUT_LOCK" "$ILM_FAIL_FILE"' EXIT
+ILM_OUTPUT_LOCK="$ILM_TMP_DIR/output.lock"
+ILM_FAIL_FILE="$ILM_TMP_DIR/failed"
+touch "$ILM_FAIL_FILE"

 # Policies are loaded concurrently (up to MAX_JOBS at a time) for speed. Each policy's block is
 # printed the moment its curl returns, so output appears in COMPLETION ORDER, not the order
@@ -20,13 +26,30 @@ echo "Loading ILM policies concurrently; output below appears in completion orde
 echo

 put_policy() {
-  local desc="$1" policyname="$2" data="$3" result rc=0
-  if ! result=$(curl -K /opt/so/conf/elasticsearch/curl.config -s -k -L --fail \
-    -X PUT "https://localhost:9200/_ilm/policy/${policyname}" \
-    -H 'Content-Type: application/json' -d"${data}" 2>&1); then
-    rc=1
-  elif ! jq -e '.acknowledged == true' <<<"$result" >/dev/null 2>&1; then
-    rc=1
+  local desc="$1" policyname="$2" data="$3" rc=1 attempt out http_code body result reason tmpdata
+  tmpdata=$(mktemp "$ILM_TMP_DIR/${policyname}.XXXXXX.json")
+  printf '%s' "$data" >"$tmpdata"
+
+  for attempt in 1 2 3; do
+    out=$(so-elasticsearch-query "_ilm/policy/${policyname}" -d@"${tmpdata}" -XPUT -w '\n%{http_code}' 2>/dev/null)
+    http_code=${out##*$'\n'} body=${out%$'\n'*}
+
+    if [[ "$http_code" == 200 ]] && jq -e '.acknowledged == true' <<<"$body" >/dev/null 2>&1; then
+      rc=0
+      break
+    fi
+    # 429 = ES rejected/throttled (may succeed later); other 4xx will not
+    [[ "$http_code" == 4?? && "$http_code" != 429 ]] && break
+    (( attempt < 3 )) && sleep 3
+  done
+
+  if (( rc == 0 )); then
+    result='{"acknowledged":true}'
+  else
+    result=${body:-"HTTP ${http_code:-000}"}
+    # flatten tabs/newlines so the TAB fail-file stays one record
+    reason=$(jq -r '(if (.error|type) == "object" then (.error.reason // .error.type) elif (.error|type) == "string" then .error else empty end) // empty | tostring | gsub("[\t\r\n]+"; " ")' <<<"$body" 2>/dev/null)
+    reason="HTTP ${http_code:-000}${reason:+: ${reason}}"
  fi

  # curl above ran in parallel; serialize just this block write so concurrent jobs never interleave.
@@ -34,7 +57,7 @@ put_policy() {
    flock 200
    printf 'Setting up %s policy...\n%s\n\n' "${desc}" "${result}"
    if (( rc != 0 )); then
-      printf '%s\n' "${policyname}" >>"$ILM_FAIL_FILE"
+      printf '%s\t%s\n' "${policyname}" "${reason}" >>"$ILM_FAIL_FILE"
    fi
  } 200>>"${ILM_OUTPUT_LOCK}"

@@ -82,10 +105,15 @@ wait || true

 if [[ -s "$ILM_FAIL_FILE" ]]; then
  echo "ERROR: Failed to load ILM policy(s):"
-  while read -r POLICY; do
-    echo " - $POLICY"
+  while IFS=$'\t' read -r POLICY REASON; do
+    if [[ -n "$REASON" ]]; then
+      echo " - ${POLICY}: ${REASON}"
+    else
+      echo " - ${POLICY}"
+    fi
  done < "$ILM_FAIL_FILE"
  exit 1
 else
  echo "Successfully loaded all ILM policies."
+  touch "$ILM_STATEFILE"
 fi
@@ -13,9 +13,11 @@
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   if 'api' in salt['pillar.get']('features', []) %}

 include:
+  - ca.trust
  - hydra.config
  - hydra.sostatus

@@ -29,6 +31,7 @@ so-hydra:
      - soauth:
        - ipv4_address: {{ DOCKERMERGED.containers['so-hydra'].ips['soauth'] }}
    - binds:
+      - {{ CATRUST.dir }}
      - /opt/so/conf/hydra/:/hydra-conf:ro
      - /opt/so/log/hydra/:/hydra-log:rw
      - /nsm/hydra/db:/hydra-data:rw
@@ -60,6 +63,7 @@ so-hydra:
    {%   endfor %}
    {% endif %}
    - watch:
+      - cmd: ca_trust_extract
      - file: hydraconfig
    - require:
      - file: hydraconfig
@@ -6,9 +6,11 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}

 include:
+  - ca.trust
  - idh.config
  - idh.sostatus

@@ -20,6 +22,7 @@ so-idh:
    - detach: True
    - network_mode: host
    - binds:
+      - {{ CATRUST.dir }}
      - /nsm/idh:/var/tmp:rw
      - /opt/so/conf/idh/http-skins:/opt/opencanary/http-skins:ro
      - /opt/so/conf/idh/opencanary.conf:/etc/opencanaryd/opencanary.conf:ro
@@ -34,12 +37,13 @@ so-idh:
      - {{ XTRAHOST }}
      {% endfor %}
    {% endif %}
-    {% if DOCKERMERGED.containers['so-idh'].extra_env %}
    - environment:
-      {% for XTRAENV in DOCKERMERGED.containers['so-idh'].extra_env %}
+      - REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt
+      {% if DOCKERMERGED.containers['so-idh'].extra_env %}
+        {% for XTRAENV in DOCKERMERGED.containers['so-idh'].extra_env %}
      - {{ XTRAENV }}
-      {% endfor %}
-    {% endif %}
+        {% endfor %}
+      {% endif %}
    {% if DOCKERMERGED.containers['so-idh'].ulimits %}
    - ulimits:
    {%   for ULIMIT in DOCKERMERGED.containers['so-idh'].ulimits %}
@@ -47,6 +51,7 @@ so-idh:
    {%   endfor %}
    {% endif %}
    - watch:
+      - cmd: ca_trust_extract
      - file: opencanary_config
    - require:
      - file: opencanary_config
@@ -6,11 +6,13 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   set PASSWORD = salt['pillar.get']('secrets:influx_pass') %}
 {%   set TOKEN = salt['pillar.get']('influxdb:token') %}

 include:
+  - ca.trust
  - influxdb.ssl
  - influxdb.config
  - influxdb.sostatus
@@ -38,6 +40,7 @@ so-influxdb:
        {% endfor %}
      {% endif %}
    - binds:
+      - {{ CATRUST.pem }}:/etc/ssl/certs/ca-certificates.crt:ro
      - /opt/so/log/influxdb/:/log:rw
      - /opt/so/conf/influxdb/config.yaml:/conf/config.yaml:ro
      - /opt/so/conf/influxdb/etc:/etc/influxdb2:rw
@@ -66,6 +69,7 @@ so-influxdb:
    {%   endfor %}
    {% endif %}
    - watch:
+      - cmd: ca_trust_extract
      - file: influxdbconf
      - x509: influxdb_key
      - x509: influxdb_crt
@@ -12,12 +12,14 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   set KAFKANODES = salt['pillar.get']('kafka:nodes') %}
 {%   set KAFKA_EXTERNAL_ACCESS = salt['pillar.get']('kafka:config:external_access:enabled', default=False) %}
 {%   if 'gmd' in salt['pillar.get']('features', []) %}

 include:
+  - ca.trust
  - kafka.ca
  - kafka.config
  - kafka.ssl
@@ -51,6 +53,7 @@ so-kafka:
      - {{ BINDING }}
      {% endfor %}
    - binds:
+      - {{ CATRUST.java }}:/opt/java/openjdk/lib/security/cacerts:ro
      - /etc/pki/kafka.p12:/etc/pki/kafka.p12:ro
      - /opt/so/conf/kafka/kafka-truststore.jks:/etc/pki/kafka-truststore.jks:ro
      - /nsm/kafka/data/:/nsm/kafka/data/:rw
@@ -61,6 +64,11 @@ so-kafka:
      {% if KAFKA_EXTERNAL_ACCESS %}
      - /opt/so/conf/kafka/kafka_server_jaas.conf:/opt/kafka/config/kafka_server_jaas.conf:ro
      {% endif %}
+      {% if DOCKERMERGED.containers['so-kafka'].custom_bind_mounts %}
+        {% for BIND in DOCKERMERGED.containers['so-kafka'].custom_bind_mounts %}
+      - {{ BIND }}
+        {% endfor %}
+      {% endif %}
    {% if DOCKERMERGED.containers['so-kafka'].ulimits %}
    - ulimits:
    {%   for ULIMIT in DOCKERMERGED.containers['so-kafka'].ulimits %}
@@ -68,6 +76,7 @@ so-kafka:
    {%   endfor %}
    {% endif %}
    - watch:
+      - cmd: ca_trust_extract
      {% for sc in ['server', 'client'] %}
      - file: kafka_kraft_{{sc}}_properties
      {% endfor %}
@@ -8,8 +8,10 @@
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'elasticsearch/config.map.jinja' import ELASTICSEARCHMERGED %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}

 include:
+  - ca.trust
  - kibana.config
  - kibana.sostatus

@@ -27,6 +29,7 @@ so-kibana:
      - ELASTICSEARCH_HOST={{ GLOBALS.manager }}
      - ELASTICSEARCH_PORT=9200
      - MANAGER={{ GLOBALS.manager }}
+      - NODE_OPTIONS=--use-openssl-ca
      {% if DOCKERMERGED.containers['so-kibana'].extra_env %}
        {% for XTRAENV in DOCKERMERGED.containers['so-kibana'].extra_env %}
      - {{ XTRAENV }}
@@ -40,6 +43,7 @@ so-kibana:
      {% endfor %}
    {% endif %}
    - binds:
+      - {{ CATRUST.dir }}
      - /opt/so/conf/kibana/etc:/usr/share/kibana/config:rw
      - /opt/so/log/kibana:/var/log/kibana:rw
      - /opt/so/conf/kibana/customdashboards:/usr/share/kibana/custdashboards:ro
@@ -60,6 +64,7 @@ so-kibana:
    {%   endfor %}
    {% endif %}
    - watch:
+      - cmd: ca_trust_extract
      - file: kibanaconfig

 wait_for_so-kibana:
@@ -7,8 +7,10 @@
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}

 include:
+  - ca.trust
  - kratos.config
  - kratos.sostatus

@@ -22,6 +24,7 @@ so-kratos:
      - soauth:
        - ipv4_address: {{ DOCKERMERGED.containers['so-kratos'].ips['soauth'] }}
    - binds:
+      - {{ CATRUST.dir }}
      - /opt/so/conf/kratos/:/kratos-conf:ro
      - /opt/so/log/kratos/:/kratos-log:rw
      - /nsm/kratos/db:/kratos-data:rw
@@ -53,6 +56,7 @@ so-kratos:
    {%   endfor %}
    {% endif %}
    - watch:
+      - cmd: ca_trust_extract
      - file: kratosschema
      - file: kratosconfig
    - require:
@@ -6,6 +6,7 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'logstash/map.jinja' import LOGSTASH_MERGED %}
 {%   from 'logstash/map.jinja' import LOGSTASH_NODES %}
@@ -13,9 +14,6 @@

 include:
  - ca
-{%   if GLOBALS.role not in ['so-receiver','so-fleet'] %}
-  - elasticsearch.ca
-{%   endif %}
 {# Kafka ca runs on nodes that can run logstash for Kafka input / output. Only when Kafka is global pipeline #}
 {%   if GLOBALS.role in ['so-searchnode', 'so-manager', 'so-managerhype', 'so-managersearch', 'so-receiver', 'so-standalone'] and GLOBALS.pipeline == 'KAFKA' %}
  - kafka.ca
@@ -58,6 +56,8 @@ so-logstash:
      - {{ BINDING }}
      {% endfor %}
    - binds:
+      - {{ CATRUST.dir }}
+      - {{ CATRUST.java }}:/usr/share/logstash/jdk/lib/security/cacerts:ro
      - /opt/so/conf/elasticsearch/templates/:/templates/:ro
      - /opt/so/conf/logstash/etc/:/usr/share/logstash/config/:ro
      - /opt/so/conf/logstash/pipelines:/usr/share/logstash/pipelines:ro
@@ -78,10 +78,6 @@ so-logstash:
      - /etc/pki/filebeat.p8:/usr/share/logstash/filebeat.key:ro
          {% endif %}
      {% endif %}
-      {% if GLOBALS.role not in ['so-receiver','so-fleet'] %}
-      - /opt/so/conf/ca/cacerts:/etc/pki/ca-trust/extracted/java/cacerts:ro
-      - /opt/so/conf/ca/tls-ca-bundle.pem:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:ro
-      {% endif %}
      {% if GLOBALS.pipeline == "KAFKA" and GLOBALS.role in ['so-manager', 'so-managerhype', 'so-managersearch', 'so-standalone', 'so-searchnode'] %}
      - /etc/pki/kafka-logstash.p12:/usr/share/logstash/kafka-logstash.p12:ro
      - /opt/so/conf/kafka/kafka-truststore.jks:/etc/pki/kafka-truststore.jks:ro
@@ -104,14 +100,11 @@ so-logstash:
    {%   endfor %}
    {% endif %}
    - watch:
+      - cmd: ca_trust_extract
      - file: lsetcsync
      - file: lslog4j2
      - file: lspipelinesyml
      - file: trusttheca
-      {% if GLOBALS.is_manager %}
-      - file: elasticsearch_cacerts
-      - file: elasticsearch_capems
-      {% endif %}
      {% if GLOBALS.is_manager or GLOBALS.role in ['so-fleet', 'so-heavynode', 'so-receiver'] %}
      - x509: etc_elasticfleet_logstash_crt
      - x509: etc_elasticfleet_logstash_key
@@ -133,10 +126,6 @@ so-logstash:
      {% endif %}
    - require:
      - file: trusttheca
-      {% if GLOBALS.is_manager %}
-      - file: elasticsearch_cacerts
-      - file: elasticsearch_capems
-      {% endif %}
      {% if GLOBALS.is_manager or GLOBALS.role in ['so-fleet', 'so-heavynode', 'so-receiver'] %}
      - x509: etc_elasticfleet_logstash_crt
      - x509: etc_elasticfleet_logstash_key
@@ -121,9 +121,6 @@ check_err() {
      161)
        echo 'Required intermediate Elasticsearch upgrade not complete'
      ;;
-      162)
-        echo 'One or more Elastic Agent nodes do not support the x86-64-v3 CPU instruction set'
-      ;;
      170)
        echo "Intermediate upgrade completed successfully to $next_step_so_version, but next soup to Security Onion $originally_requested_so_version could not be started automatically."
        echo "Start soup again manually to continue the upgrade to Security Onion $originally_requested_so_version."
@@ -351,83 +348,6 @@ check_cluster_health() {
  exit 0
 }

-no_soup_for_you() {
-  echo ""
-  echo "No soup for you!"
-  exit 162
-}
-
-check_cpu_compatibility() {
-  # Roles running a container built from the so-elastic-agent image; mirrors the
-  # elasticagent and elasticfleet entries in salt/reactor/pillar_push_map.yaml.
-  local cpu_target='G@role:so-heavynode or G@role:so-eval or G@role:so-fleet or G@role:so-import or G@role:so-manager or G@role:so-managerhype or G@role:so-managersearch or G@role:so-standalone'
-  local expected_nodes cpu_results node result confirm
-  local -a unsupported=() offline=()
-
-  echo "Checking that Elastic Agent nodes support the x86-64-v3 CPU instruction set now required by Elastic."
-
-  if [[ "$SKIP_CPU_CHECK" == "true" ]]; then
-    printf "\nSkipping the x86-64-v3 CPU check because --skip-cpu-check was specified.\n\n"
-    return
-  fi
-
-  # Nodes that never answer are absent from the results, so diff against who should have.
-  expected_nodes=$(salt -C "$cpu_target" --preview-target --out=json 2>/dev/null | jq -r '.[]?') || true
-  if [[ -z "$expected_nodes" ]]; then
-    printf "\nCould not determine which nodes run the Elastic Agent, so the x86-64-v3 CPU check cannot run.\n"
-    no_soup_for_you
-  fi
-
-  cpu_results=$(salt -t 30 -C "$cpu_target" cmd.run "/lib64/ld-linux-x86-64.so.2 --help | grep x86-64-v3" --out=json 2>/dev/null) || true
-
-  while IFS= read -r node; do
-    [[ -z "$node" ]] && continue
-    result=$(jq -r --arg node "$node" '.[$node] // empty' <<< "$cpu_results" 2>/dev/null)
-    if [[ -z "$result" || "$result" == *"did not return"* ]]; then
-      offline+=("$node")
-    elif [[ "$result" != *"x86-64-v3 (supported"* ]]; then
-      # glibc appends "(supported, searched)" only when supported; the open paren keeps
-      # this from matching a future "(unsupported".
-      unsupported+=("$node")
-    fi
-  done <<< "$expected_nodes"
-
-  if [[ ${#unsupported[@]} -eq 0 && ${#offline[@]} -eq 0 ]]; then
-    printf "\nAll Elastic Agent nodes support x86-64-v3. We can proceed with SOUP.\n\n"
-    return
-  fi
-
-  echo ""
-  if [[ ${#unsupported[@]} -gt 0 ]]; then
-    echo "The following node(s) do NOT support the x86-64-v3 CPU instruction set:"
-    printf '  %s\n' "${unsupported[@]}"
-    echo ""
-    echo "Upstream Elastic now builds its binaries for x86-64-v3, so these nodes can no"
-    echo "longer run Elastic. Upgrading them WILL BREAK them."
-    echo ""
-  fi
-  if [[ ${#offline[@]} -gt 0 ]]; then
-    echo "The following node(s) did not respond and could not be checked:"
-    printf '  %s\n' "${offline[@]}"
-    echo ""
-    echo "These nodes are offline, so we cannot confirm they support x86-64-v3, which"
-    echo "upstream Elastic now requires."
-    echo ""
-  fi
-
-  if [[ -n $UNATTENDED ]]; then
-    echo "Unattended mode cannot prompt for an override. Re-run soup interactively, or pass --skip-cpu-check to bypass this check."
-    no_soup_for_you
-  fi
-
-  read -rp "Type 'override' to continue anyway, or press Enter to exit: " confirm
-  if [[ "${confirm,,}" == "override" ]]; then
-    printf "\nOverride accepted. Continuing at your own risk.\n\n"
-  else
-    no_soup_for_you
-  fi
-}
-
 check_fleet_server() {
  echo "Checking that Elastic Fleet Server is responding."
  # Modeled on the wait_for_so-elastic-fleet state check in elasticfleet/enabled.sls,
@@ -626,6 +546,7 @@ postupgrade_changes() {
    [[ "$POSTVERSION" == "3.1.0" ]] && post_to_3.2.0
    [[ "$POSTVERSION" == "3.2.0" ]] && post_to_3.3.0
    [[ "$POSTVERSION" == "3.3.0" ]] && post_to_3.4.0
+    [[ "$POSTVERSION" == "3.4.0" ]] && post_to_3.5.0
    # All applicable post-upgrade steps completed; clear the resume marker.
    rm -f "$POSTVERSION_FILE"
    true
@@ -1149,11 +1070,14 @@ telegraf_repair() {

  echo "Checking Telegraf metric partitions."
  local status=0
-  "$repair" --check >> "$SOUP_LOG" 2>&1 || status=$?
+  local max_age
+  max_age=$(lookup_pillar maintenance_max_age_hours postgres:telegraf 2>/dev/null) || true
+  max_age=${max_age:-3}
+  MAINTENANCE_MAX_AGE_HOURS="$max_age" "$repair" --check >> "$SOUP_LOG" 2>&1 || status=$?
  case "$status" in
    0) echo "  Telegraf partitions are healthy; nothing to repair." ;;
    1) echo "  Repairing stalled Telegraf partitions."
-       "$repair" --yes \
+       MAINTENANCE_MAX_AGE_HOURS="$max_age" "$repair" --yes \
         || echo "  warning: so-telegraf-repair failed; run it manually" >&2 ;;
    *) echo "  Skipping; Telegraf is not storing metrics in Postgres on this host." ;;
  esac
@@ -1261,9 +1185,15 @@ post_to_3.4.0() {

  add_detection_runtime_fields

+  remove_frozen_ca_bundles
+
  set_postversion 3.4.0
 }

+remove_frozen_ca_bundles() {
+  rm -f /opt/so/saltstack/local/salt/elasticsearch/cacerts /opt/so/saltstack/local/salt/elasticsearch/tls-ca-bundle.pem >> $SOUP_LOG 2>&1
+}
+
 add_detection_runtime_fields() {
  local template=$DEFAULT_SALT_DIR/salt/elasticsearch/templates/component/so/detection-mappings.json
  local runtime
@@ -1276,6 +1206,14 @@ add_detection_runtime_fields() {
 }
 ### 3.4.0 End ###

+### 3.5.0 Scripts ###
+post_to_3.5.0() {
+  telegraf_repair
+
+  set_postversion 3.5.0
+}
+### 3.5.0 End ###
+

 repo_sync() {
  echo "Sync the local repo."
@@ -2162,8 +2100,6 @@ main() {
  echo "Let's see if we need to update Security Onion."
  upgrade_check

-  check_cpu_compatibility
-
  upgrade_space

  echo "Verifying Elasticsearch version compatibility across the grid before upgrading."
@@ -2442,17 +2378,6 @@ fi
  echo "### soup has been served at $(date) ###"
 }

-SKIP_CPU_CHECK=false
-declare -a SOUP_ARGS=()
-for arg in "$@"; do
-  if [[ "$arg" == "--skip-cpu-check" ]]; then
-    SKIP_CPU_CHECK=true
-  else
-    SOUP_ARGS+=("$arg")
-  fi
-done
-set -- "${SOUP_ARGS[@]}"
-
 while getopts ":b:f:y" opt; do
  case ${opt} in
    b )
@@ -2476,7 +2401,7 @@ while getopts ":b:f:y" opt; do
      ISOLOC="$OPTARG"
    ;;
    \? )
-      echo "Usage: soup [-b] [-y] [-f <iso location>] [--skip-cpu-check]"
+      echo "Usage: soup [-b] [-y] [-f <iso location>]"
      exit 1
    ;;
    : )
@@ -6,10 +6,12 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'nginx/map.jinja' import NGINXMERGED %}

 include:
+  - ca.trust
  - nginx.ssl
  - nginx.config
  - nginx.sostatus
@@ -47,6 +49,7 @@ so-nginx:
      {% endfor %}
    {% endif %}
    - binds:
+      - {{ CATRUST.pem }}:/etc/ssl/certs/ca-certificates.crt:ro
      - /opt/so/conf/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - /opt/so/log/nginx/:/var/log/nginx:rw
      - /opt/so/tmp/nginx/:/var/lib/nginx:rw
@@ -89,6 +92,7 @@ so-nginx:
      - {{ BINDING }}
      {% endfor %}
    - watch:
+      - cmd: ca_trust_extract
      - file: nginxconf
      - file: nginxconfdir
      {% if GLOBALS.is_manager %}
@@ -2,6 +2,7 @@ postgres:
  enabled: True
  telegraf:
    retention_days: 14
+    maintenance_max_age_hours: 3
  config:
    listen_addresses: '*'
    port: 5432
@@ -17,3 +18,6 @@ postgres:
    log_min_messages: 'warning'
    shared_preload_libraries: pg_cron
    cron.database_name: postgres
+    # pg_hba.conf only trusts local socket connections; over TCP pg_cron has no
+    # password to send and every job fails to connect.
+    cron.host: /run/postgresql
@@ -24,6 +24,15 @@ so_postgres_backup:
    - identifier: so_postgres_backup
    - user: root

+so_postgres_maintenance_status:
+  cron.absent:
+    - identifier: so_postgres_maintenance_status
+    - user: root
+
+so_postgres_maintenance_status_file:
+  file.absent:
+    - name: /opt/so/log/postgres/maintenance.status
+
 {% else %}

 {{sls}}_state_not_allowed:
@@ -6,10 +6,13 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
+{%   from 'telegraf/map.jinja' import TELEGRAFMERGED %}
 {%   set SO_POSTGRES_USER = salt['pillar.get']('postgres:auth:users:so_postgres_user:user', 'so_postgres') %}

 include:
+  - ca.trust
  - postgres.auth
  - postgres.ssl
  - postgres.config
@@ -42,6 +45,7 @@ so-postgres:
        {% endfor %}
      {% endif %}
    - binds:
+      - {{ CATRUST.dir }}
      - /opt/so/log/postgres/:/log:rw
      - /nsm/postgres:/var/lib/postgresql/data:rw
      - /opt/so/conf/postgres/postgresql.conf:/conf/postgresql.conf:ro
@@ -69,6 +73,7 @@ so-postgres:
    {%   endfor %}
    {% endif %}
    - watch:
+      - cmd: ca_trust_extract
      - file: postgresconf
      - file: postgreshba
      - file: postgresinitdb
@@ -118,6 +123,29 @@ so_postgres_backup:
    - month: '*'
    - dayweek: '*'

+{% if TELEGRAFMERGED.output | upper in ['POSTGRES', 'BOTH'] %}
+{%   set max_age = salt['pillar.get']('postgres:telegraf:maintenance_max_age_hours', 3) | int %}
+so_postgres_maintenance_status:
+  cron.present:
+    - name: MAINTENANCE_MAX_AGE_HOURS={{ max_age }} /usr/sbin/so-telegraf-postgres status > /dev/null 2>&1
+    - identifier: so_postgres_maintenance_status
+    - user: root
+    - minute: '*/15'
+    - hour: '*'
+    - daymonth: '*'
+    - month: '*'
+    - dayweek: '*'
+{% else %}
+so_postgres_maintenance_status:
+  cron.absent:
+    - identifier: so_postgres_maintenance_status
+    - user: root
+
+so_postgres_maintenance_status_file:
+  file.absent:
+    - name: /opt/so/log/postgres/maintenance.status
+{% endif %}
+
 {% else %}

 {{sls}}_state_not_allowed:
@@ -9,6 +9,11 @@ postgres:
      description: Number of days of Telegraf metrics to keep in the so_telegraf database. Older partitions are dropped hourly by pg_partman.
      forcedType: int
      helpLink: postgres
+    maintenance_max_age_hours:
+      description: Number of hours without a successful run of the hourly pg_partman maintenance job before the Telegraf metrics database is reported as unhealthy in grid status and repaired by soup.
+      forcedType: int
+      advanced: True
+      helpLink: postgres
  config:
    max_connections:
      description: Maximum number of concurrent PostgreSQL connections.
@@ -87,3 +92,8 @@ postgres:
      global: True
      advanced: True
      helpLink: postgres
+    cron.host:
+      description: Unix socket directory pg_cron uses to run scheduled jobs. Must be a local socket directory so jobs authenticate with the local trust rule; a TCP host requires a password and jobs will fail.
+      global: True
+      advanced: True
+      helpLink: postgres
@@ -12,7 +12,11 @@ set -e
 #   retention    Reconcile partman retention and premake on telegraf parents.
 #                Env: RETENTION_DAYS.
 #   maintenance  Drain default partitions and run partman maintenance.
-#   check        Report partition health. Non-zero if any parent is unhealthy.
+#   check        Report partition and maintenance health. Non-zero if any parent
+#                is unhealthy or the maintenance job has no successful run within
+#                the last MAINTENANCE_MAX_AGE_HOURS (default 3).
+#   status       Run check and write `ok` or `failed: <reason>` to
+#                /opt/so/log/postgres/maintenance.status for so-status.
 #
 # A default partition holding rows for a day blocks creating that day's child,
 # so maintenance drains defaults before calling partman. Use so-telegraf-repair
@@ -189,6 +193,11 @@ SELECT cron.schedule_in_database(
  'CALL so_admin.telegraf_maintenance()',
  'so_telegraf'
 );
+-- pg_cron copies cron.host into nodename only when a job is first created, so
+-- jobs registered before cron.host changed keep connecting to the old host.
+UPDATE cron.job SET nodename = current_setting('cron.host')
+ WHERE jobname = 'telegraf-partman-maintenance'
+   AND nodename IS DISTINCT FROM current_setting('cron.host');
 \else
 \echo 'pg_cron metadata database is not `postgres` yet; skipping job registration.'
 \endif
@@ -278,16 +287,80 @@ LIMIT 5;
 \echo 'pg_cron is not installed in this database.'
 \endif
 EOSQL
+    max_age="${MAINTENANCE_MAX_AGE_HOURS:-3}"
+    [[ "$max_age" =~ ^[0-9]+$ ]] || { echo "MAINTENANCE_MAX_AGE_HOURS must be an integer" >&2; exit 1; }
+    failures=()
    unhealthy=$(docker exec so-postgres psql -U postgres -d so_telegraf -tAc \
      "SELECT count(*) FROM so_admin.telegraf_partition_status()
       WHERE coalesce(default_rows, 0) > 0 OR coalesce(days_ahead, -1) < 1")
    if [ "${unhealthy:-1}" != "0" ]; then
-      echo "so-telegraf-postgres check: $unhealthy telegraf parent(s) unhealthy" >&2
+      failures+=("$unhealthy telegraf parent(s) unhealthy")
+    fi
+    has_cron=$(docker exec so-postgres psql -U postgres -d postgres -tAc \
+      "SELECT to_regclass('cron.job_run_details') IS NOT NULL")
+    if [ "$has_cron" != "t" ]; then
+      failures+=("pg_cron is not installed in the postgres database")
+    else
+      runs=$(docker exec so-postgres psql -U postgres -d postgres -tA -F '|' -c \
+        "SELECT (SELECT count(*) FROM cron.job WHERE jobname = 'telegraf-partman-maintenance'),
+                (SELECT count(*) FROM cron.job WHERE jobname = 'telegraf-partman-maintenance' AND active),
+                (SELECT count(*) FROM pg_stat_activity
+                  WHERE backend_type ILIKE '%pg_cron%' OR application_name ILIKE '%pg_cron%'),
+                (SELECT count(*) FROM pg_stat_activity
+                  WHERE (backend_type ILIKE '%pg_cron%' OR application_name ILIKE '%pg_cron%')
+                    AND backend_start < now() - make_interval(hours => $max_age)),
+                count(*),
+                count(*) FILTER (WHERE d.status = 'succeeded'
+                                   AND d.start_time > now() - make_interval(hours => $max_age)),
+                coalesce((array_agg(format('last run %s at %s: %s', d.status, d.start_time,
+                                           regexp_replace(coalesce(d.return_message, ''), '\s+', ' ', 'g'))
+                                    ORDER BY d.start_time DESC))[1], '')
+           FROM cron.job j
+           JOIN cron.job_run_details d ON d.jobid = j.jobid
+          WHERE j.jobname = 'telegraf-partman-maintenance'") || runs=""
+      IFS='|' read -r jobs active launchers old_launchers run_count recent_ok last_run <<< "$runs"
+      # With no runs recorded, a live launcher younger than N hours means the
+      # job was only just scheduled (fresh install or a restart for a config
+      # change) and has not reached its first :17 yet. A launcher older than
+      # that should have fired it by now. Tradeoff: a job first scheduled long
+      # after the launcher started reports failed until its first run.
+      if [ -z "$runs" ]; then
+        failures+=("could not read telegraf-partman-maintenance runs from cron.job_run_details")
+      elif [ "$jobs" = "0" ]; then
+        failures+=("telegraf-partman-maintenance is not scheduled")
+      elif [ "$active" = "0" ]; then
+        failures+=("telegraf-partman-maintenance is inactive")
+      elif [ "$launchers" = "0" ]; then
+        failures+=("pg_cron launcher is not running; telegraf-partman-maintenance cannot run")
+      elif [ "$run_count" = "0" ]; then
+        if [ "$old_launchers" != "0" ]; then
+          failures+=("telegraf-partman-maintenance has never run although pg_cron has been up for more than $max_age hour(s)")
+        fi
+      elif [ "$recent_ok" = "0" ]; then
+        failures+=("no successful telegraf-partman-maintenance run in the last $max_age hour(s); $last_run")
+      fi
+    fi
+    if [ "${#failures[@]}" -gt 0 ]; then
+      msg=$(printf '%s; ' "${failures[@]}")
+      echo "so-telegraf-postgres check: ${msg%; }" >&2
      exit 1
    fi
    echo "so-telegraf-postgres check: all telegraf parents healthy"
    ;;

+  status)
+    status_file=/opt/so/log/postgres/maintenance.status
+    tmp=$(mktemp "$status_file.XXXXXX")
+    if reason=$("$0" check 2>&1 >/dev/null); then
+      echo ok > "$tmp"
+    else
+      reason=$(printf '%s\n' "$reason" | tail -n 1)
+      echo "failed: ${reason#so-telegraf-postgres check: }" > "$tmp"
+    fi
+    chmod 644 "$tmp"
+    mv -f "$tmp" "$status_file"
+    ;;
+
  *)
    echo "Unknown subcommand: $cmd" >&2
    exit 1
@@ -15,6 +15,9 @@
 #   --yes         Skip the confirmation prompt (for soup and other automation).
 #   --no-restart  Never restart so-postgres, even if pg_cron's launcher is dead.
 #
+# Env: MAINTENANCE_MAX_AGE_HOURS (default 3) is how long the maintenance job may
+# go without a successful run before repair is needed.
+#
 # Exit status:
 #   0  healthy, or repair completed
 #   1  repair is needed (--check only)
@@ -25,6 +28,8 @@ set -e
 # Matches p_premake in telegraf.conf's create_parent template.
 PREMAKE=7
 JOB_NAME=telegraf-partman-maintenance
+MAX_AGE_HOURS="${MAINTENANCE_MAX_AGE_HOURS:-3}"
+[[ "$MAX_AGE_HOURS" =~ ^[0-9]+$ ]] || { echo "MAINTENANCE_MAX_AGE_HOURS must be an integer" >&2; exit 2; }

 CHECK_ONLY=false
 ASSUME_YES=false
@@ -118,14 +123,30 @@ launcher=$(psql_pg -tAc \
 # so_telegraf before the postgres state lands, postgres after.
 cron_db=$(docker exec so-postgres psql -U postgres -tAc \
  "SELECT current_setting('cron.database_name', true)" | tr -d '[:space:]')
+run_count=0
+recent_ok=0
 last_run=never
-if [[ -n "$cron_db" ]]; then
-  last_run=$(docker exec so-postgres psql -U postgres -d "$cron_db" -tAc \
-    "SELECT coalesce(max(d.start_time)::text, 'never')
+has_run_details=$([[ -n "$cron_db" ]] && docker exec so-postgres psql -U postgres -d "$cron_db" -tAc \
+  "SELECT to_regclass('cron.job_run_details') IS NOT NULL" 2>/dev/null || echo f)
+if [[ "$has_run_details" == t ]]; then
+  runs=$(docker exec so-postgres psql -U postgres -d "$cron_db" -tA -F '|' -c \
+    "SELECT count(*),
+            count(*) FILTER (WHERE d.status = 'succeeded'
+                               AND d.start_time > now() - make_interval(hours => $MAX_AGE_HOURS)),
+            coalesce((array_agg(format('%s at %s', d.status, d.start_time)
+                                ORDER BY d.start_time DESC))[1], 'never')
       FROM cron.job_run_details d JOIN cron.job j USING (jobid)
-      WHERE j.jobname = '$JOB_NAME'" 2>/dev/null | tr -d '[:space:]' || echo unknown)
-  [[ -n "$last_run" ]] || last_run=never
+      WHERE j.jobname = '$JOB_NAME'" 2>/dev/null) \
+    || { echo "Could not read $JOB_NAME runs from cron.job_run_details; cannot judge maintenance health." >&2; exit 2; }
+  IFS='|' read -r run_count recent_ok last_run <<< "$runs"
 fi
+# An unreadable run history exits 2 rather than 1: soup repairs on 1, which
+# truncates the default partitions, and a transient query error must not do that.
+# No runs at all means a freshly scheduled job (or pg_cron not loaded, handled
+# by restart_needed); the partition checks catch a job that never fires. Runs
+# with no recent success mean the job fires but fails, e.g. cannot connect.
+maintenance_failing=false
+[[ "$run_count" -gt 0 && "$recent_ok" -eq 0 ]] && maintenance_failing=true

 # A grid that has never written a metric has nothing to recover, and an empty
 # cron_db means pg_cron is not loaded at all, which no restart fixes.
@@ -137,6 +158,7 @@ repair_needed=false
 [[ "$behind" -gt 0 ]] && repair_needed=true
 [[ "$misconfigured" -gt 0 ]] && repair_needed=true
 $restart_needed && repair_needed=true
+$maintenance_failing && repair_needed=true

 echo "Telegraf partition status:"
 psql_tg -c "$REPORT"
@@ -157,6 +179,7 @@ if $CHECK_ONLY; then
  [[ "$behind" -gt 0 ]]      && echo "  * $behind parent(s) with no partition for the current window"
  [[ "$misconfigured" -gt 0 ]] && echo "  * $misconfigured parent(s) with stale partman settings"
  $restart_needed            && echo "  * pg_cron's launcher is dead; maintenance is not running at all"
+  $maintenance_failing       && echo "  * no successful $JOB_NAME run in the last $MAX_AGE_HOURS hour(s)"
  echo
  echo "Re-run without --check to repair."
  exit 1
@@ -241,5 +264,6 @@ EOSQL
 echo
 echo "Telegraf partition status after repair:"
 psql_tg -c "$REPORT"
-echo "The $JOB_NAME job runs hourly at :17. Confirm it fired with:"
+echo "The $JOB_NAME job runs hourly at :17. Until its next run succeeds, grid status"
+echo "may still report so-postgres-maintenance as failed. Confirm it fired with:"
 echo "  so-telegraf-repair --check"
@@ -7,6 +7,7 @@
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}

 include:
  - ca
@@ -28,6 +29,7 @@ so-redis:
      - {{ BINDING }}
      {% endfor %}
    - binds:
+      - {{ CATRUST.pem }}:/etc/ssl/certs/ca-certificates.crt:ro
      - /opt/so/log/redis:/var/log/redis:rw
      - /opt/so/conf/redis/etc/redis.conf:/usr/local/etc/redis/redis.conf:ro
      - /opt/so/conf/redis/working:/redis:rw
@@ -60,6 +62,7 @@ so-redis:
    {% endif %}
    - entrypoint: "redis-server /usr/local/etc/redis/redis.conf"
    - watch:
+      - cmd: ca_trust_extract
      - file: trusttheca
      - x509: redis_crt
      - x509: redis_key
@@ -6,9 +6,11 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}

 include:
+  - ca.trust
  - registry.ssl
  - registry.config
  - registry.sostatus
@@ -29,6 +31,7 @@ so-dockerregistry:
      - {{ BINDING }}
      {% endfor %}
    - binds:
+      - {{ CATRUST.pem }}:/etc/ssl/certs/ca-certificates.crt:ro
      - /opt/so/conf/docker-registry/etc/config.yml:/etc/distribution/config.yml:ro
      - /opt/so/conf/docker-registry:/var/lib/registry:rw
      - /nsm/docker-registry/docker:/var/lib/registry/docker:rw
@@ -63,6 +66,7 @@ so-dockerregistry:
        attempts: 5
        interval: 30
    - watch:
+      - cmd: ca_trust_extract
      - x509: registry_crt
      - x509: registry_key
    - require:
@@ -4,10 +4,12 @@
 # Elastic License 2.0.

 {% from 'vars/globals.map.jinja' import GLOBALS %}
+{% from 'ca/trust.map.jinja' import CATRUST %}
 {% from 'docker/docker.map.jinja' import DOCKERMERGED %}


 include:
+  - ca.trust
  - sensoroni.config
  - sensoroni.sostatus

@@ -17,6 +19,7 @@ so-sensoroni:
    - restart_policy: unless-stopped
    - network_mode: host
    - binds:
+      - {{ CATRUST.dir }}
      - /nsm/import:/nsm/import:rw
      - /nsm/pcapout:/nsm/pcapout:rw
      - /opt/so/conf/sensoroni/sensoroni.json:/opt/sensoroni/sensoroni.json:ro
@@ -48,6 +51,7 @@ so-sensoroni:
    {%   endfor %}
    {% endif %}
    - watch:
+      - cmd: ca_trust_extract
      - file: /opt/so/conf/sensoroni/sensoroni.json
    - require:
      - file: sensoroniagentconf
@@ -1441,6 +1441,8 @@ soc:
          rulesFingerprintFile: /opt/sensoroni/fingerprints/sigma.fingerprint
          stateFilePath: /opt/sensoroni/fingerprints/elastalertengine.state
          integrityCheckFrequencySeconds: 1200
+          partialSyncTimeoutSeconds: 300
+          bulkSyncTimeoutSeconds: 1800
          rulesRepos:
            default:
              - repo: https://github.com/Security-Onion-Solutions/securityonion-resources
@@ -1700,6 +1702,8 @@ soc:
          yaraRulesFolder: /opt/sensoroni/yara/rules
          stateFilePath: /opt/sensoroni/fingerprints/strelkaengine.state
          integrityCheckFrequencySeconds: 1200
+          partialSyncTimeoutSeconds: 300
+          bulkSyncTimeoutSeconds: 1800
        suricataengine:
          aiRepoUrl: https://github.com/Security-Onion-Solutions/securityonion-resources
          aiRepoBranch: generated-summaries-published
@@ -1715,6 +1719,8 @@ soc:
          rulesFingerprintFile: /opt/sensoroni/fingerprints/emerging-all.fingerprint
          stateFilePath: /opt/sensoroni/fingerprints/suricataengine.state
          integrityCheckFrequencySeconds: 1200
+          partialSyncTimeoutSeconds: 300
+          bulkSyncTimeoutSeconds: 1800
          ignoredSidRanges:
            - '1100000-1101000'
          rulesetSources:
@@ -6,6 +6,7 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'soc/merged.map.jinja' import DOCKER_EXTRA_HOSTS %}
 {%   from 'soc/merged.map.jinja' import SOCMERGED %}
@@ -27,6 +28,7 @@ so-soc:
      - soauth:
        - ipv4_address: {{ DOCKERMERGED.containers['so-soc'].ips['soauth'] }}
    - binds:
+      - {{ CATRUST.dir }}
      - /nsm/rules:/nsm/rules:rw
      - /opt/so/conf/strelka:/opt/sensoroni/yara:rw
      - /opt/so/conf/sigma:/opt/sensoroni/sigma:rw
@@ -62,6 +64,11 @@ so-soc:
      - /nsm/backup/detections-migration:/nsm/backup/detections-migration:ro
      - /opt/so/state:/opt/so/state:rw
      - /etc/pki/tls/certs/intca.crt:/opt/sensoroni/html/so-ca.crt:ro
+      {% if DOCKERMERGED.containers['so-soc'].custom_bind_mounts %}
+        {% for BIND in DOCKERMERGED.containers['so-soc'].custom_bind_mounts %}
+      - {{ BIND }}
+        {% endfor %}
+      {% endif %}
    - extra_hosts:
    {% for node in DOCKER_EXTRA_HOSTS %}
    {%   for hostname, ip in node.items() %}
@@ -90,6 +97,7 @@ so-soc:
    {%   endfor %}
    {% endif %}
    - watch:
+      - cmd: ca_trust_extract
      - file: trusttheca
      - file: /opt/so/conf/soc/*
    - require:
@@ -370,6 +370,18 @@ soc:
            description: 'How often the ElastAlert integrity checker runs (in seconds). This verifies the integrity of deployed rules.'
            global: True
            advanced: True
+          partialSyncTimeoutSeconds:
+            description: 'How long to wait for Sigma rules to deploy after a detection is created, updated, or deleted. Must allow time for a scheduled rule sync already in progress to finish. Increase on systems with limited resources.'
+            global: True
+            advanced: True
+            forcedType: int
+            helpLink: sigma
+          bulkSyncTimeoutSeconds:
+            description: 'How long to wait for Sigma rules to deploy after a bulk enable, disable, or delete has saved its detections. Must allow time for a scheduled rule sync already in progress to finish. Increase on systems with limited resources.'
+            global: True
+            advanced: True
+            forcedType: int
+            helpLink: sigma
          rulesRepos:
            default: &eerulesRepos
              description: "Custom Git repositories to pull Sigma rules from. 'license' field is required, 'folder' is optional. 'community' disables some management options for the imported rules - they can't be deleted or edited, just tuned, duplicated and Enabled | Disabled. The new settings will be applied within 15 minutes. At that point, you will need to wait for the scheduled rule update to take place (by default, every 24 hours), or you can force the update by nagivating to Detections -->  Options dropdown menu --> Elastalert --> Full Update."
@@ -649,6 +661,18 @@ soc:
            description: 'How often the Strelka integrity checker runs (in seconds). This verifies the integrity of deployed rules.'
            global: True
            advanced: True
+          partialSyncTimeoutSeconds:
+            description: 'How long to wait for YARA rules to deploy after a detection is created, updated, or deleted. Must allow time for a scheduled rule sync already in progress to finish. Increase on systems with limited resources.'
+            global: True
+            advanced: True
+            forcedType: int
+            helpLink: yara
+          bulkSyncTimeoutSeconds:
+            description: 'How long to wait for YARA rules to deploy after a bulk enable, disable, or delete has saved its detections. Must allow time for a scheduled rule sync already in progress to finish. Increase on systems with limited resources.'
+            global: True
+            advanced: True
+            forcedType: int
+            helpLink: yara
          rulesRepos:
            default: &serulesRepos
              description: "Custom Git repositories to pull YARA rules from. 'license' field is required, 'folder' is optional. 'community' disables some management options for the imported rules - they can't be deleted or edited, just tuned, duplicated and Enabled | Disabled. The new settings will be applied within 15 minutes. At that point, you will need to wait for the scheduled rule update to take place (by default, every 24 hours), or you can force the update by nagivating to Detections -->  Options dropdown menu --> Strelka --> Full Update."
@@ -713,6 +737,18 @@ soc:
            description: 'How often the Suricata integrity checker runs (in seconds). This verifies the integrity of deployed rules.'
            global: True
            advanced: True
+          partialSyncTimeoutSeconds:
+            description: 'How long to wait for Suricata rules to deploy after a detection is created, updated, or deleted. Must allow time for a scheduled rule sync already in progress to finish. Increase on systems with limited resources.'
+            global: True
+            advanced: True
+            forcedType: int
+            helpLink: suricata
+          bulkSyncTimeoutSeconds:
+            description: 'How long to wait for Suricata rules to deploy after a bulk enable, disable, or delete has saved its detections. Must allow time for a scheduled rule sync already in progress to finish. Increase on systems with limited resources.'
+            global: True
+            advanced: True
+            forcedType: int
+            helpLink: suricata
          customRulesets:
            description: 'This setting is no longer used. Use Ruleset Sources setting instead.'
            global: True
@@ -7,8 +7,10 @@
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}

 include:
+  - ca.trust
  - strelka.backend.config
  - strelka.backend.sostatus

@@ -17,6 +19,7 @@ strelka_backend:
    - image: {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-strelka-backend:{{ GLOBALS.so_version }}
    - restart_policy: unless-stopped
    - binds:
+      - {{ CATRUST.dir }}
      - /opt/so/conf/strelka/backend/:/etc/strelka/:ro
      - /opt/so/conf/strelka/rules/compiled/:/etc/yara/:ro
      {% if DOCKERMERGED.containers['so-strelka-backend'].custom_bind_mounts %}
@@ -36,12 +39,14 @@ strelka_backend:
      - {{ XTRAHOST }}
        {% endfor %}
      {% endif %}
-    {% if DOCKERMERGED.containers['so-strelka-backend'].extra_env %}
    - environment:
-      {% for XTRAENV in DOCKERMERGED.containers['so-strelka-backend'].extra_env %}
+      - REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt
+      - SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt
+      {% if DOCKERMERGED.containers['so-strelka-backend'].extra_env %}
+        {% for XTRAENV in DOCKERMERGED.containers['so-strelka-backend'].extra_env %}
      - {{ XTRAENV }}
-      {% endfor %}
-    {% endif %}
+        {% endfor %}
+      {% endif %}
    {% if DOCKERMERGED.containers['so-strelka-backend'].ulimits %}
    - ulimits:
    {%   for ULIMIT in DOCKERMERGED.containers['so-strelka-backend'].ulimits %}
@@ -49,6 +54,7 @@ strelka_backend:
    {%   endfor %}
    {% endif %}
    - watch:
+      - cmd: ca_trust_extract
      - file: strelkasensorcompiledrules
      - file: backend_backend_config
      - file: backend_logging_config
@@ -7,8 +7,10 @@
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}

 include:
+  - ca.trust
  - strelka.coordinator.config
  - strelka.coordinator.sostatus

@@ -39,6 +41,7 @@ strelka_coordinator:
      {% endfor %}
    {% endif %}
    - binds:
+      - {{ CATRUST.pem }}:/etc/ssl/certs/ca-certificates.crt:ro
      - /nsm/strelka/coord-redis-data:/data:rw
      {% if DOCKERMERGED.containers['so-strelka-coordinator'].custom_bind_mounts %}
        {% for BIND in DOCKERMERGED.containers['so-strelka-coordinator'].custom_bind_mounts %}
@@ -51,6 +54,9 @@ strelka_coordinator:
      - {{ ULIMIT.name }}={{ ULIMIT.soft }}:{{ ULIMIT.hard }}
    {%   endfor %}
    {% endif %}
+    - watch:
+      - cmd: ca_trust_extract
+
 delete_so-strelka-coordinator_so-status.disabled:
  file.uncomment:
    - name: /opt/so/conf/so-status/so-status.conf
@@ -7,8 +7,10 @@
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}

 include:
+  - ca.trust
  - strelka.filestream.config
  - strelka.filestream.sostatus

@@ -17,6 +19,7 @@ strelka_filestream:
    - image: {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-strelka-manager:{{ GLOBALS.so_version }}
    - restart_policy: unless-stopped
    - binds:
+      - {{ CATRUST.pem }}:/etc/ssl/certs/ca-certificates.crt:ro
      - /opt/so/conf/strelka/filestream/:/etc/strelka/:ro
      - /nsm/strelka:/nsm/strelka
      {% if DOCKERMERGED.containers['so-strelka-filestream'].custom_bind_mounts %}
@@ -49,6 +52,7 @@ strelka_filestream:
    {%   endfor %}
    {% endif %}
    - watch:
+      - cmd: ca_trust_extract
      - file: filestream_config

 delete_so-strelka-filestream_so-status.disabled:
@@ -7,8 +7,10 @@
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}

 include:
+  - ca.trust
  - strelka.frontend.config
  - strelka.frontend.sostatus

@@ -17,6 +19,7 @@ strelka_frontend:
    - image: {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-strelka-manager:{{ GLOBALS.so_version }}
    - restart_policy: unless-stopped
    - binds:
+      - {{ CATRUST.pem }}:/etc/ssl/certs/ca-certificates.crt:ro
      - /opt/so/conf/strelka/frontend/:/etc/strelka/:ro
      - /nsm/strelka/log/:/var/log/strelka/:rw
      {% if DOCKERMERGED.containers['so-strelka-frontend'].custom_bind_mounts %}
@@ -54,6 +57,7 @@ strelka_frontend:
    {%   endfor %}
    {% endif %}
    - watch:
+      - cmd: ca_trust_extract
      - file: frontend_config

 delete_so-strelka-frontend_so-status.disabled:
@@ -7,8 +7,10 @@
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}

 include:
+  - ca.trust
  - strelka.gatekeeper.config
  - strelka.gatekeeper.sostatus

@@ -33,6 +35,7 @@ strelka_gatekeeper:
      - {{ BINDING }}
      {% endfor %}
    - binds:
+      - {{ CATRUST.pem }}:/etc/ssl/certs/ca-certificates.crt:ro
      - /nsm/strelka/gk-redis-data:/data:rw
      {% if DOCKERMERGED.containers['so-strelka-gatekeeper'].custom_bind_mounts %}
        {% for BIND in DOCKERMERGED.containers['so-strelka-gatekeeper'].custom_bind_mounts %}
@@ -51,6 +54,8 @@ strelka_gatekeeper:
      - {{ ULIMIT.name }}={{ ULIMIT.soft }}:{{ ULIMIT.hard }}
    {%   endfor %}
    {% endif %}
+    - watch:
+      - cmd: ca_trust_extract

 delete_so-strelka-gatekeeper_so-status.disabled:
  file.uncomment:
@@ -7,8 +7,10 @@
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}

 include:
+  - ca.trust
  - strelka.manager.config
  - strelka.manager.sostatus

@@ -17,6 +19,7 @@ strelka_manager:
    - image: {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-strelka-manager:{{ GLOBALS.so_version }}
    - restart_policy: unless-stopped
    - binds:
+      - {{ CATRUST.pem }}:/etc/ssl/certs/ca-certificates.crt:ro
      - /opt/so/conf/strelka/manager/:/etc/strelka/:ro
      {% if DOCKERMERGED.containers['so-strelka-manager'].custom_bind_mounts %}
        {% for BIND in DOCKERMERGED.containers['so-strelka-manager'].custom_bind_mounts %}
@@ -48,6 +51,7 @@ strelka_manager:
    {%   endfor %}
    {% endif %}
    - watch:
+      - cmd: ca_trust_extract
      - file: manager_config

 delete_so-strelka-manager_so-status.disabled:
@@ -6,11 +6,13 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'suricata/map.jinja' import SURICATAMERGED %}


 include:
+  - ca.trust
  - suricata.config
  - suricata.sostatus

@@ -33,6 +35,7 @@ so-suricata:
    {%   endfor %}
    {% endif %}
    - binds:
+      - {{ CATRUST.dir }}
      - /opt/so/conf/suricata/suricata.yaml:/etc/suricata/suricata.yaml:ro
      - /opt/so/conf/suricata/threshold.conf:/etc/suricata/threshold.conf:ro
      - /opt/so/conf/suricata/classification.config:/etc/suricata/classification.config:ro
@@ -55,6 +58,7 @@ so-suricata:
      {% endfor %}
    {% endif %}
    - watch:
+      - cmd: ca_trust_extract
      - file: suriconfig
      - file: surithresholding
      - file: /opt/so/conf/suricata/bpf
@@ -2,6 +2,10 @@
 {% if sls in allowed_states %}

 {% from 'vars/globals.map.jinja' import GLOBALS %}
+{% from 'ca/trust.map.jinja' import CATRUST %}
+
+include:
+  - ca.trust

 so-tcpreplay:
  docker_container.running:
@@ -13,7 +17,10 @@ so-tcpreplay:
    - interactive: True
    - tty: True
    - binds:
+      - {{ CATRUST.dir }}
      - /opt/so/samples:/opt/so/samples:ro
+    - watch:
+      - cmd: ca_trust_extract


 {% else %}
@@ -6,6 +6,7 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'telegraf/map.jinja' import TELEGRAFMERGED %}

@@ -33,6 +34,7 @@ so-telegraf:
    - network_mode: host
    - init: True
    - binds:
+      - {{ CATRUST.pem }}:/etc/ssl/certs/ca-certificates.crt:ro
      - /opt/so/log/telegraf:/var/log/telegraf:rw
      - /opt/so/conf/telegraf/etc/telegraf.conf:/etc/telegraf/telegraf.conf:ro
      - /opt/so/conf/telegraf/node_config.json:/etc/telegraf/node_config.json:ro
@@ -73,6 +75,7 @@ so-telegraf:
    {%   endfor %}
    {% endif %}
    - watch:
+      - cmd: ca_trust_extract
      - file: tgraf_sbin_jinja
      - file: lasthighstate_placeholder
      - file: trusttheca
@@ -6,10 +6,12 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}


 include:
+  - ca.trust
  - zeek.config
  - zeek.sostatus

@@ -30,6 +32,7 @@ so-zeek:
    {%   endfor %}
    {% endif %}
    - binds:
+      - {{ CATRUST.dir }}
      - /nsm/zeek/logs:/nsm/zeek/logs:rw
      - /nsm/zeek/spool:/nsm/zeek/spool:rw
      - /nsm/zeek/extracted:/nsm/zeek/extracted:rw
@@ -63,6 +66,7 @@ so-zeek:
      {% endfor %}
    {% endif %}
    - watch:
+      - cmd: ca_trust_extract
      - file: /opt/so/conf/zeek/local.zeek
      - file: /opt/so/conf/zeek/node.cfg
      - file: /opt/so/conf/zeek/networks.cfg
@@ -976,27 +976,50 @@ docker_seed_update() {
 	set_progress_str "$docker_seed_update_percent" "Downloading $name"
 }

+registry_tar_has_soversion() {
+	local tarfile=$1
+	local version=$2
+
+	[ -n "$version" ] || return 1
+	[ -f "$tarfile" ] || return 1
+	# grep -q in a pipe SIGPIPEs tar; under pipefail a match would look like failure.
+	grep -qF "/_manifests/tags/${version}/" < <(tar -tf "$tarfile" 2>/dev/null)
+}
+
 docker_seed_registry() {
 	local VERSION="$SOVERSION"
+	local REGISTRY_TAR=/nsm/docker-registry/docker/registry.tar

-	if [ -f /nsm/docker-registry/docker/registry.tar ]; then
-		logCmd "tar xvf /nsm/docker-registry/docker/registry.tar -C /nsm/docker-registry/docker"
-		logCmd "rm /nsm/docker-registry/docker/registry.tar"
+	if [ -f "$REGISTRY_TAR" ] && registry_tar_has_soversion "$REGISTRY_TAR" "$VERSION"; then
+		info "Extracting $REGISTRY_TAR"
+		if ! tar xvf "$REGISTRY_TAR" -C /nsm/docker-registry/docker >> "$setup_log" 2>&1; then
+			error "Failed to extract $REGISTRY_TAR"
+			return 1
+		fi
+		logCmd "rm -f $REGISTRY_TAR"
+		return 0
+	elif [ -f "$REGISTRY_TAR" ]; then
+		info "Not using $REGISTRY_TAR; it has no images tagged '${VERSION}'"
+		if [[ "$is_airgap" == true ]]; then
+			error "Airgap install: $REGISTRY_TAR does not match Security Onion version '${VERSION}'. Use the ISO for this version."
+			return 1
+		fi
 	elif [[ -d /nsm/docker-registry/docker/registry && ( -f /etc/SOCLOUD || "$is_airgap" == true ) ]]; then
 		echo "Using existing docker registry content"
-	else
-		if [ "$install_type" == 'IMPORT' ]; then
-			container_list 'so-import'	
-		else
-			container_list
-		fi
-
-		docker_seed_update_percent=25
-
-		update_docker_containers 'netinstall' '' 'docker_seed_update' '/dev/stdout' 2>&1 | tee -a "$setup_log"
-        # Use pipe exit status of 'update_docker_containers' for return code
-		return ${PIPESTATUS[0]}
+		return 0
 	fi
+
+	if [ "$install_type" == 'IMPORT' ]; then
+		container_list 'so-import'	
+	else
+		container_list
+	fi
+
+	docker_seed_update_percent=25
+
+	update_docker_containers 'netinstall' '' 'docker_seed_update' '/dev/stdout' 2>&1 | tee -a "$setup_log"
+        # Use pipe exit status of 'update_docker_containers' for return code
+	return ${PIPESTATUS[0]}
 }

 elasticfleet_pillar() {
Author	SHA1	Message	Date
Mike Reeves	1dfdae9ea3	Postgres: run pg_partman maintenance over the local socket and detect failed runs	2026-11-10 11:38:23 -05:00
Jorge Reyes	b7b00f8233	Merge pull request #16378 from Security-Onion-Solutions/reyesj2/fix-16369 surface ilm policy errors in so-elasticsearch-ilm-policy-load output	2026-11-05 14:05:29 -06:00
Jorge Reyes	45d2b29a42	Merge pull request #16374 from Security-Onion-Solutions/fix/skip-mismatched-iso-registry Skip leftover ISO registry.tar when tags do not match SOVERSION	2026-11-05 11:11:45 -06:00
Doug Burks	7b494439e7	Merge pull request #16377 from Security-Onion-Solutions/feature/desktop-so-crates Install securityonion-so-crates on Security Onion Desktop	2026-11-04 15:56:12 -05:00
Doug Burks	9a8dc097b2	Install securityonion-so-crates on Security Onion Desktop	2026-11-04 15:51:16 -05:00
reyesj2	274da18929	Surface ILM policy validation errors in highstate Fail 4xx ILM PUTs immediately (retry 429/5xx/connect), print the Elasticsearch reason in the salt summary, and retry the load on the next highstate via /opt/so/state/esilm.txt when the last run failed. Fixes https://github.com/Security-Onion-Solutions/sos/issues/277	2026-11-04 13:18:31 -06:00
reyesj2	99a8ad2d4b	Wait for Elasticsearch and retry ILM policy loads Require wait_for_so-elasticsearch, check ES is responsive before dispatch, and retry each policy PUT so soup no longer fails with empty output when ES is not ready yet. Fixes https://github.com/Security-Onion-Solutions/securityonion/issues/16369	2026-11-04 11:14:12 -06:00
Mike Reeves	bd9cdeadef	Merge pull request #16373 from Security-Onion-Solutions/foundry/mw1jl4boa6hshh Remove x86-64-v3 CPU pre-flight check from soup	2026-11-03 22:50:54 -05:00
Mike Reeves	bdf01a5a5a	Remove x86-64-v3 CPU check from soup	2026-11-03 22:13:58 -05:00
reyesj2	1849c74cea	Skip leftover ISO registry.tar when tags do not match SOVERSION Fixes #16372	2026-11-03 21:07:29 -06:00
Mike Reeves	021b069706	Merge pull request #16367 from Security-Onion-Solutions/TOoSmOotH-patch-1 Bump version from 3.4.0 to 3.5.0	2026-11-02 11:52:06 -05:00
Mike Reeves	824d674eec	Add version 3.5.0 to discussion template	2026-11-02 11:50:49 -05:00
Mike Reeves	d43791c63e	Bump version from 3.4.0 to 3.5.0	2026-11-02 11:50:08 -05:00
Mike Reeves	4dc7f2d97c	Merge pull request #16365 from Security-Onion-Solutions/3.4 3.4.0	2026-11-02 10:52:49 -05:00
Mike Reeves	092be7f222	3.4.0	2026-11-02 10:46:19 -05:00
Mike Reeves	340e7928f6	3.4.0	2026-10-30 11:36:47 -04:00
Jason Ertel	f5ff40bd20	Merge pull request #16361 from Security-Onion-Solutions/jertel/wip switch analyzer deps to py 314	2026-10-29 16:15:29 -04:00
coreyogburn	6794af6f68	Merge pull request #16358 from Security-Onion-Solutions/cogburn/sync-timeouts New Configurable Timeouts for Detection Engine Syncs	2026-10-28 11:34:02 -06:00
Corey Ogburn	691bef2cf4	New Configurable Timeouts for Detection Engine Syncs Each engine now has timeouts for how long to allow a sync to go on after detections are modified.	2026-10-28 10:48:03 -06:00
Mike Reeves	1bbaa213cb	Merge pull request #16354 from Security-Onion-Solutions/mreeves/remove-es-sokeys	2026-10-27 17:36:48 -04:00
Mike Reeves	5342765ca8	Merge pull request #16357 from Security-Onion-Solutions/mreeves/host-ca-trust Use the host CA trust store, with the grid CA, for all containers	2026-10-27 16:40:58 -04:00
Mike Reeves	02394acf69	Use the host CA trust store for sensor, strelka and idh containers Mount the host trust store into zeek, suricata, tcpreplay, the strelka containers and idh, and restart them when it changes. Point the Python requests clients in strelka-backend and idh at it instead of certifi. Remove the frozen /opt/so/conf/ca bundles from every node in ca.trust rather than only on the manager in soup.	2026-10-27 16:24:33 -04:00
Josh Patterson	d8f69f7457	Merge pull request #16356 from Security-Onion-Solutions/custombindmounts Add custom_bind_mounts to so-kafka and so-soc	2026-10-27 15:25:36 -04:00
Mike Reeves	11c8fe3093	Use the host CA trust store for all grid containers Anchor the grid CA and blocklist the eMudhra roots in the host trust store (new ca.trust state, also used by the desktop role), then mount the extracted store into each container instead of the frozen copies made by so-catrust. Containers restart when the host trust changes.	2026-10-27 15:08:21 -04:00
Mike Reeves	d6fecc761a	Remove unused elasticsearch.p12/sokeys keystore	2026-10-27 14:43:46 -04:00
Josh Patterson	40e3dafbee	Add custom_bind_mounts to so-kafka and so-soc Every container in docker/defaults.yaml exposes custom_bind_mounts via soc_docker.yaml, but kafka and soc enabled.sls never appended them to binds, so the setting was silently ignored for those two containers.	2026-10-27 14:06:58 -04:00
Jason Ertel	1fb7e1bc79	Merge pull request #16351 from Security-Onion-Solutions/jertel/wip fix false positive scan results	2026-10-26 15:50:37 -04:00
@@ -1 +1 @@
 .4.0
 .5.0