Surface ILM policy validation errors in highstate

Fail 4xx ILM PUTs immediately (retry 429/5xx/connect), print the
Elasticsearch reason in the salt summary, and retry the load on the
next highstate via /opt/so/state/esilm.txt when the last run failed.

Fixes https://github.com/Security-Onion-Solutions/sos/issues/277

.github/DISCUSSION_TEMPLATE/3-0.yml

@@ -14,6 +14,7 @@ body:
         - 3.2.0
         - 3.3.0
         - 3.4.0
+        - 3.5.0
         - Other (please provide detail below)
     validations:
       required: true

DOWNLOAD_AND_VERIFY_ISO.md

@@ -1,17 +1,17 @@
-### 3.3.0-20260911 ISO image released on 2026/09/11
+### 3.4.0-20261102 ISO image released on 2026/11/02
 
 
 ### Download and Verify
 
-3.3.0-20260911 ISO image:  
-https://download.securityonion.net/file/securityonion/securityonion-3.3.0-20260911.iso
+3.4.0-20261102 ISO image:  
+https://download.securityonion.net/file/securityonion/securityonion-3.4.0-20261102.iso
  
-MD5: 12B18433D3A2198A185892FF79CF638F  
-SHA1: 2B3C2E1FA7A78ED1F956E7EDCC12E32593C14EEE  
-SHA256: 0938C73B76CE30EC9E4394D312C79EA7CAC721B6818541697279A6221F7D870D  
+MD5: FB0380AE996E175D6CC7E7A35D8CC178  
+SHA1: D8960F1DA352710BCA209773BF0B3AE8791D0D1B  
+SHA256: DFC4C87668836852A328EC7A0B755C8C1511FF0E43D423DEE23047A95F676D99  
 
 Signature for ISO image:  
-https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.3.0-20260911.iso.sig
+https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.4.0-20261102.iso.sig
 
 Signing key:  
 https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/3/main/KEYS  
@@ -25,22 +25,22 @@ wget https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/3/
 
 Download the signature file for the ISO:  
 ```
-wget https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.3.0-20260911.iso.sig
+wget https://github.com/Security-Onion-Solutions/securityonion/raw/3/main/sigs/securityonion-3.4.0-20261102.iso.sig
 ```
 
 Download the ISO image:  
 ```
-wget https://download.securityonion.net/file/securityonion/securityonion-3.3.0-20260911.iso
+wget https://download.securityonion.net/file/securityonion/securityonion-3.4.0-20261102.iso
 ```
 
 Verify the downloaded ISO image using the signature file:  
 ```
-gpg --verify securityonion-3.3.0-20260911.iso.sig securityonion-3.3.0-20260911.iso
+gpg --verify securityonion-3.4.0-20261102.iso.sig securityonion-3.4.0-20261102.iso
 ```
 
 The output should show "Good signature" and the Primary key fingerprint should match what's shown below:
 ```
-gpg: Signature made Fri 11 Sep 2026 11:23:56 AM EDT using RSA key ID FE507013
+gpg: Signature made Sat 31 Oct 2026 09:27:50 AM EDT using RSA key ID FE507013
 gpg: Good signature from "Security Onion Solutions, LLC <info@securityonionsolutions.com>"
 gpg: WARNING: This key is not certified with a trusted signature!
 gpg:          There is no indication that the signature belongs to the owner.

VERSION

@@ -1 +1 @@
-3.4.0
+3.5.0

salt/ca/files/blocklist/emsign-ecc-root-ca-c3.pem

@@ -0,0 +1,14 @@
+-----BEGIN CERTIFICATE-----
+MIICKzCCAbGgAwIBAgIKe3G2gla4EnycqDAKBggqhkjOPQQDAzBaMQswCQYDVQQG
+EwJVUzETMBEGA1UECxMKZW1TaWduIFBLSTEUMBIGA1UEChMLZU11ZGhyYSBJbmMx
+IDAeBgNVBAMTF2VtU2lnbiBFQ0MgUm9vdCBDQSAtIEMzMB4XDTE4MDIxODE4MzAw
+MFoXDTQzMDIxODE4MzAwMFowWjELMAkGA1UEBhMCVVMxEzARBgNVBAsTCmVtU2ln
+biBQS0kxFDASBgNVBAoTC2VNdWRocmEgSW5jMSAwHgYDVQQDExdlbVNpZ24gRUND
+IFJvb3QgQ0EgLSBDMzB2MBAGByqGSM49AgEGBSuBBAAiA2IABP2lYa57JhAd6bci
+MK4G9IGzsUJxlTm801Ljr6/58pc1kjZGDoeVjbk5Wum739D+yAdBPLtVb4Ojavti
+sIGJAnB9SMVK4+kiVCJNk7tCDK93nCOmfddhEc5lx/h//vXyqaNCMEAwHQYDVR0O
+BBYEFPtaSNCAIEDyqOkAB2kZd6fmw/TPMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMB
+Af8EBTADAQH/MAoGCCqGSM49BAMDA2gAMGUCMQC02C8Cif22TGK6Q04ThHK1rt0c
+3ta13FaPWEBaLd4gTCKDypOofu4SQMfWh0/434UCMBwUZOR8loMRnLDRWmFLpg9J
+0wD8ofzkpf9/rdcw0Md3f76BB1UwUCAU9Vc4CqgxUQ==
+-----END CERTIFICATE-----

salt/ca/files/blocklist/emsign-ecc-root-ca-g3.pem

@@ -0,0 +1,15 @@
+-----BEGIN CERTIFICATE-----
+MIICTjCCAdOgAwIBAgIKPPYHqWhwDtqLhDAKBggqhkjOPQQDAzBrMQswCQYDVQQG
+EwJJTjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNo
+bm9sb2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0g
+RzMwHhcNMTgwMjE4MTgzMDAwWhcNNDMwMjE4MTgzMDAwWjBrMQswCQYDVQQGEwJJ
+TjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNobm9s
+b2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0gRzMw
+djAQBgcqhkjOPQIBBgUrgQQAIgNiAAQjpQy4LRL1KPOxst3iAhKAnjlfSU2fySU0
+WXTsuwYc58Byr+iuL+FBVIcUqEqy6HyC5ltqtdyzdc6LBtCGI79G1Y4PPwT01xyS
+fvalY8L1X44uT6EYGQIrMgqCZH0Wk9GjQjBAMB0GA1UdDgQWBBR8XQKEE9TMipuB
+zhccLikenEhjQjAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAKBggq
+hkjOPQQDAwNpADBmAjEAvvNhzwIQHWSVB7gYboiFBS+DCBeQyh+KTOgNG3qxrdWB
+CUfvO6wIBHxcmbHtRwfSAjEAnbpV/KlK6O3t5nYBQnvI+GDZjVGLVTv7jHvrZQnD
++JbNR6iC8hZVdyR+EhCVBCyj
+-----END CERTIFICATE-----

salt/ca/files/blocklist/emsign-root-ca-c1.pem

@@ -0,0 +1,21 @@
+-----BEGIN CERTIFICATE-----
+MIIDczCCAlugAwIBAgILAK7PALrEzzL4Q7IwDQYJKoZIhvcNAQELBQAwVjELMAkG
+A1UEBhMCVVMxEzARBgNVBAsTCmVtU2lnbiBQS0kxFDASBgNVBAoTC2VNdWRocmEg
+SW5jMRwwGgYDVQQDExNlbVNpZ24gUm9vdCBDQSAtIEMxMB4XDTE4MDIxODE4MzAw
+MFoXDTQzMDIxODE4MzAwMFowVjELMAkGA1UEBhMCVVMxEzARBgNVBAsTCmVtU2ln
+biBQS0kxFDASBgNVBAoTC2VNdWRocmEgSW5jMRwwGgYDVQQDExNlbVNpZ24gUm9v
+dCBDQSAtIEMxMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAz+upufGZ
+BczYKCFK83M0UYRWEPWgTywS4/oTmifQz/l5GnRfHXk5/Fv4cI7gklL35CX5VIPZ
+HdPIWoU/Xse2B+4+wM6ar6xWQio5JXDWv7V7Nq2s9nPczdcdioOl+yuQFTdrHCZH
+3DspVpNqs8FqOp099cGXOFgFixwR4+S0uF2FHYP+eF8LRWgYSKVGczQ7/g/IdrvH
+GPMF0Ybzhe3nudkyrVWIzqa2kbBPrH4VI5b2P/AgNBbeCsbEBEV5f6f9vtKppa+c
+xSMq9zwhbL2vj07FOrLzNBL834AaSaTUqZX3noleoomslMuoaJuvimUnzYnu3Yy1
+aylwQ6BpC+S5DwIDAQABo0IwQDAdBgNVHQ4EFgQU/qHgcB4qAzlSWkK+XJGFehiq
+TbUwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQEL
+BQADggEBAMJKVvoVIXsoounlHfv4LcQ5lkFMOycsxGwYFYDGrK9HWS8mC+M2sO87
+/kOXSTKZEhVb3xEp/6tT+LvBeA+snFOvV71ojD1pM/CjoCNjO2RnIkSt1XHLVip4
+kqNPEjE2NuLe/gDEo2APJ62gsIq1NnpSob0n9CAnYuhNlCQT5AoE6TyrLshDCUrG
+YQTlSTR+08TI9Q/Aqum6VF7zYytPT1DU/rl7mYw9wC68AivTxEDkigcxHpvOJpkT
++xHqmiIMERnHXhuBUDDIlhJu58tBf5E7oke3VIAb3ADMmpDqw8NQBmIMMMAVSKeo
+WXzhriKi4gp6D/piq1JM4fHfyr6DDUI=
+-----END CERTIFICATE-----

salt/ca/files/blocklist/emsign-root-ca-c2.pem

@@ -0,0 +1,32 @@
+-----BEGIN CERTIFICATE-----
+MIIFcjCCA1qgAwIBAgIKLwq3aw3LSq8nWDANBgkqhkiG9w0BAQwFADBWMQswCQYD
+VQQGEwJVUzETMBEGA1UECxMKZW1TaWduIFBLSTEUMBIGA1UEChMLZU11ZGhyYSBJ
+bmMxHDAaBgNVBAMTE2VtU2lnbiBSb290IENBIC0gQzIwHhcNMTgwMjE4MTgzMDAw
+WhcNNDMwMjE4MTgzMDAwWjBWMQswCQYDVQQGEwJVUzETMBEGA1UECxMKZW1TaWdu
+IFBLSTEUMBIGA1UEChMLZU11ZGhyYSBJbmMxHDAaBgNVBAMTE2VtU2lnbiBSb290
+IENBIC0gQzIwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCMfX1lA+Tb
+mh9YInmRgOW97IVx4LUJf2DRZfs837Jrml+py64aVnYgWO4t6C78fgjfS7jX+c4T
+inIzEquWcI+zi0fd4Sc8NDf7JONp27VWX0qwUYqzLDRCt+s7zpLcfx1ky0zVIJj6
+L06uPyK3kIr9+YAsrVj+39utm6e2MBQsRNstSI3fCQYAGvoQTQ8fULauTqNWaYAk
+NYFe6HUHHQPp2u1Ua00odMXiD5oRFxLcDnGAcE1I/9E9mLCdkggXijYUmico7+Xw
+ZeFoPhva6eIJ5p03Lt3Du5W3EcHR0cJmmY1pyeA36JaXKWRNM9IRjYMVNCcp4jhB
+2tIYiZ+LVk8bwQ9/1c23txmv3u97taZlV22NF4ttS1qq3J+MOp0oGULBzpKfRx0q
+GVqbPukQNGAjOLIN8KDNQNzbR1iAl2d8H+MSoicBo4Aid8TjLWcNv48oCWL53ZrF
+BMTDjaIA6frG1t4IpbnHadA7qCJJe2qpJN6n2eQKAUn6UiQDHPsSqNBlcUhQ4Y/0
+Y0mU5rghm2OB9rXQS1Fb1JRCfJMNnJIm5AUB2+2RWzq5Tgz7SbSho8NsZk0UbQnF
+xciqQ9uoVTAsK14Sk9oG8Q3zfsM08cdPoRb0WlIZklR6mKD7L8nH/zfGu8PIJv94
+GGB9RZ9U4A69r3ePmy8MvrzfNxHKtH6svwIDAQABo0IwQDAdBgNVHQ4EFgQUs/eK
+pNYPiABZ6FEXT9V+7IYigZ0wDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB
+Af8wDQYJKoZIhvcNAQEMBQADggIBADQlpiWM0cv2nZ0H5jVsBq0x2q62Q0LwqATs
+CFvyub7gxNCytRuoA8stmPOEu/lg8Igxj4FIjoyhIrWUVxyiLU7No4P+WjEUOwUT
+xIpkEOtvGUQ9fiOlcGHtIZDNBlZq7WpktXAxeV55RPPsor26p2FNAMRFfZQh0sLX
+hKgk8iulSSggqx8ezgPye63FaiYEi4c/dzRj3HOCnsZiwZZU02df5YpNFjxSwZvE
+41cjGpsrpWMfQFI2s53RbeXp47lSAxYE4NzjBFMe+EwFuEveBCJBEAH5rvYu3pi2
+orsJ424TqWEQV1tCsCkQz+Yq/Okal7yHAkKDeOXcP7oN4A+TdXc2pdqxuVCnBO0R
+mWz2JpGSSeJjiTk/OPwRsPNWtwG/KXL04o2ta3jiPpJuICVtWDAc9R3auBEgJl5r
+ShRmBdszG0LmzsHuZPCFSYC15RBDCOBsa8bDRJ8pBFU2Wi/CVXCACEuavgoveA4F
+a5bt38o0PWxsBP+MpocCdVtDMqzQhxy9IohKuXWAGresoIvKDg3xFk6rBOrjfVwJ
+elwi/xAisojHPJVQv9W1zVIoHp+EQg/4MQC21NbIX2RoioB+V3hK439b/w7deU8x
+2M8cl1OG0nPfbnARl5GPM7vJgi470jto4SeMg6HMAW3Egb56tQcNLwI9U8mZnNvR
+gUMrkAgL
+-----END CERTIFICATE-----

salt/ca/files/blocklist/emsign-root-ca-g1.pem

@@ -0,0 +1,22 @@
+-----BEGIN CERTIFICATE-----
+MIIDlDCCAnygAwIBAgIKMfXkYgxsWO3W2DANBgkqhkiG9w0BAQsFADBnMQswCQYD
+VQQGEwJJTjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBU
+ZWNobm9sb2dpZXMgTGltaXRlZDEcMBoGA1UEAxMTZW1TaWduIFJvb3QgQ0EgLSBH
+MTAeFw0xODAyMTgxODMwMDBaFw00MzAyMTgxODMwMDBaMGcxCzAJBgNVBAYTAklO
+MRMwEQYDVQQLEwplbVNpZ24gUEtJMSUwIwYDVQQKExxlTXVkaHJhIFRlY2hub2xv
+Z2llcyBMaW1pdGVkMRwwGgYDVQQDExNlbVNpZ24gUm9vdCBDQSAtIEcxMIIBIjAN
+BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAk0u76WaK7p1b1TST0Bsew+eeuGQz
+f2N4aLTNLnF115sgxk0pvLZoYIr3IZpWNVrzdr3YzZr/k1ZLpVkGoZM0Kd0WNHVO
+8oG0x5ZOrRkVUkr+PHB1cM2vK6sVmjM8qrOLqs1D/fXqcP/tzxE7lM5OMhbTI0Aq
+d7OvPAEsbO2ZLIvZTmmYsvePQbAyeGHWDV/D+qJAkh1cF+ZwPjXnorfCYuKrpDhM
+tTk1b+oDafo6VGiFbdbyL0NVHpENDtjVaqSW0RM8LHhQ6DqS0hdW5TUaQBw+jSzt
+Od9C4INBdN+jzcKGYEho42kLVACL5HZpIQ15TjQIXhTCzLG3rdd8cIrHhQIDAQAB
+o0IwQDAdBgNVHQ4EFgQU++8Nhp6w492pufEhF38+/PB3KxowDgYDVR0PAQH/BAQD
+AgEGMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAFn/8oz1h31x
+PaOfG1vR2vjTnGs2vZupYeveFix0PZ7mddrXuqe8QhfnPZHr5X3dPpzxz5KsbEjM
+wiI/aTvFthUvozXGaCocV685743QNcMYDHsAVhzNixl03r4PEuDQqqE/AjSxcM6d
+GNYIAwlG7mDgfrbESQRRfXBgvKqy/3lyeqYdPV8q+Mri/Tm3R7nrft8EI6/6nAYH
+6ftjk4BAtcZsCjEozgyfz7MjNYBBjWzEN3uBL4ChQEKF6dk4jeihU80Bv2noWgby
+RQuQ+q7hv53yrlc8pa6yVvSLZUDp/TGBLPQ5Cdjua6e0ph0VpZj3AYHYhX3zUVxx
+iN66zB+Afko=
+-----END CERTIFICATE-----

salt/ca/files/blocklist/emsign-root-ca-g2.pem

@@ -0,0 +1,32 @@
+-----BEGIN CERTIFICATE-----
+MIIFlTCCA32gAwIBAgILAIZNvw/jXtd9jtgwDQYJKoZIhvcNAQEMBQAwZzELMAkG
+A1UEBhMCSU4xEzARBgNVBAsTCmVtU2lnbiBQS0kxJTAjBgNVBAoTHGVNdWRocmEg
+VGVjaG5vbG9naWVzIExpbWl0ZWQxHDAaBgNVBAMTE2VtU2lnbiBSb290IENBIC0g
+RzIwHhcNMTgwMjE4MTgzMDAwWhcNNDMwMjE4MTgzMDAwWjBnMQswCQYDVQQGEwJJ
+TjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNobm9s
+b2dpZXMgTGltaXRlZDEcMBoGA1UEAxMTZW1TaWduIFJvb3QgQ0EgLSBHMjCCAiIw
+DQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMNwGIWW2kHfHK+sXTNwxF07K+IV
+ySTuyFM2r1v002wUfcdT+zs5OM5QbMYFFnedXQI6gCFLsjKrcaej48Zt37OyEb3i
+aPs7CsP4kAyTwzKH9aZe6gXYHrJq40/ZVMNcQVI2PcIp40B/SAN2gUZ+ZaUtIOvV
+jEx26/ebNaXRIsthlkOG/caB+QRwDw1tl7338Zlv0M2oTBUy4B3e7dGP5pgXH71M
+jqHPCoNo+xv9f0NTBT+hUDa8h8wUtcGQq9CDeJTpjWcD2bP2AMdVG6oVpMAUeUzo
+cCyglvtFdUMjggxBbw4qhau1HXPG8Ot9hwL7ZMi8tkTzrvUIxxb8G9LF/7kKeCE7
+tGZaVzDTnXuifl3msR4ErHsQ4P7lVu2AIjIAhrAXoedDidb7pMcf7TABdrYUT1Jo
+G/AiK+J9jO6GTjeADD4LMDSBZhHMuBK/PJ/g0kGBt+/C1L+/HURzQhJkMlRnM6Rv
+XoCtfKopSlns5trZmTi971Wjbn88QXP61lGpBCUPwCjs7rpOYvSUJtI+lcbF+37q
+kIqOXYkVT3cupDSpw+H89kFtj5GKY+Xny4LxY+3IvDIRiyd6ky1DPj713DI0yqve
+EpsIr3A0PdwuyUI7CS1jg0NnGFT6Xxyr0xB+VDt83FJYW8v16k2pbaQ4kVxA3aXd
+X9dZYyVR1S59KM75AgMBAAGjQjBAMB0GA1UdDgQWBBTt7E1FYRgo57MjKBEcTaUn
+DV7s9DAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0B
+AQwFAAOCAgEACFC/ilQg8KTCVBxFJW/sazomkS0kNYbEIZg4B3obqwsJ7SX98z8Z
+gfzBpz0nYClwwJjWbFN1R2zY8pCEot6/dgmA8Vbq0GxhwPM5YN/SZquNyRIxO3cU
+dlAcwf+vSezdVCf9wOzvSAF3q0a5ljvbdbNJNpfScQVp7UUd5sBsZk8jXO1KQ/go
+/Vf/GDPnrIFmxpAIGE3sgnO8lAv9FzUaAeuv7HWe47xN9J7+bQzF93yHuIXACPTL
+pQHhg2zMv5C7BAbuDHfbj1Cu294Z832yhSfBcziWGskOvl3es2EcHytbS9c9P+0z
+Mpka7zGC1FHrvLb/FoduH86TeZt0QjZ6pcplNzoaxDnDvzTJ6CC2Eny+qH/APFCu
+VUv5/wjwF+HPm8Pup2ARj9cEp92+0qcerfHacNq5hMeGZdbA/dzdUR/5z5zXdxAk
+nl8mcfGb0eMNSTXQmmB/i4AecNnr72uYjzlaXUGYN7Nrb6XouG0pnh0/BBtWWp0U
+ShIPpWEAqs7RJBj6+1ZUYXZ4ObrCw962DxhN2p19Hxw9LtuUUcLqqTPrFXYvwO4t
+ouj7KJnAkaTUfXGdEaFVtFig1EA30WzJY2X1vAQ7hVnniCjgaXAGqjsU6sklNM9n
+xDx5rFCCCEtj9Kh8UHjGK2QqgP5kwgttjOApQMaCoezMfK4KD7WpOXU=
+-----END CERTIFICATE-----

salt/ca/files/blocklist/emsign-root-cs-ca-g2.pem

@@ -0,0 +1,32 @@
+-----BEGIN CERTIFICATE-----
+MIIFdjCCA16gAwIBAgIQCIYhdXKnxAzDuy7l1y1ubDANBgkqhkiG9w0BAQwFADBV
+MQswCQYDVQQGEwJJTjElMCMGA1UECgwcZU11ZGhyYSBUZWNobm9sb2dpZXMgTGlt
+aXRlZDEfMB0GA1UEAwwWZW1TaWduIFJvb3QgQ1MgQ0EgLSBHMjAeFw0yNDAyMDgw
+ODE1MzlaFw00OTAyMDcwODE1MzlaMFUxCzAJBgNVBAYTAklOMSUwIwYDVQQKDBxl
+TXVkaHJhIFRlY2hub2xvZ2llcyBMaW1pdGVkMR8wHQYDVQQDDBZlbVNpZ24gUm9v
+dCBDUyBDQSAtIEcyMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA4CVO
+xuBjdLJXpkrjosA+7ygTGaYzufUJrhQOeIZEuCqeCPI8hDqIu+MIqEqOlgDFetVM
+3Cor3CGBlGJzfzzlI31vyElvQq88jjCfo12Ghqcf+NAZDFduJk5+NgC0z3lfdiB6
+NNzxbfVkA+X6oDNNRioVUZsVFuIzT9Qt0Sh4KwQPRFBc/wI2yGpDuKHNsLehcECx
+Uw00m33xGYh9UkeL3reOmAaLYWZiO+Pd2vxPtrkXI4wyN4BMlESJ73uP0tyX9lq4
+M4yN2yh3i+ETI8/jzPjxHD7QFaVkCoS+uj2YmBa3mytlqlBgyOoKueNeotX8hfzB
+FNm962abpV4mN0SoC6hjkYv8IayseRPFv9VI0meQXg9OLRoNvBOqG5kw937FH0Li
+q3CIzKyvdyeYRT0LJdWA2iuB9l+4zzIfCwGyT4M4VDmJuPBhrTmjXKae6AUtm1eO
+q2f8gftA6J6AlbD0gRd/t8CrFssFNm+LYAEd36kWRBuolbaOfbSbEww84xNAgj4U
+SPwYjtNDsCkQjtgSOaDtDOmolN+Mbk528Y+LHMlbd4T3Xmp+VIWp5+70GhBDOA91
+hhtims+hHkXjRI2U7WrJLcTlcF8ykiymRbFKhwOBvTmtZNbHmDI7scU4wWUAsmW6
+tWaeeoM50KBl8ai28A6yBGYMO//03Ubvqiqst80CAwEAAaNCMEAwHQYDVR0OBBYE
+FOYcxqontIvyvYVkX6/0unKm0AF1MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8E
+BTADAQH/MA0GCSqGSIb3DQEBDAUAA4ICAQCv5Asord7fdQMT30emwb+oLuDUxdmJ
+DQHnzxIXoNvymDTbNHF9LqjeXc+X+XcjIV8PTp4o2I9wyVppHmcaVgRjN/G9FxeN
+5tCQ97v5sdx4eAdPljK1xqg7Vhfmzcp+7rSjJG7Sn9gMlZx81kL3Oa2qZj+W5Fi6
+tN4vaR9OLEyvKKHDG/eFxS/qghD1d/Clkb7UGGjBcWJlaZTWVALMKG+QCxiMoELi
+oeKaoBbZoXooxfF0mWdkIl9RWNU4r9O/yt66LBE9rj49dX19L+rGg71/4bTi5ua+
++Wz60XaOc2CdHJaBSols5hWEx/LBjS7UzfU3uJfnpQXVDi0US5sFzIHIlS6kjh7h
+YuF4+/9ZAl4Bx3w2TNN7pVm7I39pgYl7XN6KZRwfkAv4aRswPqhWbQfr3+qfcC93
+cgTyFy2RG5YE/OfmpSs6VtYKawqsdEODz/p2nhZfSDrVDnEPvERqahttY5+gaWNc
+2fbibgKMfuejTZaJ3yttPslVSko5SDs0xKU423swu34sb/fdISqtOu4NzPCZiHSG
+ladaoITmSA6UWFk4fdqaLBI+xSxY5Et3S1APp5r3j/l1KrWPNbb1/+ydHN+YLbh8
+hCBIecKdk6VbnnFRlulm0d2RSgpfCRrWLomN8JHCxCMVv/hBQXz3AXGM1ZlfYTyl
+E1BpDby0G2eEkw==
+-----END CERTIFICATE-----

salt/ca/init.sls

@@ -10,3 +10,4 @@ include:
   - ca.server
 {% endif %}
   - ca.trustca
+  - ca.trust

salt/ca/trust.map.jinja

@@ -0,0 +1,12 @@
+# Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
+# or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at 
+# https://securityonion.net/license; you may not use this file except in compliance with the
+# Elastic License 2.0.
+
+{# Containers mount the extracted directory rather than single files: update-ca-trust replaces
+   files by rename, so a file bind mount would keep pointing at the old inode. #}
+{% set CATRUST = {
+    'dir': '/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro',
+    'java': '/etc/pki/ca-trust/extracted/java/cacerts',
+    'pem': '/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'
+} %}

salt/ca/trust.sls

@@ -0,0 +1,37 @@
+# Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
+# or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at 
+# https://securityonion.net/license; you may not use this file except in compliance with the
+# Elastic License 2.0.
+
+ca_trust_anchor:
+  file.managed:
+    - name: /etc/pki/ca-trust/source/anchors/so-intca.crt
+    - source: salt://ca/files/ca.crt
+    - show_changes: False
+    - makedirs: True
+
+ca_trust_legacy_anchor:
+  file.absent:
+    - name: /etc/pki/ca-trust/source/anchors/ca.crt
+
+ca_trust_legacy_bundles:
+  file.absent:
+    - name: /opt/so/conf/ca
+
+# clean removes blocklist entries that are no longer shipped
+ca_trust_blocklist:
+  file.recurse:
+    - name: /etc/pki/ca-trust/source/blocklist
+    - source: salt://ca/files/blocklist
+    - clean: True
+    - makedirs: True
+    - file_mode: '0644'
+    - dir_mode: '0755'
+
+ca_trust_extract:
+  cmd.run:
+    - name: update-ca-trust extract
+    - onchanges:
+      - file: ca_trust_anchor
+      - file: ca_trust_legacy_anchor
+      - file: ca_trust_blocklist

salt/desktop/trusted-ca.sls

@@ -1,18 +1,10 @@
- {% from 'vars/globals.map.jinja' import GLOBALS %}
+{% from 'vars/globals.map.jinja' import GLOBALS %}
 
 {# we only want this state to run it is CentOS #}
 {% if GLOBALS.os == 'OEL' %}
 
-trusted_ca:
-  file.managed:
-    - name: /etc/pki/ca-trust/source/anchors/ca.crt
-    - source: salt://ca/files/ca.crt
-
-update_ca_certs:
-  cmd.run:
-    - name: update-ca-trust
-    - onchanges:
-      - file: trusted_ca
+include:
+  - ca.trust
 
 {% else %}
 

salt/elastalert/enabled.sls

@@ -6,9 +6,11 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 
 include:
+  - ca.trust
   - elastalert.config
   - elastalert.sostatus
 
@@ -28,6 +30,7 @@ so-elastalert:
         - ipv4_address: {{ DOCKERMERGED.containers['so-elastalert'].ip }}
     - detach: True
     - binds:
+      - {{ CATRUST.dir }}
       - /opt/so/rules/elastalert:/opt/elastalert/rules/:ro
       - /opt/so/log/elastalert:/var/log/elastalert:rw
       - /opt/so/conf/elastalert/modules/:/opt/elastalert/modules/:ro
@@ -46,8 +49,9 @@ so-elastalert:
       - {{ XTRAHOST }}
         {% endfor %}
       {% endif %}
-      {% if DOCKERMERGED.containers['so-elastalert'].extra_env %}
     - environment:
+      - REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt
+      {% if DOCKERMERGED.containers['so-elastalert'].extra_env %}
         {% for XTRAENV in DOCKERMERGED.containers['so-elastalert'].extra_env %}
       - {{ XTRAENV }}
         {% endfor %}
@@ -65,6 +69,7 @@ so-elastalert:
       - file: elastacustmodulesdir
       - file: elastaconf
     - watch:
+      - cmd: ca_trust_extract
       - file: elastaconf
     - onlyif:
       - "so-elasticsearch-query / | jq -r '.version.number[0:1]' | grep -q 9" {# only run this state if elasticsearch is version 9 #}

salt/elastic-fleet-package-registry/enabled.sls

@@ -6,9 +6,11 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 
 include:
+  - ca.trust
   - elastic-fleet-package-registry.config
   - elastic-fleet-package-registry.sostatus
 
@@ -34,8 +36,9 @@ so-elastic-fleet-package-registry:
       {% for BINDING in DOCKERMERGED.containers['so-elastic-fleet-package-registry'].port_bindings %}
       - {{ BINDING }}
       {% endfor %}
-    {% if DOCKERMERGED.containers['so-elastic-fleet-package-registry'].custom_bind_mounts %}
     - binds:
+      - {{ CATRUST.pem }}:/etc/ssl/certs/ca-certificates.crt:ro
+    {% if DOCKERMERGED.containers['so-elastic-fleet-package-registry'].custom_bind_mounts %}
         {% for BIND in DOCKERMERGED.containers['so-elastic-fleet-package-registry'].custom_bind_mounts %}
       - {{ BIND }}
         {% endfor %}
@@ -52,6 +55,8 @@ so-elastic-fleet-package-registry:
       - {{ ULIMIT.name }}={{ ULIMIT.soft }}:{{ ULIMIT.hard }}
     {%   endfor %}
     {% endif %}
+    - watch:
+      - cmd: ca_trust_extract
 
 wait_for_so-elastic-fleet-package-registry:
   http.wait_for_successful_query:

salt/elasticagent/enabled.sls

@@ -6,6 +6,7 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 
 include:
@@ -37,6 +38,7 @@ so-elastic-agent:
       - {{ BINDING }}
       {% endfor %}
     - binds:
+      - {{ CATRUST.dir }}
       - /opt/so/conf/elastic-agent/elastic-agent.yml:/usr/share/elastic-agent/elastic-agent.yml:ro
       - /opt/so/log/elasticagent:/usr/share/elastic-agent/logs
       - /etc/pki/tls/certs/intca.crt:/etc/pki/tls/certs/intca.crt:ro 
@@ -65,6 +67,7 @@ so-elastic-agent:
       - file: create-elastic-agent-config
       - file: trusttheca
     - watch:
+      - cmd: ca_trust_extract
       - file: create-elastic-agent-config
       - file: trusttheca
 

salt/elasticfleet/enabled.sls

@@ -6,6 +6,7 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'elasticfleet/map.jinja' import ELASTICFLEETMERGED %}
 
@@ -63,6 +64,7 @@ so-elastic-fleet:
       - {{ BINDING }}
       {% endfor %}
     - binds:
+      - {{ CATRUST.dir }}
       - /etc/pki/elasticfleet-server.crt:/etc/pki/elasticfleet-server.crt:ro
       - /etc/pki/elasticfleet-server.key:/etc/pki/elasticfleet-server.key:ro
       - /etc/pki/tls/certs/intca.crt:/etc/pki/tls/certs/intca.crt:ro
@@ -95,6 +97,7 @@ so-elastic-fleet:
     {%   endfor %}
     {% endif %}
     - watch:
+      - cmd: ca_trust_extract
       - file: trusttheca
       - x509: etc_elasticfleet_key
       - x509: etc_elasticfleet_crt

salt/elasticsearch/ca.sls

@@ -1,49 +0,0 @@
-# Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
-# or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at 
-# https://securityonion.net/license; you may not use this file except in compliance with the
-# Elastic License 2.0.
-
-{% from 'allowed_states.map.jinja' import allowed_states %}
-{% if sls.split('.')[0] in allowed_states or sls in allowed_states %}
-{%   from 'vars/globals.map.jinja' import GLOBALS %}
-
-# Move our new CA over so Elastic and Logstash can use SSL with the internal CA
-catrustdir:
-  file.directory:
-    - name: /opt/so/conf/ca
-    - user: 939
-    - group: 939
-    - makedirs: True
-
-{%   if GLOBALS.is_manager %}
-# We have to add the Manager CA to the CA list
-catrustscript:
-  cmd.script:
-    - source: salt://elasticsearch/tools/sbin_jinja/so-catrust
-    - template: jinja
-    - cwd: /opt/so
-    - defaults:
-        GLOBALS: {{ GLOBALS }}
-{%   endif %}
-
-elasticsearch_cacerts:
-  file.managed:
-    - name: /opt/so/conf/ca/cacerts
-    - source: salt://elasticsearch/cacerts
-    - user: 939
-    - group: 939
-
-elasticsearch_capems:
-  file.managed:
-    - name: /opt/so/conf/ca/tls-ca-bundle.pem
-    - source: salt://elasticsearch/tls-ca-bundle.pem
-    - user: 939
-    - group: 939
-
-{% else %}
-
-{{sls}}_state_not_allowed:
-  test.fail_without_changes:
-    - name: {{sls}}_state_not_allowed
-
-{% endif %}

salt/elasticsearch/cluster.sls

@@ -107,6 +107,12 @@ so-elasticsearch-system-indices-patch:
       - file: so-elasticsearch-system-indices-patch-script
 {%  endif %}
 
+so-elasticsearch-ilm-policy-reload:
+  file.absent:
+    - name: /opt/so/state/esilm.txt
+    - onchanges:
+      - file: so-elasticsearch-ilm-policy-load-script
+
 # heavynodes will only load ILM policies for SO managed indices. (Indicies defined in elasticsearch/defaults.yaml)
 so-elasticsearch-ilm-policy-load:
   cmd.run:
@@ -115,8 +121,9 @@ so-elasticsearch-ilm-policy-load:
     - require:
       - docker_container: so-elasticsearch
       - file: so-elasticsearch-ilm-policy-load-script
-    - onchanges:
-      - file: so-elasticsearch-ilm-policy-load-script
+      - http: wait_for_so-elasticsearch
+      - file: so-elasticsearch-ilm-policy-reload
+    - unless: test -f /opt/so/state/esilm.txt
 
 so-elasticsearch-templates-reload:
   file.absent:

salt/elasticsearch/enabled.sls

@@ -6,6 +6,7 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'elasticsearch/config.map.jinja' import ELASTICSEARCH_NODES %}
 {%   from 'elasticsearch/config.map.jinja' import ELASTICSEARCH_SEED_HOSTS %}
@@ -13,7 +14,6 @@
 
 include:
   - ca
-  - elasticsearch.ca
   - elasticsearch.ssl
   - elasticsearch.config
   - elasticsearch.sostatus
@@ -63,15 +63,14 @@ so-elasticsearch:
       - {{ BINDING }}
       {% endfor %}
     - binds:
+      - {{ CATRUST.dir }}
       - /opt/so/conf/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
       - /opt/so/conf/elasticsearch/log4j2.properties:/usr/share/elasticsearch/config/log4j2.properties:ro
       - /nsm/elasticsearch:/usr/share/elasticsearch/data:rw
       - /opt/so/log/elasticsearch:/var/log/elasticsearch:rw
-      - /opt/so/conf/ca/cacerts:/usr/share/elasticsearch/jdk/lib/security/cacerts:ro
       - /etc/pki/tls/certs/intca.crt:/usr/share/elasticsearch/config/ca.crt:ro
       - /etc/pki/elasticsearch.crt:/usr/share/elasticsearch/config/elasticsearch.crt:ro
       - /etc/pki/elasticsearch.key:/usr/share/elasticsearch/config/elasticsearch.key:ro
-      - /etc/pki/elasticsearch.p12:/usr/share/elasticsearch/config/elasticsearch.p12:ro
       - /opt/so/conf/elasticsearch/users_roles:/usr/share/elasticsearch/config/users_roles:ro
       - /opt/so/conf/elasticsearch/users:/usr/share/elasticsearch/config/users:ro
       {% if ELASTICSEARCHMERGED.config.path.get('repo', False) %}
@@ -85,21 +84,19 @@ so-elasticsearch:
         {% endfor %}
       {% endif %}
     - watch:
+      - cmd: ca_trust_extract
       - file: trusttheca
       - x509: elasticsearch_crt
       - x509: elasticsearch_key
-      - file: elasticsearch_cacerts
       - file: esyml
     - require:
       - file: trusttheca
       - x509: elasticsearch_crt
       - x509: elasticsearch_key
-      - file: elasticsearch_cacerts
       - file: esyml
       - file: eslog4jfile
       - file: nsmesdir
       - file: eslogdir
-      - file: elasticp12perms
       - cmd: auth_users_roles_inode
       - cmd: auth_users_inode
 

salt/elasticsearch/ssl.sls

@@ -38,10 +38,6 @@ elasticsearch_crt:
     - retry:
         attempts: 5
         interval: 30
-  cmd.run:
-    - name: "/usr/bin/openssl pkcs12 -inkey /etc/pki/elasticsearch.key -in /etc/pki/elasticsearch.crt -export -out /etc/pki/elasticsearch.p12 -nodes -passout pass:"
-    - onchanges:
-      - x509: /etc/pki/elasticsearch.key
 
 elastickeyperms:
   file.managed:
@@ -50,13 +46,6 @@ elastickeyperms:
     - mode: 640
     - group: 930
     
-elasticp12perms:
-  file.managed:
-    - replace: False
-    - name: /etc/pki/elasticsearch.p12
-    - mode: 640
-    - group: 930
-
 {% else %}
 
 {{sls}}_state_not_allowed:

salt/elasticsearch/tools/sbin_jinja/so-catrust

@@ -1,26 +0,0 @@
-#!/bin/bash
-#
-# Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
-# or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at 
-# https://securityonion.net/license; you may not use this file except in compliance with the
-# Elastic License 2.0.
-
-. /usr/sbin/so-common
-get_elastic_agent_vars
-
-# Exit on errors, since all lines must succeed
-set -e
-
-# Check to see if we have extracted the ca cert.
-if [ ! -f /opt/so/saltstack/local/salt/elasticsearch/cacerts ]; then
-    docker run -v /etc/pki/ca.crt:/etc/ssl/ca.crt --name so-elasticsearchca --user root --entrypoint jdk/bin/keytool {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-elasticsearch:$ELASTIC_AGENT_TARBALL_VERSION -keystore /usr/share/elasticsearch/jdk/lib/security/cacerts -alias SOSCA -import -file /etc/ssl/ca.crt -storepass changeit -noprompt
-    # Make sure symbolic links are followed when copying from container
-    docker cp -L so-elasticsearchca:/usr/share/elasticsearch/jdk/lib/security/cacerts /opt/so/saltstack/local/salt/elasticsearch/cacerts
-    docker cp -L so-elasticsearchca:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem /opt/so/saltstack/local/salt/elasticsearch/tls-ca-bundle.pem
-    docker rm so-elasticsearchca
-    echo "" >> /opt/so/saltstack/local/salt/elasticsearch/tls-ca-bundle.pem
-    echo "sosca" >> /opt/so/saltstack/local/salt/elasticsearch/tls-ca-bundle.pem
-    cat /etc/pki/ca.crt >> /opt/so/saltstack/local/salt/elasticsearch/tls-ca-bundle.pem
-else
-    exit 0
-fi

salt/elasticsearch/tools/sbin_jinja/so-elasticsearch-ilm-policy-load

@@ -7,11 +7,17 @@
 . /usr/sbin/so-common
 
 MAX_JOBS=${MAX_ILM_JOBS:-10}
+ILM_STATEFILE=/opt/so/state/esilm.txt
+rm -f "$ILM_STATEFILE"
 
+check_elasticsearch_responsive
+
+ILM_TMP_DIR=$(mktemp -d)
+trap 'rm -rf "$ILM_TMP_DIR"' EXIT
 # Lock used to serialize block writes so concurrent jobs never interleave their output.
-ILM_OUTPUT_LOCK=$(mktemp)
-ILM_FAIL_FILE=$(mktemp)
-trap 'rm -f "$ILM_OUTPUT_LOCK" "$ILM_FAIL_FILE"' EXIT
+ILM_OUTPUT_LOCK="$ILM_TMP_DIR/output.lock"
+ILM_FAIL_FILE="$ILM_TMP_DIR/failed"
+touch "$ILM_FAIL_FILE"
 
 # Policies are loaded concurrently (up to MAX_JOBS at a time) for speed. Each policy's block is
 # printed the moment its curl returns, so output appears in COMPLETION ORDER, not the order
@@ -20,13 +26,30 @@ echo "Loading ILM policies concurrently; output below appears in completion orde
 echo
 
 put_policy() {
-  local desc="$1" policyname="$2" data="$3" result rc=0
-  if ! result=$(curl -K /opt/so/conf/elasticsearch/curl.config -s -k -L --fail \
-    -X PUT "https://localhost:9200/_ilm/policy/${policyname}" \
-    -H 'Content-Type: application/json' -d"${data}" 2>&1); then
-    rc=1
-  elif ! jq -e '.acknowledged == true' <<<"$result" >/dev/null 2>&1; then
-    rc=1
+  local desc="$1" policyname="$2" data="$3" rc=1 attempt out http_code body result reason tmpdata
+  tmpdata=$(mktemp "$ILM_TMP_DIR/${policyname}.XXXXXX.json")
+  printf '%s' "$data" >"$tmpdata"
+
+  for attempt in 1 2 3; do
+    out=$(so-elasticsearch-query "_ilm/policy/${policyname}" -d@"${tmpdata}" -XPUT -w '\n%{http_code}' 2>/dev/null)
+    http_code=${out##*$'\n'} body=${out%$'\n'*}
+
+    if [[ "$http_code" == 200 ]] && jq -e '.acknowledged == true' <<<"$body" >/dev/null 2>&1; then
+      rc=0
+      break
+    fi
+    # 429 = ES rejected/throttled (may succeed later); other 4xx will not
+    [[ "$http_code" == 4?? && "$http_code" != 429 ]] && break
+    (( attempt < 3 )) && sleep 3
+  done
+
+  if (( rc == 0 )); then
+    result='{"acknowledged":true}'
+  else
+    result=${body:-"HTTP ${http_code:-000}"}
+    # flatten tabs/newlines so the TAB fail-file stays one record
+    reason=$(jq -r '(if (.error|type) == "object" then (.error.reason // .error.type) elif (.error|type) == "string" then .error else empty end) // empty | tostring | gsub("[\t\r\n]+"; " ")' <<<"$body" 2>/dev/null)
+    reason="HTTP ${http_code:-000}${reason:+: ${reason}}"
   fi
 
   # curl above ran in parallel; serialize just this block write so concurrent jobs never interleave.
@@ -34,7 +57,7 @@ put_policy() {
     flock 200
     printf 'Setting up %s policy...\n%s\n\n' "${desc}" "${result}"
     if (( rc != 0 )); then
-      printf '%s\n' "${policyname}" >>"$ILM_FAIL_FILE"
+      printf '%s\t%s\n' "${policyname}" "${reason}" >>"$ILM_FAIL_FILE"
     fi
   } 200>>"${ILM_OUTPUT_LOCK}"
 
@@ -82,10 +105,15 @@ wait || true
 
 if [[ -s "$ILM_FAIL_FILE" ]]; then
   echo "ERROR: Failed to load ILM policy(s):"
-  while read -r POLICY; do
-    echo " - $POLICY"
+  while IFS=$'\t' read -r POLICY REASON; do
+    if [[ -n "$REASON" ]]; then
+      echo " - ${POLICY}: ${REASON}"
+    else
+      echo " - ${POLICY}"
+    fi
   done < "$ILM_FAIL_FILE"
   exit 1
 else
   echo "Successfully loaded all ILM policies."
+  touch "$ILM_STATEFILE"
 fi

salt/hydra/enabled.sls

@@ -13,9 +13,11 @@
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   if 'api' in salt['pillar.get']('features', []) %}
 
 include:
+  - ca.trust
   - hydra.config
   - hydra.sostatus
 
@@ -29,6 +31,7 @@ so-hydra:
       - soauth:
         - ipv4_address: {{ DOCKERMERGED.containers['so-hydra'].ips['soauth'] }}
     - binds:
+      - {{ CATRUST.dir }}
       - /opt/so/conf/hydra/:/hydra-conf:ro
       - /opt/so/log/hydra/:/hydra-log:rw
       - /nsm/hydra/db:/hydra-data:rw
@@ -60,6 +63,7 @@ so-hydra:
     {%   endfor %}
     {% endif %}
     - watch:
+      - cmd: ca_trust_extract
       - file: hydraconfig
     - require:
       - file: hydraconfig

salt/idh/enabled.sls

@@ -6,9 +6,11 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 
 include:
+  - ca.trust
   - idh.config
   - idh.sostatus
 
@@ -20,6 +22,7 @@ so-idh:
     - detach: True
     - network_mode: host
     - binds:
+      - {{ CATRUST.dir }}
       - /nsm/idh:/var/tmp:rw
       - /opt/so/conf/idh/http-skins:/opt/opencanary/http-skins:ro
       - /opt/so/conf/idh/opencanary.conf:/etc/opencanaryd/opencanary.conf:ro
@@ -34,12 +37,13 @@ so-idh:
       - {{ XTRAHOST }}
       {% endfor %}
     {% endif %}
-    {% if DOCKERMERGED.containers['so-idh'].extra_env %}
     - environment:
-      {% for XTRAENV in DOCKERMERGED.containers['so-idh'].extra_env %}
+      - REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt
+      {% if DOCKERMERGED.containers['so-idh'].extra_env %}
+        {% for XTRAENV in DOCKERMERGED.containers['so-idh'].extra_env %}
       - {{ XTRAENV }}
-      {% endfor %}
-    {% endif %}
+        {% endfor %}
+      {% endif %}
     {% if DOCKERMERGED.containers['so-idh'].ulimits %}
     - ulimits:
     {%   for ULIMIT in DOCKERMERGED.containers['so-idh'].ulimits %}
@@ -47,6 +51,7 @@ so-idh:
     {%   endfor %}
     {% endif %}
     - watch:
+      - cmd: ca_trust_extract
       - file: opencanary_config
     - require:
       - file: opencanary_config

salt/influxdb/enabled.sls

@@ -6,11 +6,13 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   set PASSWORD = salt['pillar.get']('secrets:influx_pass') %}
 {%   set TOKEN = salt['pillar.get']('influxdb:token') %}
 
 include:
+  - ca.trust
   - influxdb.ssl
   - influxdb.config
   - influxdb.sostatus
@@ -38,6 +40,7 @@ so-influxdb:
         {% endfor %}
       {% endif %}
     - binds:
+      - {{ CATRUST.pem }}:/etc/ssl/certs/ca-certificates.crt:ro
       - /opt/so/log/influxdb/:/log:rw
       - /opt/so/conf/influxdb/config.yaml:/conf/config.yaml:ro
       - /opt/so/conf/influxdb/etc:/etc/influxdb2:rw
@@ -66,6 +69,7 @@ so-influxdb:
     {%   endfor %}
     {% endif %}
     - watch:
+      - cmd: ca_trust_extract
       - file: influxdbconf
       - x509: influxdb_key
       - x509: influxdb_crt

salt/kafka/enabled.sls

@@ -12,12 +12,14 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   set KAFKANODES = salt['pillar.get']('kafka:nodes') %}
 {%   set KAFKA_EXTERNAL_ACCESS = salt['pillar.get']('kafka:config:external_access:enabled', default=False) %}
 {%   if 'gmd' in salt['pillar.get']('features', []) %}
 
 include:
+  - ca.trust
   - kafka.ca
   - kafka.config
   - kafka.ssl
@@ -51,6 +53,7 @@ so-kafka:
       - {{ BINDING }}
       {% endfor %}
     - binds:
+      - {{ CATRUST.java }}:/opt/java/openjdk/lib/security/cacerts:ro
       - /etc/pki/kafka.p12:/etc/pki/kafka.p12:ro
       - /opt/so/conf/kafka/kafka-truststore.jks:/etc/pki/kafka-truststore.jks:ro
       - /nsm/kafka/data/:/nsm/kafka/data/:rw
@@ -61,6 +64,11 @@ so-kafka:
       {% if KAFKA_EXTERNAL_ACCESS %}
       - /opt/so/conf/kafka/kafka_server_jaas.conf:/opt/kafka/config/kafka_server_jaas.conf:ro
       {% endif %}
+      {% if DOCKERMERGED.containers['so-kafka'].custom_bind_mounts %}
+        {% for BIND in DOCKERMERGED.containers['so-kafka'].custom_bind_mounts %}
+      - {{ BIND }}
+        {% endfor %}
+      {% endif %}
     {% if DOCKERMERGED.containers['so-kafka'].ulimits %}
     - ulimits:
     {%   for ULIMIT in DOCKERMERGED.containers['so-kafka'].ulimits %}
@@ -68,6 +76,7 @@ so-kafka:
     {%   endfor %}
     {% endif %}
     - watch:
+      - cmd: ca_trust_extract
       {% for sc in ['server', 'client'] %}
       - file: kafka_kraft_{{sc}}_properties
       {% endfor %}

salt/kibana/enabled.sls

@@ -8,8 +8,10 @@
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'elasticsearch/config.map.jinja' import ELASTICSEARCHMERGED %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 
 include:
+  - ca.trust
   - kibana.config
   - kibana.sostatus
 
@@ -27,6 +29,7 @@ so-kibana:
       - ELASTICSEARCH_HOST={{ GLOBALS.manager }}
       - ELASTICSEARCH_PORT=9200
       - MANAGER={{ GLOBALS.manager }}
+      - NODE_OPTIONS=--use-openssl-ca
       {% if DOCKERMERGED.containers['so-kibana'].extra_env %}
         {% for XTRAENV in DOCKERMERGED.containers['so-kibana'].extra_env %}
       - {{ XTRAENV }}
@@ -40,6 +43,7 @@ so-kibana:
       {% endfor %}
     {% endif %}
     - binds:
+      - {{ CATRUST.dir }}
       - /opt/so/conf/kibana/etc:/usr/share/kibana/config:rw
       - /opt/so/log/kibana:/var/log/kibana:rw
       - /opt/so/conf/kibana/customdashboards:/usr/share/kibana/custdashboards:ro
@@ -60,6 +64,7 @@ so-kibana:
     {%   endfor %}
     {% endif %}
     - watch:
+      - cmd: ca_trust_extract
       - file: kibanaconfig
 
 wait_for_so-kibana:

salt/kratos/enabled.sls

@@ -7,8 +7,10 @@
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 
 include:
+  - ca.trust
   - kratos.config
   - kratos.sostatus
 
@@ -22,6 +24,7 @@ so-kratos:
       - soauth:
         - ipv4_address: {{ DOCKERMERGED.containers['so-kratos'].ips['soauth'] }}
     - binds:
+      - {{ CATRUST.dir }}
       - /opt/so/conf/kratos/:/kratos-conf:ro
       - /opt/so/log/kratos/:/kratos-log:rw
       - /nsm/kratos/db:/kratos-data:rw
@@ -53,6 +56,7 @@ so-kratos:
     {%   endfor %}
     {% endif %}
     - watch:
+      - cmd: ca_trust_extract
       - file: kratosschema
       - file: kratosconfig
     - require:

salt/logstash/enabled.sls

@@ -6,6 +6,7 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'logstash/map.jinja' import LOGSTASH_MERGED %}
 {%   from 'logstash/map.jinja' import LOGSTASH_NODES %}
@@ -13,9 +14,6 @@
 
 include:
   - ca
-{%   if GLOBALS.role not in ['so-receiver','so-fleet'] %}
-  - elasticsearch.ca
-{%   endif %}
 {# Kafka ca runs on nodes that can run logstash for Kafka input / output. Only when Kafka is global pipeline #}
 {%   if GLOBALS.role in ['so-searchnode', 'so-manager', 'so-managerhype', 'so-managersearch', 'so-receiver', 'so-standalone'] and GLOBALS.pipeline == 'KAFKA' %}
   - kafka.ca
@@ -58,6 +56,8 @@ so-logstash:
       - {{ BINDING }}
       {% endfor %}
     - binds:
+      - {{ CATRUST.dir }}
+      - {{ CATRUST.java }}:/usr/share/logstash/jdk/lib/security/cacerts:ro
       - /opt/so/conf/elasticsearch/templates/:/templates/:ro
       - /opt/so/conf/logstash/etc/:/usr/share/logstash/config/:ro
       - /opt/so/conf/logstash/pipelines:/usr/share/logstash/pipelines:ro
@@ -78,10 +78,6 @@ so-logstash:
       - /etc/pki/filebeat.p8:/usr/share/logstash/filebeat.key:ro
           {% endif %}
       {% endif %}
-      {% if GLOBALS.role not in ['so-receiver','so-fleet'] %}
-      - /opt/so/conf/ca/cacerts:/etc/pki/ca-trust/extracted/java/cacerts:ro
-      - /opt/so/conf/ca/tls-ca-bundle.pem:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:ro
-      {% endif %}
       {% if GLOBALS.pipeline == "KAFKA" and GLOBALS.role in ['so-manager', 'so-managerhype', 'so-managersearch', 'so-standalone', 'so-searchnode'] %}
       - /etc/pki/kafka-logstash.p12:/usr/share/logstash/kafka-logstash.p12:ro
       - /opt/so/conf/kafka/kafka-truststore.jks:/etc/pki/kafka-truststore.jks:ro
@@ -104,14 +100,11 @@ so-logstash:
     {%   endfor %}
     {% endif %}
     - watch:
+      - cmd: ca_trust_extract
       - file: lsetcsync
       - file: lslog4j2
       - file: lspipelinesyml
       - file: trusttheca
-      {% if GLOBALS.is_manager %}
-      - file: elasticsearch_cacerts
-      - file: elasticsearch_capems
-      {% endif %}
       {% if GLOBALS.is_manager or GLOBALS.role in ['so-fleet', 'so-heavynode', 'so-receiver'] %}
       - x509: etc_elasticfleet_logstash_crt
       - x509: etc_elasticfleet_logstash_key
@@ -133,10 +126,6 @@ so-logstash:
       {% endif %}
     - require:
       - file: trusttheca
-      {% if GLOBALS.is_manager %}
-      - file: elasticsearch_cacerts
-      - file: elasticsearch_capems
-      {% endif %}
       {% if GLOBALS.is_manager or GLOBALS.role in ['so-fleet', 'so-heavynode', 'so-receiver'] %}
       - x509: etc_elasticfleet_logstash_crt
       - x509: etc_elasticfleet_logstash_key

salt/manager/tools/sbin/soup

@@ -1261,9 +1261,15 @@ post_to_3.4.0() {
 
   add_detection_runtime_fields
 
+  remove_frozen_ca_bundles
+
   set_postversion 3.4.0
 }
 
+remove_frozen_ca_bundles() {
+  rm -f /opt/so/saltstack/local/salt/elasticsearch/cacerts /opt/so/saltstack/local/salt/elasticsearch/tls-ca-bundle.pem >> $SOUP_LOG 2>&1
+}
+
 add_detection_runtime_fields() {
   local template=$DEFAULT_SALT_DIR/salt/elasticsearch/templates/component/so/detection-mappings.json
   local runtime

salt/nginx/enabled.sls

@@ -6,10 +6,12 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'nginx/map.jinja' import NGINXMERGED %}
 
 include:
+  - ca.trust
   - nginx.ssl
   - nginx.config
   - nginx.sostatus
@@ -47,6 +49,7 @@ so-nginx:
       {% endfor %}
     {% endif %}
     - binds:
+      - {{ CATRUST.pem }}:/etc/ssl/certs/ca-certificates.crt:ro
       - /opt/so/conf/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
       - /opt/so/log/nginx/:/var/log/nginx:rw
       - /opt/so/tmp/nginx/:/var/lib/nginx:rw
@@ -89,6 +92,7 @@ so-nginx:
       - {{ BINDING }}
       {% endfor %}
     - watch:
+      - cmd: ca_trust_extract
       - file: nginxconf
       - file: nginxconfdir
       {% if GLOBALS.is_manager %}

salt/postgres/enabled.sls

@@ -6,10 +6,12 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   set SO_POSTGRES_USER = salt['pillar.get']('postgres:auth:users:so_postgres_user:user', 'so_postgres') %}
 
 include:
+  - ca.trust
   - postgres.auth
   - postgres.ssl
   - postgres.config
@@ -42,6 +44,7 @@ so-postgres:
         {% endfor %}
       {% endif %}
     - binds:
+      - {{ CATRUST.dir }}
       - /opt/so/log/postgres/:/log:rw
       - /nsm/postgres:/var/lib/postgresql/data:rw
       - /opt/so/conf/postgres/postgresql.conf:/conf/postgresql.conf:ro
@@ -69,6 +72,7 @@ so-postgres:
     {%   endfor %}
     {% endif %}
     - watch:
+      - cmd: ca_trust_extract
       - file: postgresconf
       - file: postgreshba
       - file: postgresinitdb

salt/redis/enabled.sls

@@ -7,6 +7,7 @@
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 
 include:
   - ca
@@ -28,6 +29,7 @@ so-redis:
       - {{ BINDING }}
       {% endfor %}
     - binds:
+      - {{ CATRUST.pem }}:/etc/ssl/certs/ca-certificates.crt:ro
       - /opt/so/log/redis:/var/log/redis:rw
       - /opt/so/conf/redis/etc/redis.conf:/usr/local/etc/redis/redis.conf:ro
       - /opt/so/conf/redis/working:/redis:rw
@@ -60,6 +62,7 @@ so-redis:
     {% endif %}
     - entrypoint: "redis-server /usr/local/etc/redis/redis.conf"
     - watch:
+      - cmd: ca_trust_extract
       - file: trusttheca
       - x509: redis_crt
       - x509: redis_key

salt/registry/enabled.sls

@@ -6,9 +6,11 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 
 include:
+  - ca.trust
   - registry.ssl
   - registry.config
   - registry.sostatus
@@ -29,6 +31,7 @@ so-dockerregistry:
       - {{ BINDING }}
       {% endfor %}
     - binds:
+      - {{ CATRUST.pem }}:/etc/ssl/certs/ca-certificates.crt:ro
       - /opt/so/conf/docker-registry/etc/config.yml:/etc/distribution/config.yml:ro
       - /opt/so/conf/docker-registry:/var/lib/registry:rw
       - /nsm/docker-registry/docker:/var/lib/registry/docker:rw
@@ -63,6 +66,7 @@ so-dockerregistry:
         attempts: 5
         interval: 30
     - watch:
+      - cmd: ca_trust_extract
       - x509: registry_crt
       - x509: registry_key
     - require:

salt/sensoroni/enabled.sls

@@ -4,10 +4,12 @@
 # Elastic License 2.0.
 
 {% from 'vars/globals.map.jinja' import GLOBALS %}
+{% from 'ca/trust.map.jinja' import CATRUST %}
 {% from 'docker/docker.map.jinja' import DOCKERMERGED %}
 
 
 include:
+  - ca.trust
   - sensoroni.config
   - sensoroni.sostatus
 
@@ -17,6 +19,7 @@ so-sensoroni:
     - restart_policy: unless-stopped
     - network_mode: host
     - binds:
+      - {{ CATRUST.dir }}
       - /nsm/import:/nsm/import:rw
       - /nsm/pcapout:/nsm/pcapout:rw
       - /opt/so/conf/sensoroni/sensoroni.json:/opt/sensoroni/sensoroni.json:ro
@@ -48,6 +51,7 @@ so-sensoroni:
     {%   endfor %}
     {% endif %}
     - watch:
+      - cmd: ca_trust_extract
       - file: /opt/so/conf/sensoroni/sensoroni.json
     - require:
       - file: sensoroniagentconf

salt/soc/defaults.yaml

@@ -1441,6 +1441,8 @@ soc:
           rulesFingerprintFile: /opt/sensoroni/fingerprints/sigma.fingerprint
           stateFilePath: /opt/sensoroni/fingerprints/elastalertengine.state
           integrityCheckFrequencySeconds: 1200
+          partialSyncTimeoutSeconds: 300
+          bulkSyncTimeoutSeconds: 1800
           rulesRepos:
             default:
               - repo: https://github.com/Security-Onion-Solutions/securityonion-resources
@@ -1700,6 +1702,8 @@ soc:
           yaraRulesFolder: /opt/sensoroni/yara/rules
           stateFilePath: /opt/sensoroni/fingerprints/strelkaengine.state
           integrityCheckFrequencySeconds: 1200
+          partialSyncTimeoutSeconds: 300
+          bulkSyncTimeoutSeconds: 1800
         suricataengine:
           aiRepoUrl: https://github.com/Security-Onion-Solutions/securityonion-resources
           aiRepoBranch: generated-summaries-published
@@ -1715,6 +1719,8 @@ soc:
           rulesFingerprintFile: /opt/sensoroni/fingerprints/emerging-all.fingerprint
           stateFilePath: /opt/sensoroni/fingerprints/suricataengine.state
           integrityCheckFrequencySeconds: 1200
+          partialSyncTimeoutSeconds: 300
+          bulkSyncTimeoutSeconds: 1800
           ignoredSidRanges:
             - '1100000-1101000'
           rulesetSources:

salt/soc/enabled.sls

@@ -6,6 +6,7 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'soc/merged.map.jinja' import DOCKER_EXTRA_HOSTS %}
 {%   from 'soc/merged.map.jinja' import SOCMERGED %}
@@ -27,6 +28,7 @@ so-soc:
       - soauth:
         - ipv4_address: {{ DOCKERMERGED.containers['so-soc'].ips['soauth'] }}
     - binds:
+      - {{ CATRUST.dir }}
       - /nsm/rules:/nsm/rules:rw
       - /opt/so/conf/strelka:/opt/sensoroni/yara:rw
       - /opt/so/conf/sigma:/opt/sensoroni/sigma:rw
@@ -62,6 +64,11 @@ so-soc:
       - /nsm/backup/detections-migration:/nsm/backup/detections-migration:ro
       - /opt/so/state:/opt/so/state:rw
       - /etc/pki/tls/certs/intca.crt:/opt/sensoroni/html/so-ca.crt:ro
+      {% if DOCKERMERGED.containers['so-soc'].custom_bind_mounts %}
+        {% for BIND in DOCKERMERGED.containers['so-soc'].custom_bind_mounts %}
+      - {{ BIND }}
+        {% endfor %}
+      {% endif %}
     - extra_hosts:
     {% for node in DOCKER_EXTRA_HOSTS %}
     {%   for hostname, ip in node.items() %}
@@ -90,6 +97,7 @@ so-soc:
     {%   endfor %}
     {% endif %}
     - watch:
+      - cmd: ca_trust_extract
       - file: trusttheca
       - file: /opt/so/conf/soc/*
     - require:

salt/soc/soc_soc.yaml

@@ -370,6 +370,18 @@ soc:
             description: 'How often the ElastAlert integrity checker runs (in seconds). This verifies the integrity of deployed rules.'
             global: True
             advanced: True
+          partialSyncTimeoutSeconds:
+            description: 'How long to wait for Sigma rules to deploy after a detection is created, updated, or deleted. Must allow time for a scheduled rule sync already in progress to finish. Increase on systems with limited resources.'
+            global: True
+            advanced: True
+            forcedType: int
+            helpLink: sigma
+          bulkSyncTimeoutSeconds:
+            description: 'How long to wait for Sigma rules to deploy after a bulk enable, disable, or delete has saved its detections. Must allow time for a scheduled rule sync already in progress to finish. Increase on systems with limited resources.'
+            global: True
+            advanced: True
+            forcedType: int
+            helpLink: sigma
           rulesRepos:
             default: &eerulesRepos
               description: "Custom Git repositories to pull Sigma rules from. 'license' field is required, 'folder' is optional. 'community' disables some management options for the imported rules - they can't be deleted or edited, just tuned, duplicated and Enabled | Disabled. The new settings will be applied within 15 minutes. At that point, you will need to wait for the scheduled rule update to take place (by default, every 24 hours), or you can force the update by nagivating to Detections -->  Options dropdown menu --> Elastalert --> Full Update."
@@ -649,6 +661,18 @@ soc:
             description: 'How often the Strelka integrity checker runs (in seconds). This verifies the integrity of deployed rules.'
             global: True
             advanced: True
+          partialSyncTimeoutSeconds:
+            description: 'How long to wait for YARA rules to deploy after a detection is created, updated, or deleted. Must allow time for a scheduled rule sync already in progress to finish. Increase on systems with limited resources.'
+            global: True
+            advanced: True
+            forcedType: int
+            helpLink: yara
+          bulkSyncTimeoutSeconds:
+            description: 'How long to wait for YARA rules to deploy after a bulk enable, disable, or delete has saved its detections. Must allow time for a scheduled rule sync already in progress to finish. Increase on systems with limited resources.'
+            global: True
+            advanced: True
+            forcedType: int
+            helpLink: yara
           rulesRepos:
             default: &serulesRepos
               description: "Custom Git repositories to pull YARA rules from. 'license' field is required, 'folder' is optional. 'community' disables some management options for the imported rules - they can't be deleted or edited, just tuned, duplicated and Enabled | Disabled. The new settings will be applied within 15 minutes. At that point, you will need to wait for the scheduled rule update to take place (by default, every 24 hours), or you can force the update by nagivating to Detections -->  Options dropdown menu --> Strelka --> Full Update."
@@ -713,6 +737,18 @@ soc:
             description: 'How often the Suricata integrity checker runs (in seconds). This verifies the integrity of deployed rules.'
             global: True
             advanced: True
+          partialSyncTimeoutSeconds:
+            description: 'How long to wait for Suricata rules to deploy after a detection is created, updated, or deleted. Must allow time for a scheduled rule sync already in progress to finish. Increase on systems with limited resources.'
+            global: True
+            advanced: True
+            forcedType: int
+            helpLink: suricata
+          bulkSyncTimeoutSeconds:
+            description: 'How long to wait for Suricata rules to deploy after a bulk enable, disable, or delete has saved its detections. Must allow time for a scheduled rule sync already in progress to finish. Increase on systems with limited resources.'
+            global: True
+            advanced: True
+            forcedType: int
+            helpLink: suricata
           customRulesets:
             description: 'This setting is no longer used. Use Ruleset Sources setting instead.'
             global: True

salt/strelka/backend/enabled.sls

@@ -7,8 +7,10 @@
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 
 include:
+  - ca.trust
   - strelka.backend.config
   - strelka.backend.sostatus
 
@@ -17,6 +19,7 @@ strelka_backend:
     - image: {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-strelka-backend:{{ GLOBALS.so_version }}
     - restart_policy: unless-stopped
     - binds:
+      - {{ CATRUST.dir }}
       - /opt/so/conf/strelka/backend/:/etc/strelka/:ro
       - /opt/so/conf/strelka/rules/compiled/:/etc/yara/:ro
       {% if DOCKERMERGED.containers['so-strelka-backend'].custom_bind_mounts %}
@@ -36,12 +39,14 @@ strelka_backend:
       - {{ XTRAHOST }}
         {% endfor %}
       {% endif %}
-    {% if DOCKERMERGED.containers['so-strelka-backend'].extra_env %}
     - environment:
-      {% for XTRAENV in DOCKERMERGED.containers['so-strelka-backend'].extra_env %}
+      - REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt
+      - SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt
+      {% if DOCKERMERGED.containers['so-strelka-backend'].extra_env %}
+        {% for XTRAENV in DOCKERMERGED.containers['so-strelka-backend'].extra_env %}
       - {{ XTRAENV }}
-      {% endfor %}
-    {% endif %}
+        {% endfor %}
+      {% endif %}
     {% if DOCKERMERGED.containers['so-strelka-backend'].ulimits %}
     - ulimits:
     {%   for ULIMIT in DOCKERMERGED.containers['so-strelka-backend'].ulimits %}
@@ -49,6 +54,7 @@ strelka_backend:
     {%   endfor %}
     {% endif %}
     - watch:
+      - cmd: ca_trust_extract
       - file: strelkasensorcompiledrules
       - file: backend_backend_config
       - file: backend_logging_config

salt/strelka/coordinator/enabled.sls

@@ -7,8 +7,10 @@
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 
 include:
+  - ca.trust
   - strelka.coordinator.config
   - strelka.coordinator.sostatus
 
@@ -39,6 +41,7 @@ strelka_coordinator:
       {% endfor %}
     {% endif %}
     - binds:
+      - {{ CATRUST.pem }}:/etc/ssl/certs/ca-certificates.crt:ro
       - /nsm/strelka/coord-redis-data:/data:rw
       {% if DOCKERMERGED.containers['so-strelka-coordinator'].custom_bind_mounts %}
         {% for BIND in DOCKERMERGED.containers['so-strelka-coordinator'].custom_bind_mounts %}
@@ -51,6 +54,9 @@ strelka_coordinator:
       - {{ ULIMIT.name }}={{ ULIMIT.soft }}:{{ ULIMIT.hard }}
     {%   endfor %}
     {% endif %}
+    - watch:
+      - cmd: ca_trust_extract
+
 delete_so-strelka-coordinator_so-status.disabled:
   file.uncomment:
     - name: /opt/so/conf/so-status/so-status.conf

salt/strelka/filestream/enabled.sls

@@ -7,8 +7,10 @@
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 
 include:
+  - ca.trust
   - strelka.filestream.config
   - strelka.filestream.sostatus
 
@@ -17,6 +19,7 @@ strelka_filestream:
     - image: {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-strelka-manager:{{ GLOBALS.so_version }}
     - restart_policy: unless-stopped
     - binds:
+      - {{ CATRUST.pem }}:/etc/ssl/certs/ca-certificates.crt:ro
       - /opt/so/conf/strelka/filestream/:/etc/strelka/:ro
       - /nsm/strelka:/nsm/strelka
       {% if DOCKERMERGED.containers['so-strelka-filestream'].custom_bind_mounts %}
@@ -49,6 +52,7 @@ strelka_filestream:
     {%   endfor %}
     {% endif %}
     - watch:
+      - cmd: ca_trust_extract
       - file: filestream_config
 
 delete_so-strelka-filestream_so-status.disabled:

salt/strelka/frontend/enabled.sls

@@ -7,8 +7,10 @@
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 
 include:
+  - ca.trust
   - strelka.frontend.config
   - strelka.frontend.sostatus
 
@@ -17,6 +19,7 @@ strelka_frontend:
     - image: {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-strelka-manager:{{ GLOBALS.so_version }}
     - restart_policy: unless-stopped
     - binds:
+      - {{ CATRUST.pem }}:/etc/ssl/certs/ca-certificates.crt:ro
       - /opt/so/conf/strelka/frontend/:/etc/strelka/:ro
       - /nsm/strelka/log/:/var/log/strelka/:rw
       {% if DOCKERMERGED.containers['so-strelka-frontend'].custom_bind_mounts %}
@@ -54,6 +57,7 @@ strelka_frontend:
     {%   endfor %}
     {% endif %}
     - watch:
+      - cmd: ca_trust_extract
       - file: frontend_config
 
 delete_so-strelka-frontend_so-status.disabled:

salt/strelka/gatekeeper/enabled.sls

@@ -7,8 +7,10 @@
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 
 include:
+  - ca.trust
   - strelka.gatekeeper.config
   - strelka.gatekeeper.sostatus
 
@@ -33,6 +35,7 @@ strelka_gatekeeper:
       - {{ BINDING }}
       {% endfor %}
     - binds:
+      - {{ CATRUST.pem }}:/etc/ssl/certs/ca-certificates.crt:ro
       - /nsm/strelka/gk-redis-data:/data:rw
       {% if DOCKERMERGED.containers['so-strelka-gatekeeper'].custom_bind_mounts %}
         {% for BIND in DOCKERMERGED.containers['so-strelka-gatekeeper'].custom_bind_mounts %}
@@ -51,6 +54,8 @@ strelka_gatekeeper:
       - {{ ULIMIT.name }}={{ ULIMIT.soft }}:{{ ULIMIT.hard }}
     {%   endfor %}
     {% endif %}
+    - watch:
+      - cmd: ca_trust_extract
 
 delete_so-strelka-gatekeeper_so-status.disabled:
   file.uncomment:

salt/strelka/manager/enabled.sls

@@ -7,8 +7,10 @@
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 
 include:
+  - ca.trust
   - strelka.manager.config
   - strelka.manager.sostatus
 
@@ -17,6 +19,7 @@ strelka_manager:
     - image: {{ GLOBALS.registry_host }}:5000/{{ GLOBALS.image_repo }}/so-strelka-manager:{{ GLOBALS.so_version }}
     - restart_policy: unless-stopped
     - binds:
+      - {{ CATRUST.pem }}:/etc/ssl/certs/ca-certificates.crt:ro
       - /opt/so/conf/strelka/manager/:/etc/strelka/:ro
       {% if DOCKERMERGED.containers['so-strelka-manager'].custom_bind_mounts %}
         {% for BIND in DOCKERMERGED.containers['so-strelka-manager'].custom_bind_mounts %}
@@ -48,6 +51,7 @@ strelka_manager:
     {%   endfor %}
     {% endif %}
     - watch:
+      - cmd: ca_trust_extract
       - file: manager_config
 
 delete_so-strelka-manager_so-status.disabled:

salt/suricata/enabled.sls

@@ -6,11 +6,13 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'suricata/map.jinja' import SURICATAMERGED %}
 
 
 include:
+  - ca.trust
   - suricata.config
   - suricata.sostatus
 
@@ -33,6 +35,7 @@ so-suricata:
     {%   endfor %}
     {% endif %}
     - binds:
+      - {{ CATRUST.dir }}
       - /opt/so/conf/suricata/suricata.yaml:/etc/suricata/suricata.yaml:ro
       - /opt/so/conf/suricata/threshold.conf:/etc/suricata/threshold.conf:ro
       - /opt/so/conf/suricata/classification.config:/etc/suricata/classification.config:ro
@@ -55,6 +58,7 @@ so-suricata:
       {% endfor %}
     {% endif %}
     - watch:
+      - cmd: ca_trust_extract
       - file: suriconfig
       - file: surithresholding
       - file: /opt/so/conf/suricata/bpf

salt/tcpreplay/init.sls

@@ -2,6 +2,10 @@
 {% if sls in allowed_states %}
 
 {% from 'vars/globals.map.jinja' import GLOBALS %}
+{% from 'ca/trust.map.jinja' import CATRUST %}
+
+include:
+  - ca.trust
 
 so-tcpreplay:
   docker_container.running:
@@ -13,7 +17,10 @@ so-tcpreplay:
     - interactive: True
     - tty: True
     - binds:
+      - {{ CATRUST.dir }}
       - /opt/so/samples:/opt/so/samples:ro
+    - watch:
+      - cmd: ca_trust_extract
 
 
 {% else %}

salt/telegraf/enabled.sls

@@ -6,6 +6,7 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 {%   from 'telegraf/map.jinja' import TELEGRAFMERGED %}
 
@@ -33,6 +34,7 @@ so-telegraf:
     - network_mode: host
     - init: True
     - binds:
+      - {{ CATRUST.pem }}:/etc/ssl/certs/ca-certificates.crt:ro
       - /opt/so/log/telegraf:/var/log/telegraf:rw
       - /opt/so/conf/telegraf/etc/telegraf.conf:/etc/telegraf/telegraf.conf:ro
       - /opt/so/conf/telegraf/node_config.json:/etc/telegraf/node_config.json:ro
@@ -73,6 +75,7 @@ so-telegraf:
     {%   endfor %}
     {% endif %}
     - watch:
+      - cmd: ca_trust_extract
       - file: tgraf_sbin_jinja
       - file: lasthighstate_placeholder
       - file: trusttheca

salt/zeek/enabled.sls

@@ -6,10 +6,12 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls.split('.')[0] in allowed_states %}
 {%   from 'vars/globals.map.jinja' import GLOBALS %}
+{%   from 'ca/trust.map.jinja' import CATRUST %}
 {%   from 'docker/docker.map.jinja' import DOCKERMERGED %}
 
 
 include:
+  - ca.trust
   - zeek.config
   - zeek.sostatus
 
@@ -30,6 +32,7 @@ so-zeek:
     {%   endfor %}
     {% endif %}
     - binds:
+      - {{ CATRUST.dir }}
       - /nsm/zeek/logs:/nsm/zeek/logs:rw
       - /nsm/zeek/spool:/nsm/zeek/spool:rw
       - /nsm/zeek/extracted:/nsm/zeek/extracted:rw
@@ -63,6 +66,7 @@ so-zeek:
       {% endfor %}
     {% endif %}
     - watch:
+      - cmd: ca_trust_extract
       - file: /opt/so/conf/zeek/local.zeek
       - file: /opt/so/conf/zeek/node.cfg
       - file: /opt/so/conf/zeek/networks.cfg