Merge pull request #648 from Yamato-Security/637-separate-mitre-attck-tags-and-other-tags-when-outputting

Separate mitre attck tags and other tags when outputting

CHANGELOG-Japanese.md

@@ -9,6 +9,7 @@
 **改善:**
 
 - ルールのアップデート機能のルールパスの出力から./を削除した。 (#642) (@hitenkoku)
+- MITRE ATT&CK関連のタグとその他タグを出力するための出力用のエイリアスを追加した。 (#637) (@hitenkoku)
 
 **バグ修正:**
 

CHANGELOG.md

@@ -9,6 +9,7 @@
 **Enhancements:**
 
 - Removed ./ from rule path when updating. (#642) (@hitenkoku)
+- Added new output alias for MITRE ATT&CK tags and other tags. (#637) (@hitenkoku)
 
 **Bug Fixes:**
 

README-Japanese.md

@@ -332,7 +332,6 @@ OPTIONS:
         --RFC-3339                                     RFC 3339形式で日付と時刻を出力する (例: 2022-02-22 22:00:00.123456-06:00)
         --US-military-time                             24時間制(ミリタリータイム)のアメリカ形式で日付と時刻を出力する (例: 02-22-2022 22:00:00.123 -06:00)
         --US-time                                      アメリカ形式で日付と時刻を出力する (例: 02-22-2022 10:00:00.123 PM -06:00)
-        --all-tags                                     出力したCSVファイルにルール内のタグ情報を全て出力する
     -c, --rules-config <RULE_CONFIG_DIRECTORY>         ルールフォルダのコンフィグディレクトリ (デフォルト: ./rules/config)
         --contributors                                 コントリビュータの一覧表示
     -d, --directory <DIRECTORY>                        .evtxファイルを持つディレクトリのパス
@@ -509,7 +508,9 @@ Hayabusaの結果を標準出力に表示しているとき（デフォルト）
 * `Title`: YML検知ルールの`title`フィールドから来ています。
 * `RecordID`: イベントレコードIDです。`<Event><System><EventRecordID>`フィールドから来ています。
 * `Details`: YML検知ルールの`details`フィールドから来ていますが、このフィールドはHayabusaルールにしかありません。このフィールドはアラートとイベントに関する追加情報を提供し、ログのフィールドから有用なデータを抽出することができます。イベントキーのマッピングが間違っている場合、もしくはフィールドが存在しない場合で抽出ができなかった箇所は`n/a` (not available)と記載されます。YML検知ルールに`details`フィールドが存在しない時のdetailsのメッセージを`./rules/config/default_details.txt`で設定できます。`default_details.txt`では`Provider Name`、`EventID`、`details`の組み合わせで設定することができます。default_details.txt`やYML検知ルールに対応するルールが記載されていない場合はすべてのフィールド情報を出力します。
-* `MitreAttack`: MITRE ATT&CKの戦術。
+* `MitreTactics`: MITRE ATT&CKの戦術。
+* `MitreTags`: MITRE ATT&CKの戦術以外の情報。attack.g(グループ)、attack.t(技術)、attack.s(ソフトウェア)の情報を出力します。
+* `OtherTags`: YML検知ルールの`tags` フィールドから`MitreTactics`, `MitreTags` 以外の月情報を出力します。
 * `RuleFile`: アラートまたはイベントを生成した検知ルールのファイル名。
 * `EvtxFile`: アラートまたはイベントを起こしたevtxファイルへのパス。
 * `RecordInformation`: すべてのフィールド情報。
@@ -527,7 +528,9 @@ default_profiles.txtをprofile.txtに書かれているプロファイルで上
 |%Channel% | `Channel` |
 |%Level% | `Level` |
 |%EventID% | `EventID` |
-|%MitreAttack% | `MitreAttack` |
+|%MitreTactics% | `MitreTactics` |
+|%MitreTags% | `MitreTags` |
+|%OtherTags% | `OtherTags` |
 |%RecordID% | `RecordID` |
 |%RuleTitle% | `Title` |
 |%Details% | `Details` |

README.md

@@ -328,7 +328,6 @@ OPTIONS:
         --RFC-3339                                     Output timestamp in RFC 3339 format (ex: 2022-02-22 22:00:00.123456-06:00)
         --US-military-time                             Output timestamp in US military time format (ex: 02-22-2022 22:00:00.123 -06:00)
         --US-time                                      Output timestamp in US time format (ex: 02-22-2022 10:00:00.123 PM -06:00)
-        --all-tags                                     Output all tags when saving to a CSV file
     -c, --rules-config <RULE_CONFIG_DIRECTORY>         Specify custom rule config folder (default: ./rules/config)
         --contributors                                 Print the list of contributors
     -d, --directory <DIRECTORY>                        Directory of multiple .evtx files
@@ -506,7 +505,9 @@ When hayabusa output is being displayed to the screen (the default), it can disp
 * `RecordID`: This comes from the `<Event><System><EventRecordID>` field in the event log.
 * `Title`: This comes from the `title` field in the YML detection rule.
 * `Details`: This comes from the `details` field in the YML detection rule, however, only hayabusa rules have this field. This field gives extra information about the alert or event and can extract useful data from the fields in event logs. For example, usernames, command line information, process information, etc... When a placeholder points to a field that does not exist or there is an incorrect alias mapping, it will be outputted as `n/a` (not available). If the `details` field is not specified (i.e. sigma rules), default `details` messages to extract fields defined in `./rules/config/default_details.txt` will be outputted. You can add more default `details` messages by adding the `Provider　Name`, `EventID` and `details` message you want to output in `default_details.txt`. When no `details` field is defined in a rule nor in `default_details.txt`, all fields will be outputted to the `details` column.
-* `MitreAttack`: MITRE ATT&CK tactics.
+* `MitreTactics`: MITRE ATT&CK tactics.
+* `MitreTags`: MITRE ATT&CK group, technique, software.
+* `OtherTags`: This comes from the `tags` field in YML detection rule which is excluded `MitreTactics` and `MitreTags`.
 * `RuleFile`: The filename of the detection rule that generated the alert or event.
 * `EvtxFile`: The evtx filename that caused the alert or event.
 * `RecordInformation`: All field information.
@@ -525,7 +526,9 @@ Please use `--set-default-profile`  option when you want to overwrite `default_p
 |%Channel% | `Channel` |
 |%Level% | `Level` |
 |%EventID% | `EventID` |
-|%MitreAttack% | `MitreAttack` |
+|%MitreTactics% | `MitreTactics` |
+|%MitreTags% | `MitreTags` |
+|%OtherTags% | `OtherTags` |
 |%RecordID% | `RecordID` |
 |%RuleTitle% | `Title` |
 |%Details% | `Details` |

config/default_profile.yaml

@@ -2,12 +2,9 @@
 Timestamp: "%Timestamp%"
 Computer: "%Computer%"
 Channel: "%Channel%"
-Level: "%Level%"
 EventID: "%EventID%"
-MitreAttack: "%MitreAttack%"
+Level: "%Level%"
+MitreTactics: "%MitreTactics%"
 RecordID: "%RecordID%"
 RuleTitle: "%RuleTitle%"
-Details: "%Details%"
+Details: "%Details%"
-RecordInformation: "%RecordInformation%"
-RuleFile: "%RuleFile%"
-EvtxFile: "%EvtxFile%"

config/profiles.yaml

@@ -1,3 +1,4 @@
+#Standard profile minus MITRE ATT&CK Tactics and Record ID.
 minimal:
     Timestamp: "%Timestamp%"
     Computer: "%Computer%"
@@ -13,32 +14,56 @@ standard:
     Channel: "%Channel%"
     EventID: "%EventID%"
     Level: "%Level%"
-    Tags: "%MitreAttack%"
+    MitreTactics: "%MitreTactics%"
     RecordID: "%RecordID%"
     RuleTitle: "%RuleTitle%"
     Details: "%Details%"
 
-verbose-1:
+#Standard profile plus MitreTags(MITRE techniques, software and groups), rule filename and EVTX filename.
+verbose:
     Timestamp: "%Timestamp%"
     Computer: "%Computer%"
     Channel: "%Channel%"
     EventID: "%EventID%"
     Level: "%Level%"
-    Tags: "%MitreAttack%"
+    MitreTactics: "%MitreTactics%"
+    MitreTags: "%MitreTags%"
+    OtherTags: "%OtherTags%"
     RecordID: "%RecordID%"
     RuleTitle: "%RuleTitle%"
     Details: "%Details%"
     RuleFile: "%RuleFile%"
     EvtxFile: "%EvtxFile%"
 
-verbose-2:
+#Verbose profile with all field information instead of the minimal fields defined in Details.
+verbose-all-field-info:
     Timestamp: "%Timestamp%"
     Computer: "%Computer%"
     Channel: "%Channel%"
     EventID: "%EventID%"
     Level: "%Level%"
-    Tags: "%MitreAttack%"
+    MitreTactics: "%MitreTactics%"
+    MitreTags: "%MitreTags%"
+    OtherTags: "%OtherTags%"
+    RecordID: "%RecordID%"
+    RuleTitle: "%RuleTitle%"
+    AllFieldInfo: "%RecordInformation%"
+    RuleFile: "%RuleFile%"
+    EvtxFile: "%EvtxFile%"
+
+#Verbose profile plus all field information. (Warning: this will more than double the output file size!)
+verbose-details-and-all-field-info:
+    Timestamp: "%Timestamp%"
+    Computer: "%Computer%"
+    Channel: "%Channel%"
+    EventID: "%EventID%"
+    Level: "%Level%"
+    MitreTactics: "%MitreTactics%"
+    MitreTags: "%MitreTags%"
+    OtherTags: "%OtherTags%"
     RecordID: "%RecordID%"
     RuleTitle: "%RuleTitle%"
     Details: "%Details%"
+    RuleFile: "%RuleFile%"
+    EvtxFile: "%EvtxFile%"
     AllFieldInfo: "%RecordInformation%"

src/afterfact.rs

@@ -579,11 +579,8 @@ mod tests {
 
     #[test]
     fn test_emit_csv_output() {
-        let mock_ch_filter = message::create_output_filter_config(
+        let mock_ch_filter =
-            "rules/config/channel_abbreviations.txt",
+            message::create_output_filter_config("test_files/config/channel_abbreviations.txt");
-            true,
-            false,
-        );
         let test_filepath: &str = "test.evtx";
         let test_rulepath: &str = "test-rule.yml";
         let test_title = "test_title";

src/detections/configs.rs

@@ -93,10 +93,6 @@ pub struct Config {
     #[clap(short = 'o', long, value_name = "CSV_TIMELINE")]
     pub output: Option<PathBuf>,
 
-    /// Output all tags when saving to a CSV file
-    #[clap(long = "all-tags")]
-    pub all_tags: bool,
-
     /// Output verbose information
     #[clap(short = 'v', long)]
     pub verbose: bool,

src/detections/detection.rs

@@ -6,6 +6,7 @@ use crate::options::profile::{
     LOAEDED_PROFILE_ALIAS, PRELOAD_PROFILE, PRELOAD_PROFILE_REGEX, PROFILES,
 };
 use chrono::{TimeZone, Utc};
+use itertools::Itertools;
 use termcolor::{BufferWriter, Color, ColorChoice};
 
 use crate::detections::message::AlertMessage;
@@ -208,7 +209,7 @@ impl Detection {
 
     /// 条件に合致したレコードを格納するための関数
     fn insert_message(rule: &RuleNode, record_info: &EvtxRecordInfo) {
-        let tag_info: Vec<String> = Detection::get_tag_info(rule);
+        let tag_info: &Vec<String> = &Detection::get_tag_info(rule);
         let recinfo = record_info
             .record_information
             .as_ref()
@@ -275,9 +276,6 @@ impl Detection {
                     "%EventID%" => {
                         profile_converter.insert("%EventID%".to_string(), eid.to_owned());
                     }
-                    "%MitreAttack%" => {
-                        profile_converter.insert("%MitreAttack%".to_string(), tag_info.join(" : "));
-                    }
                     "%RecordID%" => {
                         profile_converter.insert(
                             "%RecordID%".to_string(),
@@ -319,6 +317,44 @@ impl Detection {
                                 .to_string(),
                         );
                     }
+                    "%MitreTactics%" => {
+                        let tactics: &Vec<String> = &tag_info
+                            .iter()
+                            .filter(|x| TAGS_CONFIG.values().contains(x))
+                            .map(|y| y.to_owned())
+                            .collect();
+                        profile_converter.insert("%MitreTactics%".to_string(), tactics.join(" : "));
+                    }
+                    "%MitreTags%" => {
+                        let techniques: &Vec<String> = &tag_info
+                            .iter()
+                            .filter(|x| {
+                                !TAGS_CONFIG.values().contains(x)
+                                    && (x.starts_with("attack.t")
+                                        || x.starts_with("attack.g")
+                                        || x.starts_with("attack.s"))
+                            })
+                            .map(|y| {
+                                let mut replaced_tag = y.replace("attack.", "");
+                                make_ascii_titlecase(&mut replaced_tag)
+                            })
+                            .collect();
+                        profile_converter.insert("%MitreTags%".to_string(), techniques.join(" : "));
+                    }
+                    "%OtherTags%" => {
+                        let tags: &Vec<String> = &tag_info
+                            .iter()
+                            .filter(|x| {
+                                !(TAGS_CONFIG.values().contains(x)
+                                    || x.starts_with("attack.t")
+                                    || x.starts_with("attack.g")
+                                    || x.starts_with("attack.s"))
+                            })
+                            .map(|y| y.to_owned())
+                            .collect();
+                        profile_converter.insert("%OtherTags%".to_string(), tags.join(" : "));
+                    }
+
                     _ => {}
                 }
             }
@@ -350,7 +386,7 @@ impl Detection {
 
     /// insert aggregation condition detection message to output stack
     fn insert_agg_message(rule: &RuleNode, agg_result: AggResult) {
-        let tag_info: Vec<String> = Detection::get_tag_info(rule);
+        let tag_info: &Vec<String> = &Detection::get_tag_info(rule);
         let output = Detection::create_count_output(rule, &agg_result);
         let rec_info = if LOAEDED_PROFILE_ALIAS.contains("%RecordInformation%") {
             Option::Some(String::default())
@@ -386,9 +422,6 @@ impl Detection {
                     "%EventID%" => {
                         profile_converter.insert("%EventID%".to_string(), "-".to_owned());
                     }
-                    "%MitreAttack%" => {
-                        profile_converter.insert("%MitreAttack%".to_owned(), tag_info.join(" : "));
-                    }
                     "%RecordID%" => {
                         profile_converter.insert("%RecordID%".to_string(), "".to_owned());
                     }
@@ -415,6 +448,43 @@ impl Detection {
                     "%EvtxFile%" => {
                         profile_converter.insert("%EvtxFile%".to_string(), "-".to_owned());
                     }
+                    "%MitreTactics%" => {
+                        let tactics: &Vec<String> = &tag_info
+                            .iter()
+                            .filter(|x| TAGS_CONFIG.values().contains(x))
+                            .map(|y| y.to_owned())
+                            .collect();
+                        profile_converter.insert("%MitreTactics%".to_string(), tactics.join(" : "));
+                    }
+                    "%MitreTags%" => {
+                        let techniques: &Vec<String> = &tag_info
+                            .iter()
+                            .filter(|x| {
+                                !TAGS_CONFIG.values().contains(x)
+                                    && (x.starts_with("attack.t")
+                                        || x.starts_with("attack.g")
+                                        || x.starts_with("attack.s"))
+                            })
+                            .map(|y| {
+                                let mut replaced_tag = y.replace("attack.", "");
+                                make_ascii_titlecase(&mut replaced_tag)
+                            })
+                            .collect();
+                        profile_converter.insert("%MitreTags%".to_string(), techniques.join(" : "));
+                    }
+                    "%OtherTags%" => {
+                        let tags: &Vec<String> = &tag_info
+                            .iter()
+                            .filter(|x| {
+                                !(TAGS_CONFIG.values().contains(x)
+                                    || x.starts_with("attack.t")
+                                    || x.starts_with("attack.g")
+                                    || x.starts_with("attack.s"))
+                            })
+                            .map(|y| y.to_owned())
+                            .collect();
+                        profile_converter.insert("%OtherTags%".to_string(), tags.join(" : "));
+                    }
                     _ => {}
                 }
             }
@@ -447,8 +517,14 @@ impl Detection {
                 .as_vec()
                 .unwrap_or(&Vec::default())
                 .iter()
-                .filter_map(|info| TAGS_CONFIG.get(info.as_str().unwrap_or(&String::default())))
+                .map(|info| {
-                .map(|str| str.to_owned())
+                    if let Some(tag) = TAGS_CONFIG.get(info.as_str().unwrap_or(&String::default()))
+                    {
+                        tag.to_owned()
+                    } else {
+                        info.as_str().unwrap_or(&String::default()).to_owned()
+                    }
+                })
                 .collect(),
             true => rule.yaml["tags"]
                 .as_vec()
@@ -457,7 +533,7 @@ impl Detection {
                 .map(
                     |info| match TAGS_CONFIG.get(info.as_str().unwrap_or(&String::default())) {
                         Some(s) => s.to_owned(),
-                        _ => info.as_str().unwrap_or("").replace("attack.", ""),
+                        _ => info.as_str().unwrap_or("").to_string(),
                     },
                 )
                 .collect(),

src/detections/message.rs

@@ -48,11 +48,9 @@ lazy_static! {
     pub static ref STATISTICS_FLAG: bool = configs::CONFIG.read().unwrap().args.statistics;
     pub static ref LOGONSUMMARY_FLAG: bool = configs::CONFIG.read().unwrap().args.logon_summary;
     pub static ref TAGS_CONFIG: HashMap<String, String> = create_output_filter_config(
-        utils::check_setting_path(&CURRENT_EXE_PATH.to_path_buf(), "config/output_tag.txt")
+        utils::check_setting_path(&CURRENT_EXE_PATH.to_path_buf(), "config/mitre_tactics.txt")
             .to_str()
             .unwrap(),
-        true,
-        configs::CONFIG.read().unwrap().args.all_tags
     );
     pub static ref CH_CONFIG: HashMap<String, String> = create_output_filter_config(
         utils::check_setting_path(
@@ -61,8 +59,6 @@ lazy_static! {
         )
         .to_str()
         .unwrap(),
-        false,
-        configs::CONFIG.read().unwrap().args.all_tags
     );
     pub static ref PIVOT_KEYWORD_LIST_FLAG: bool =
         configs::CONFIG.read().unwrap().args.pivot_keywords_list;
@@ -94,15 +90,8 @@ lazy_static! {
 
 /// ファイルパスで記載されたtagでのフル名、表示の際に置き換えられる文字列のHashMapを作成する関数。
 /// ex. attack.impact,Impact
-pub fn create_output_filter_config(
+pub fn create_output_filter_config(path: &str) -> HashMap<String, String> {
-    path: &str,
-    read_tags: bool,
-    pass_flag: bool,
-) -> HashMap<String, String> {
     let mut ret: HashMap<String, String> = HashMap::new();
-    if read_tags && pass_flag {
-        return ret;
-    }
     let read_result = utils::read_csv(path);
     if read_result.is_err() {
         AlertMessage::alert(read_result.as_ref().unwrap_err()).ok();
@@ -591,9 +580,9 @@ mod tests {
         );
     }
     #[test]
-    /// test of loading output filter config by output_tag.txt
+    /// test of loading output filter config by mitre_tactics.txt
-    fn test_load_output_tag() {
+    fn test_load_mitre_tactics_log() {
-        let actual = create_output_filter_config("test_files/config/output_tag.txt", true, false);
+        let actual = create_output_filter_config("test_files/config/mitre_tactics.txt");
         let expected: HashMap<String, String> = HashMap::from([
             ("attack.impact".to_string(), "Impact".to_string()),
             ("xxx".to_string(), "yyy".to_string()),
@@ -601,24 +590,11 @@ mod tests {
         _check_hashmap_element(&expected, actual);
     }
 
-    #[test]
-    /// test of loading pass by output_tag.txt
-    fn test_no_load_output_tag() {
-        let actual = create_output_filter_config("test_files/config/output_tag.txt", true, true);
-        let expected: HashMap<String, String> = HashMap::new();
-        _check_hashmap_element(&expected, actual);
-    }
-
     #[test]
     /// loading test to channel_abbrevations.txt
     fn test_load_abbrevations() {
-        let actual =
+        let actual = create_output_filter_config("test_files/config/channel_abbreviations.txt");
-            create_output_filter_config("test_files/config/channel_abbreviations.txt", false, true);
+        let actual2 = create_output_filter_config("test_files/config/channel_abbreviations.txt");
-        let actual2 = create_output_filter_config(
-            "test_files/config/channel_abbreviations.txt",
-            false,
-            false,
-        );
         let expected: HashMap<String, String> = HashMap::from([
             ("Security".to_string(), "Sec".to_string()),
             ("xxx".to_string(), "yyy".to_string()),

src/options/profile.rs

@@ -36,12 +36,14 @@ lazy_static! {
         "%Channel%",
         "%Level%",
         "%EventID%",
-        "%MitreAttack%",
         "%RecordID%",
         "%RuleTitle%",
         "%RecordInformation%",
         "%RuleFile%",
-        "%EvtxFile%"
+        "%EvtxFile%",
+        "%MitreTactics%",
+        "%MitreTags%",
+        "%OtherTags%"
     ];
     pub static ref PRELOAD_PROFILE_REGEX: RegexSet = RegexSet::new(&*PRELOAD_PROFILE).unwrap();
 }