fixed event timeline frequency font color changed when use f option #542 (#544)

* fixed event timeline frequency font color changed when use f option #542

* fixed duplicate count detection to same record in data reduction #543

* fixed error #542

* fixed fmt

* fixed error

* changed no frequency timeline output when events is less than 5 record.

* Error message update

* added newline after progress bar

* fixed  frequency timeline logic to pass test

* cargo fmt

* Added CHANGELOG #533 #538 #525

* changelog update

* updated readme #533 #538

* readme update

* startup logo update

Co-authored-by: Tanaka Zakku <71482215+YamatoSecurity@users.noreply.github.com>

CHANGELOG-Japanese.md

@@ -4,11 +4,14 @@
 
 **新機能:**
 
-- xxx
+- 検知されたイベントが5つ以上の時、イベント頻度のタイムラインを作成するようにした。 (#533) (@hitenkoku)
+- `--all-tags`オプションでルールにある全てのtagsを、outputで指定したcsvのMitreAttackの列に出力するようにした。 (#525) (@hitenkoku)
 
 **改善:**
 
-- ルールの`details`でeventkey_alias.txtやEvent.EventData内に存在しない情報を`n/a` (not available)と表記するようにした。(#528) (@hitenkoku)
+- ルールの`details`でeventkey_alias.txtやEvent.EventData内に存在しない情報を`n/a` (not available)と表記するようにした。 (#528) (@hitenkoku)
+- 読み込んだイベント数と検知しなかったイベント数を表示するようにした。 (#538) (@hitenkoku)
+- 新しいロゴ。 (@YamatoSecurity)
 
 **バグ修正:**
 

CHANGELOG.md

@@ -4,11 +4,14 @@
 
 **New Features:**
 
-- xxx
+- Added Event Frequency Timeline feature to visualize the number of events. (Note: There needs to be more than 5 events.) (#533)(@hitenkoku)
+- Display all the `tags` defined in a rule to the `MitreAttack` column when saving to CSV file with the `--all-tags` option. (#525) (@hitenkoku)
 
 **Enhancements:**
 
 - In the `details` line in a rule, when a placeholder points to a field that does not exist or there is an incorrect alias mapping, it will be outputted as `n/a` (not available). (#528) (@hitenkoku)
+- Display total event and data reduction count. (How many and what percent of events were ignored.) (#538) (@hitenkoku)
+- New logo. (@YamatoSecurity)
 
 **Bug Fixes:**
 

README-Japanese.md

@@ -33,6 +33,7 @@ Hayabusaは、日本の[Yamato Security](https://yamatosecurity.connpass.com/)
 - [スクリーンショット](#スクリーンショット)
   - [起動画面:](#起動画面)
   - [ターミナル出力画面:](#ターミナル出力画面)
+  - [イベント頻度タイムライン出力画面:](#イベント頻度タイムライン出力画面)
   - [結果サマリ画面:](#結果サマリ画面)
   - [Excelでの解析:](#excelでの解析)
   - [Timeline Explorerでの解析:](#timeline-explorerでの解析)
@@ -64,6 +65,7 @@ Hayabusaは、日本の[Yamato Security](https://yamatosecurity.connpass.com/)
   - [Channel情報の省略](#channel情報の省略)
   - [プログレスバー](#プログレスバー)
   - [標準出力へのカラー設定](#標準出力へのカラー設定)
+  - [イベント頻度タイムライン](#イベント頻度タイムライン)
 - [Hayabusaルール](#hayabusaルール)
   - [Hayabusa v.s. 変換されたSigmaルール](#hayabusa-vs-変換されたsigmaルール)
   - [検知ルールのチューニング](#検知ルールのチューニング)
@@ -104,6 +106,10 @@ Windowsのイベントログは、
 
 ![Hayabusa ターミナル出力画面](/screenshots/Hayabusa-Results.png)
 
+## イベント頻度タイムライン出力画面:
+
+![Hayabusa イベント頻度タイムライン出力画面](/screenshots/HayabusaEventFrequencyTimeline.png)
+
 ## 結果サマリ画面:
 
 ![Hayabusa 結果サマリ画面](/screenshots/HayabusaResultsSummary.png)
@@ -118,6 +124,7 @@ Windowsのイベントログは、
 
 
 ## Criticalアラートのフィルタリングとコンピュータごとのグルーピング:
+
 ![Timeline ExplorerでCriticalアラートのフィルタリングとコンピュータグルーピング](screenshots/TimelineExplorer-CriticalAlerts-ComputerGrouping.png)
 
 ## Elastic Stackダッシュボードでの解析:
@@ -560,6 +567,11 @@ Hayabusaの結果は`level`毎に文字色が変わります。
 形式は`level名,(6桁のRGBのカラーhex)`です。
 カラー出力をしないようにしたい場合は`--no-color`オプションをご利用ください。
 
+## イベント頻度タイムライン
+
+検知したイベントの数が5以上の時、頻度のタイムライン(スパークライン)を画面に出力します。
+マーカーの数は最大10個です。
+
 # Hayabusaルール
 
 Hayabusa検知ルールはSigmaのようなYML形式で記述されています。`rules`ディレクトリに入っていますが、将来的には[https://github.com/Yamato-Security/hayabusa-rules](https://github.com/Yamato-Security/hayabusa-rules)のレポジトリで管理する予定なので、ルールのissueとpull requestはhayabusaのレポジトリではなく、ルールレポジトリへお願いします。

README.md

@@ -32,6 +32,7 @@ Hayabusa is a **Windows event log fast forensics timeline generator** and **thre
 - [Screenshots](#screenshots)
   - [Startup](#startup)
   - [Terminal Output](#terminal-output)
+  - [Time Fequency Timeline](#time-fequency-timeline)
   - [Results Summary](#results-summary)
   - [Analysis in Excel](#analysis-in-excel)
   - [Analysis in Timeline Explorer](#analysis-in-timeline-explorer)
@@ -63,6 +64,7 @@ Hayabusa is a **Windows event log fast forensics timeline generator** and **thre
   - [Channel Abbreviations](#channel-abbreviations)
   - [Progress Bar](#progress-bar)
   - [Color Output](#color-output)
+  - [Event Fequency Timeline](#event-fequency-timeline)
 - [Hayabusa Rules](#hayabusa-rules)
   - [Hayabusa v.s. Converted Sigma Rules](#hayabusa-vs-converted-sigma-rules)
   - [Detection Rule Tuning](#detection-rule-tuning)
@@ -100,6 +102,9 @@ Hayabusa is not intended to be a replacement for tools like [Evtx Explorer](http
 
 ![Hayabusa terminal output](/screenshots/Hayabusa-Results.png)
 
+## Time Fequency Timeline
+
+![Hayabusa Event Frequency Timeline](/screenshots/HayabusaEventFrequencyTimeline.png)
 
 ## Results Summary
 
@@ -214,6 +219,7 @@ Please let us know if anything breaks after you update.
 ## Cross-compiling 32-bit Windows Binaries
 
 You can create 32-bit binaries on 64-bit Windows systems with the following:
+
 ```bash
 rustup install stable-i686-pc-windows-msvc
 rustup target add i686-pc-windows-msvc
@@ -223,6 +229,7 @@ rustup run stable-i686-pc-windows-msvc cargo build --release
 ## macOS Compiling Notes
 
 If you receive compile errors about openssl, you will need to install [Homebrew](https://brew.sh/) and then install the following packages:
+
 ```bash
 brew install pkg-config
 brew install openssl
@@ -233,16 +240,18 @@ brew install openssl
 If you receive compile errors about openssl, you will need to install the following package.
 
 Ubuntu-based distros:
+
 ```bash
 sudo apt install libssl-dev
 ```
 
 Fedora-based distros:
+
 ```bash
 sudo yum install openssl-devel
 ```
 
-# Running Hayabusa 
+# Running Hayabusa
 
 ## Caution: Anti-Virus/EDR Warnings
 
@@ -556,6 +565,11 @@ The alerts will be outputted in color based on the alert `level`.
 You can change the default colors in the config file at `./config/level_color.txt` in the format of `level,(RGB 6-digit ColorHex)`.
 If you want to disable color output, you can use `--no-color` option.
 
+## Event Fequency Timeline
+
+The Event Frequency Timeline feature displays a sparkline frequency timeline of detected events.
+Note: There needs to be more than 5 events.
+
 # Hayabusa Rules
 
 Hayabusa detection rules are written in a sigma-like YML format and are located in the `rules` folder. In the future, we plan to host the rules at [https://github.com/Yamato-Security/hayabusa-rules](https://github.com/Yamato-Security/hayabusa-rules) so please send any issues and pull requests for rules there instead of the main hayabusa repository.

src/afterfact.rs

@@ -5,9 +5,11 @@ use crate::detections::utils;
 use chrono::{DateTime, Local, TimeZone, Utc};
 use csv::QuoteStyle;
 use hashbrown::HashMap;
+use hashbrown::HashSet;
 use krapslog::{build_sparkline, build_time_markers};
 use lazy_static::lazy_static;
 use serde::Serialize;
+use std::cmp::min;
 use std::error::Error;
 use std::fs::File;
 use std::io;
@@ -101,18 +103,24 @@ fn _get_output_color(color_map: &HashMap<String, Color>, level: &str) -> Option<
 }
 
 /// print timeline histogram
-fn _print_timeline_hist(
-    timestamps: Vec<i64>,
-    marker_count: usize,
-    length: usize,
-    side_margin_size: usize,
-) {
+fn _print_timeline_hist(timestamps: Vec<i64>, length: usize, side_margin_size: usize) {
     if timestamps.is_empty() {
         return;
     }
 
     let buf_wtr = BufferWriter::stdout(ColorChoice::Always);
     let mut wtr = buf_wtr.buffer();
+    wtr.set_color(ColorSpec::new().set_fg(None)).ok();
+    if timestamps.len() < 5 {
+        write!(
+            wtr,
+            "Event Frequency Timeline could not be displayed as there needs to be more than 5 events.",
+        )
+        .ok();
+        writeln!(wtr).ok();
+        buf_wtr.print(&wtr).ok();
+        return;
+    }
 
     let title = "Event Frequency Timeline";
     let header_row_space = (length - title.len()) / 2;
@@ -121,8 +129,15 @@ fn _print_timeline_hist(
     writeln!(wtr, "{}", title).ok();
     writeln!(wtr).ok();
 
+    let timestamp_marker_max = if timestamps.len() < 2 {
+        0
+    } else {
+        timestamps.len() - 2
+    };
+    let marker_num = min(timestamp_marker_max, 10);
+
     let (header_raw, footer_raw) =
-        build_time_markers(&timestamps, marker_count, length - (side_margin_size * 2));
+        build_time_markers(&timestamps, marker_num, length - (side_margin_size * 2));
     let sparkline = build_sparkline(&timestamps, length - (side_margin_size * 2));
     for header_str in header_raw.lines() {
         writeln!(wtr, "{}{}", " ".repeat(side_margin_size - 1), header_str).ok();
@@ -197,9 +212,11 @@ fn emit_csv<W: std::io::Write>(
     println!();
     let mut timestamps: Vec<i64> = Vec::new();
     let mut plus_header = true;
+    let mut detected_record_idset: HashSet<String> = HashSet::new();
     for (time, detect_infos) in messages.iter() {
         timestamps.push(_get_timestamp(time));
         for detect_info in detect_infos {
+            detected_record_idset.insert(format!("{}_{}", time, detect_info.eventid));
             let mut level = detect_info.level.to_string();
             if level == "informational" {
                 level = "info".to_string();
@@ -276,10 +293,10 @@ fn emit_csv<W: std::io::Write>(
         Some((Width(w), _)) => w as usize,
         None => 100,
     };
-    _print_timeline_hist(timestamps, 10, terminal_width, 3);
+
+    _print_timeline_hist(timestamps, terminal_width, 3);
     println!();
-    let reducted_record_cnt: u128 =
-        all_record_cnt - total_detect_counts_by_level.iter().sum::<u128>();
+    let reducted_record_cnt: u128 = all_record_cnt - detected_record_idset.len() as u128;
     let reducted_percent = if all_record_cnt == 0 {
         0 as f64
     } else {

src/main.rs

@@ -467,6 +467,7 @@ impl App {
             total_records += cnt_tmp;
             pb.inc();
         }
+        println!();
         detection.add_aggcondition_msges(&self.rt);
         if !(*STATISTICS_FLAG || *LOGONSUMMARY_FLAG || *PIVOT_KEYWORD_LIST_FLAG) {
             after_fact(total_records);