LogLead is designed to efficiently benchmark log anomaly detection algorithms and log representations. LogLead is also usedas a backend for projects such as LogDelta and VisualLogAnalyzer, which offer a more user-friendly approach to log analysis and log anomaly detection.
Currently, it features nearly 1,000 unique anomaly detection combinations, encompassing 8 public datasets, 11 log representations (enhancers), and 11 classifiers. These resources enable you to benchmark your own data, log representation, or classifier against a diverse range of scenarios. If there's something you believe should be included, please submit a request for a dataset, enhancer, or classifier in the issue tracker.
A key strength of LogLead is its custom loader system, which efficiently isolates the unique aspects of logs from different systems. This design allows for a reduction in redundant code, as the same enhancement and anomaly detection code can be applied universally once the logs are loaded.
Install with uv:
uv add loglead
Or with pip:
python -m pip install loglead
Then clone the project, move to demo folder and run some demos
git clone https://github.com/EvoTestOps/LogLead.git
cd LogLead
uv run demo/HDFS_samples.py
uv run demo/TB_samples.py
Or with pip (after installing LogLead into your environment):
cd LogLead/demo
python HDFS_samples.py
python TB_samples.py
uv run syncs the environment from pyproject.toml/uv.lock on first use, so there's no separate install step before running anything.
To start working with your own data, it is easiest to begin with the RawLoader. To try out RawLoader, run the RawLoaderDemo. For this, you will need the original BGL and HDFS datasets. You will also need to edit the RawLoaderDemo script or add a ".env" file to your LogLead root so that the demo knows where the data is located on your machine. See .env.sample as an example of how the ".env" file should look. After that run the demo
uv run demo/RawLoader_NoLabels.py
Or with pip:
python RawLoader_NoLabels.py
Finally, you can try downloading all data. The downloader script downloads the public datasets listed in downloader/datasets.yml:
uv run downloader/download_data.py
Or with pip (after cloning the repo):
python downloader/download_data.py
If you've cloned the repo and want to run the test suite too, point it at tests/datasets.yml instead, which also controls what gets loaded and how it's used in testing:
uv run downloader/download_data.py --config tests/datasets.yml
Or with pip:
python downloader/download_data.py --config tests/datasets.yml
Disk space: downloading everything in downloader/datasets.yml transfers roughly 7 GB and the datasets expand to about 104 GB once unzipped. Make sure you have at least ~110 GB free before running the full downloader. The three supercomputer logs — Liberty, Spirit, and Thunderbird — account for most of it, at 30-38 GB each once unzipped.
If you're short on space, edit the datasets: list in downloader/datasets.yml (or tests/datasets.yml if you're using --config tests/datasets.yml) and set download: false for datasets you don't need.
| Dataset | Download size | Unzipped size |
|---|---|---|
| BGL | 58 MB | 709 MB |
| Hadoop | 3 MB | 49 MB |
| HDFS | 187 MB | 1.8 GB |
| Liberty | 672 MB | 30 GB |
| Spirit | 906 MB | 38 GB |
| Thunderbird | 2.0 GB | 30 GB |
| Nezha (git clone) | ~2.9 GB | 2.9 GB |
| ADFA-LD | 2.4 MB | 26 MB |
| AWSCTD | 10 MB | 559 MB |
| Total | ~6.7 GB | ~104 GB |
- If
scikit-learnwheel fails to compile, check that you cangccandg++installed. - pip version does not have the
tensorflowdependencies necessary forBertEmbeddings. Install them manually (preferably in a conda enviroment).
In the following demonstrations, you'll notice a significant aspect of LogLead's design efficiency: code reusability. Both demos, while analyzing different datasets, share a substantial amount of their underlying code. This not only showcases LogLead's versatility in handling various log formats but also its ability to streamline the analysis process through reusable code components.
- Script: TB_samples.py
- Description: This demo presents a Thunderbird supercomputer log, labeled at the line (event) level. A first column marked with “-” indicates normal behavior, while other markings represent anomalies.
- Log Snapshot: View the log here.
- Dataset: The demo includes a parquet file containing a subset of 263,408 log events, with 21,955 anomalies.
- Screencast: For an overview of the demo, watch our 5-minute screencast on YouTube.
- Script: HDFS_samples.py
- Description: This demo showcases logs from the Hadoop Distributed File System (HDFS), labeled at the sequence level (a sequence is a collection of multiple log events).
- Log Snapshot: View the log here.
- Anomaly Labels: Provided in a separate file.
- Dataset: The demo includes a parquet file containing a subset of 222,579 log events, forming 11,501 sequences with 350 anomalies.
Typically, our test procedure includes running the following. The demos can reveal obvious errors quickly, while the full test set takes a bit longer to run—up to 30minutes.
Basic demos
uv run demo/HDFS_samples.py
uv run demo/TB_samples.py
Or with pip:
cd demo
python HDFS_samples.py
python TB_samples.py
Parser benchmark
uv run demo/parser_benchmark/ano_detection.py
uv run demo/parser_benchmark/parsing_speed.py
Or with pip:
cd demo/parser_benchmark
python ano_detection.py
python parsing_speed.py
Run full tests
uv run tests/main.py
Or with pip:
cd tests
python main.py
Below you can see anomaly detection results (F1-Binary) trained on 0.5% subset of HDFS data. We use 5 different log message enhancement strategies: Words, Drain, LenMa, Spell, and BERT
The enhancement strategies are tested with 5 different machine learning algorithms: DT (Decision Tree), SVM (Support Vector Machine), LR (Logistic Regression), RF (Random Forest), and XGB (eXtreme Gradient Boosting).
| Words | Drain | Lenma | Spell | Bert | Average | |
|---|---|---|---|---|---|---|
| DT | 0.9719 | 0.9816 | 0.9803 | 0.9828 | 0.9301 | 0.9693 |
| SVM | 0.9568 | 0.9591 | 0.9605 | 0.9559 | 0.8569 | 0.9378 |
| LR | 0.9476 | 0.8879 | 0.8900 | 0.9233 | 0.5841 | 0.8466 |
| RF | 0.9717 | 0.9749 | 0.9668 | 0.9809 | 0.9382 | 0.9665 |
| XGB | 0.9721 | 0.9482 | 0.9492 | 0.9535 | 0.9408 | 0.9528 |
| --------- | -------- | -------- | -------- | -------- | -------- | --------- |
| Average | 0.9640 | 0.9503 | 0.9494 | 0.9593 | 0.8500 |
LogLead is composed of distinct modules: the Loader, Enhancer, and Anomaly Detector. We use Polars dataframes as its notably faster than Pandas.
Loader: This module reads in the log files and deals with the specifics features of each log file. It produces a dataframe with certain semi-mandatory fields. These fields enable actions in the subsequent stages. LogLead has a raw loader that can load any log file. It also has custom loaders to the following public datasets from 10 different systems. Custom loaders should result in more accurate anomaly detection:
- 3: HDFS_v1, Hadoop, BGL thanks to amazing LogHub team. For full data see Zenodo.
- 3: Sprit, Thunderbird and Liberty can be found from Usenix site.
- 2: Nezha has data from two systems TrainTicket and Google Cloud Webshop demo. It is the first dataset of microservice-based systems. Like other traditional log datasets it has Log data but additionally there are Traces and Metrics.
- 2: ADFA and AWSCTD are two datasets designed for intrusion detection.
Enhancer: This module extracts additional data from logs. The enhancement takes place directly within the dataframes, where new columns are added as a result of the enhancement process. For example, log parsing, the creation of tokens from log messages, and measuring log sequence lengths are all considered forms of log enhancement. Enhancement can happen at the event level or be aggregated to the sequence level. Some of the enhancers available: Event Length (chracters, words, lines), Sequence Length, Sequence Duration, following "NLP" enhancers: Regex, Words, Character n-grams. Log parsers: Drain, LenMa, Spell, IPLoM, AEL, Brain, Fast-IPLoM, Tipping, and BERT. NextEventPrediction including its probablities and perplexity. Next event prediction can be computed on top of any of the parser output.
Anomaly Detector: This module uses the enhanced log data to perform Anomaly Detection. It is mainly using SKlearn at the moment but there are few customer algorithms as well. LogLead has been integrated and tested with following models:
- Supervised (5): Decision Tree, Support Vector Machine, Logistic Regression, Random Forest, eXtreme Gradient Boosting
- Unsupervised (4): One-class SVM, Local Outlier Factor, Isolation Forest, K-Means
- Custom Unsupervised (2): Out-of-Vocabulary Detector counts amount words or character n-grams that are novel in test set. Rarity Model, scores seen words or character n-grams based on their rarity in training set. See our public preprint for more details
Mäntylä MV, Wang Y, Nyyssölä J. Loglead-fast and integrated log loader, enhancer, and anomaly detector. In2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) 2024 Mar 12 (pp. 395-399). IEEE. PDF, preprint