top of page
Search

Benchmark Storage Reduction: Elasticsearch LogsDB index mode vs Standard index mode 

Introduction In this blog, we’ll walk through a performance and storage comparison between standard Elasticsearch indices and LogsDB. Using Elasticsearch Rally (esrally), we benchmarked data ingestion, storage size, — and the results 

Understanding LogsDB Mode Introduced in Elasticsearch version 8.17, LogsDB is an advanced index mode designed to optimize log data storage . By implementing smart index sorting, advanced compression techniques, and synthetic _source reconstruction, LogsDB enhances storage efficiency. This mode is particularly beneficial for organizations to manage extensive log data more effectively.

Benchmark Setup

We used Elasticsearch Rally to simulate real-world data ingestion across both standard and LogsDB indices.

Tools and Environment:

  • esrally for data ingestion benchmarking

  • Machine: Ubuntu VM running Docker for Rally

    • GCP Instance: e2-standard-16 (16 vCPUs, 64 GB RAM, 250 GB storage)

  • Elastic Cloud Cluster: Logsdb-logs cluster setup

    • Version: 8.17.3

    • Hot Storage: 720 GB

    • Hot Memory: 16 GB

    • Kibana Memory: 1 GB

Setting Up Elasticsearch Rally

Let’s walk through the setup process:

  1. Install Docker

    • Ensure Docker is installed and running on your VM.

  2. Pull the Rally Image

docker pull elastic/rally

  1. List Available Tracks

sudo docker run elastic/rally list tracks

  1. Create a Persistent Volume

    • This lets us reuse data between runs:

docker volume create rally_data

  1. Run the Benchmark

    • We use the tsdb track and benchmark with 123gb data:

    • This tsdb track uses a dataset containing Kubernetes cluster metrics collected by Metricbeat — including pod performance, node stats, and container-level telemetry. This dataset represents time-series data, and the Es Rally run will ingest this data to Elasticsearch's index .

docker run -d --rm -v rally_data:/rally/.rally elastic/rally race \--track=tsdb \--pipeline=benchmark-only \--target-hosts=<es-url>:9243 \--client-options="use_ssl:true,verify_certs:true,basic_auth_user:'elastic',basic_auth_password:'<password>'" \--track-params='{"bulk_indexing_clients":25,"ingest_percentage":100}' \

  1.  This setup produces 116,633,698 documents in the Elasticsearch.


  1. We’ll do this process for standard and logsDb 

For that, we’ll prepare two index templates:

  • One with Standard index settings

  • One with Logs DB index settings (outlined below);


Index Settings Update

During the benchmark setup, we updated the index settings to optimize performance for both standard indices and LogsDB. Below are the configurations applied:

The settings were applied using an Elasticsearch index template, ensuring all indices matching the pattern inherit these configurations automatically


Standard Index Settings:

{  "settings": {    "index": {      "mode": "standard",      "codec": "default",      "mapping": {        "synthetic_source_keep": "none"      },      "sort": {        "field": [],        "order": [],        "mode": [],        "missing": []      }    }  }}

LogsDB Index Settings:

{  "index": {    "mode": "logsdb",    "mapping.synthetic_source_keep": "arrays",    "sort.field": ["host.name", "@timestamp"],    "sort.order": ["desc", "desc"],    "sort.mode": ["min", "min"],    "sort.missing": ["_first", "_first"],    "codec": "best_compression",    "mapping.ignore_malformed": true,    "mapping.ignore_above": 8191,    "mapping.total_fields.ignore_dynamic_beyond_limit": true  }}

LogsDB Optimization Settings

LogsDB achieves its efficiency through three key techniques, reflected in the Elasticsearch index settings:

  1. Smart Index Sorting

    • "sort.field": ["host.name", "@timestamp"] ensures logs are grouped by host and timestamp, reducing fragmentation .

    • "sort.order": ["desc", "desc"] keeps the latest data first for faster retrieval.

  2. Advanced Compression

    • "codec": "best_compression" applies optimized compression (Zstandard, delta encoding) to reduce storage without sacrificing performance.

  3. synthetic_source

    • "mapping.synthetic_source_keep": "arrays" keeps essential arrays while discarding the full _source, reconstructing data only when needed — cutting storage usage significantly.

Other Performance Tweaks:

  • "mapping.ignore_malformed": true skips broken data to avoid ingestion errors.

  • "mapping.ignore_above": 8191 limits oversized fields for faster indexing.

  • "mapping.total_fields.ignore_dynamic_beyond_limit": true prevents dynamic field explosions from bloating the index.


Data Ingestion and Storage Results

Let’s break down the numbers!

Each Rally run ingests 116,633,698 documents into the Elasticsearch index. To complete the full dataset, we ran 8 times in standard mode index and 8 times with LogsDB index . 

Standard Index:

  • Total Documents: 933,069,584

  • Total Storage Size: 381.82 GB

  • Average Ingestion Time: 1:15 - 1:30 hours for 1 es rally run in standard mode . 


LogsDB Index:

  • Total Documents: 933,069,584 (same document count)

  • Total Storage Size: 206.06 GB

  • Average Ingestion Time: 1:45 - 2:00 hours for 1 es rally run in logsdb mode


Storage Reduction: LogsDB slashed storage by 46.01% — almost half the size of standard mode!

Key Takeaways

  • 46.01% Storage Savings compared to standard: LogsDB drastically reduced storage needs without losing a single document.

  • Elastic Cloud Friendly: Less data stored = lower costs and faster snapshots.

If you’re scaling log ingestion in Elastic, LogsDB is a game-changer worth exploring.


bottom of page