Benchmarking Full-Text Search Performance when using LogsDB mode

Choosing the right indexing strategy in Elasticsearch is crucial for balancing speed, storage, and stability. In this guide, we compare Logs DB Index and Standard Index with real-world benchmarks using Elastic Rally, the official performance testing tool from Elastic. We'll also explore how ES|QL's full-text search capabilities enhance Logs DB, making it a powerful choice for log analytics.

Why Compare Logs DB vs. Standard Index?

Elasticsearch provides two main indexing approaches:

• Logs DB Index: Specially optimized for log data, focusing on efficient storage and stable performance.

• Standard Index: A more general-purpose option, offering flexibility across different workloads.

But which one is better for log-heavy applications? We tested both under real-world conditions to find out!

How We Tested (Benchmarking Methodology)

We used Elastic Rally with realistic log datasets and measured key performance metrics:

• Indexing performance (speed of adding new data)

• Query speed & efficiency (how fast searches return results)
  

Full-Text Query Used for Testing

To benchmark query performance, we executed a MATCH query using Elasticsearch’s full-text search capabilities. The test query used was: 

{  "query": {    "bool": {      "must": [        {          "match": {            "message": "Database connection failed"          }        }      ],      "should": [        {          "match": {            "details": "359+charcters."          }        }      ]    }  }}

Test Parameters [Rally config]:

• Iterations: 1000

• Clients: 5

• Warmup Iterations: 10

• Workload: Simulated real-world log searches in a Kubernetes environment
  

Performance Comparison: Logs DB index vs. Standard Index

Full-Text Search Performance

Metric	Logs DB Index	Standard Index	Winner
Min Throughput	11.88 ops/s	10.55 ops/s	Logs DB
Mean Throughput	14.12 ops/s	13.96 ops/s	Logs DB
Max Throughput	15.01 ops/s	14.86 ops/s	Logs DB

Takeaway: Logs DB performs better in full-text search across all throughput metrics, making it a better choice for log-based searches 

How ES|QL Enhances Logs DB's Full-Text Search

Logs DB is designed for efficient log storage, retrieval, and analytics, but search performance is just as important as indexing speed. This is where ES|QL (Elasticsearch Query Language) comes in—it optimizes and accelerates full-text searches in Logs DB, making it highly effective for log-heavy workloads.

Here’s how ES|QL enhances Logs DB’s performance

• MATCH function: Enables fast, Lucene-style full-text search. 
  

Example Query 

GET logs/_search?size=1

{

  "query": {

    "match": {

      "message": "error"

    }

  },

  "sort": [

    { "timestamp": "desc" }

  ]

}

• QSTR function: Allows advanced filtering using Lucene query syntax.

Example Query

GET logs/_search?size=1

{

  "query": {

    "query_string": {

      "query": "message:\"server error\" OR status:500"

    }

  },

  "sort": [

    { "timestamp": "desc" }

  ]

}

Why This Matters for Logs DB

• Pre-indexed fields mean instant searches with no runtime slowdown. [ Use GET /your-index/_mapping to check which fields are indexed. Indexed fields will have "index": true in their mapping ]

• Improved geo-searching for log data with location-based filters. 
  

Example Use Case:

If your logs contain location-based data (e.g., IP addresses mapped to latitude/longitude), you can now:

• Filter logs from a specific region (e.g., "Show logs from New York only").

• Sort logs by proximity (e.g., "Find the nearest server experiencing errors").
  

Which One Should You Use?

Choosing the right index depends on your workload and priorities. Here’s a breakdown of when each option makes the most sense:

Logs DB Index if:

• Storage efficiency matters to reduce infrastructure costs.

• Log analytics & search is your primary use case
  

Standard Index if:

• Fast indexing is top priority..

• Flexibility is required to handle various data types beyond logs.

• Latency spikes are acceptable, as Standard Index may experience performance variability.
  

By understanding these, you can optimize your Elasticsearch setup to best match your use case and performance needs.

Final Thoughts

For log analytics, Logs DB Index is a high-performance, cost-effective solution, especially when combined with ES|QL’s powerful search features. Standard Index remains a strong general-purpose option but can struggle with high-volume logging.

Pro Tip: Run your own Elastic Rally tests to see what works best for your data!.

Note: These benchmarks were performed using Elastic Rally. Your results may vary depending on workload and cluster configurations.