Make your
data smaller.
Bitmech is a lossless compression engine for logs and telemetry. It stores them in a fraction of the space of standard compression, with every file still readable on its own.
THE PROBLEM
Storage grows.
So does the cost.
Organisations generate ever-growing volumes of machine data, such as logs, events and telemetry, and often keep it for years. That data is highly repetitive, but standard compression leaves much of the redundancy in place, so it is paid for again and again.
WHAT IT DOES
Smaller storage,
nothing lost.
Lossless
Every file is restored exactly, byte for byte. Bitmech is designed for data where the original must be preserved.
Individually accessible
Any file can be read on its own, without unpacking an archive.
Built for machine data
System logs, JSON telemetry and event data: high-volume data written continuously, where formats and values recur.
RESULTS
Less storage than
every method we measured.
Across 5.4 GB of real-world system logs and JSON telemetry, Bitmech needed less storage in total than any other method we measured, including log-specific compression, while keeping every file individually accessible. Every file was restored exactly.
On every dataset we tested, Bitmech needed less storage than every method that keeps files individually readable: maximum-strength xz, zstd with a trained dictionary, and the log-specific compressor CLP.
Less storage than the best per-file alternative on each dataset, from 21% on Spark telemetry to 52% on Hadoop logs.
Even when CLP, the log-specific compressor, packs each whole dataset into one archive at maximum compression with its best configuration, Bitmech needed 19–58% less storage on the three Hadoop and supercomputer log datasets.
Total storage Bitmech needed, including everything required to restore every file. The best single archive needed 130.9 MB (CLP), xz 134.4 MB, and the best per-file alternative 166.7 MB (xz).
Most systems compress with zstd or gzip at their default settings. On the same five datasets, zstd (level 3) needed 264.3 MB and gzip (level 6) 388.8 MB: 2.8 and 4.1 times the storage Bitmech needed, with every file still individually accessible in all three. Bitmech needed 58–71% less than zstd on every dataset.
Storage needed, % of original size, as the dataset grows (GB). Lower is better.
- Bitmech1.56%
- Single archive (xz)2.64%
- Per file (xz)3.05%
- Per file (zstd + dictionary)3.42%
Storage needed, % of original size, after 1.07 GB. Lower is better.
Bitmech is in active development, and these results come from the current prototype. Speed. The prototype is tuned for storage savings first and is not yet as fast as established compressors. On a single laptop it stores 32–108 MB/s and reads back 97–215 MB/s with eight files in flight. Throughput work is ongoing. Where single archives still win. On Windows event logs and Spark telemetry, packing an entire dataset into one archive still stores it more compactly than Bitmech does today, at the cost of access to individual files. Improving results on these datasets is ongoing. Logs and telemetry are where our benchmark started. Results on other data will vary.
Five public datasets of 1.07 GB each (four LogHub log datasets, and Spark event logs by K. Rodrigues, R. Wang and D. Gibson, YScope, CC BY 4.0), arriving in ten batches on a standard laptop; every file restored and checked. Standard compressors at their common defaults (gzip 6, zstd 3) and their strongest settings (xz -9e, zstd 19 with a trained dictionary, CLP level 19 in its best configuration per dataset), measured on exactly the same data. Detailed methodology available on request.
Benchmark datasets: LogHub ↗USE CASES
Where
it fits.
Machine-generated data
Logs, telemetry, events and metrics: high-volume data written continuously.
Long-term retention
Archives and data kept for years for compliance or analysis, where every saving applies for as long as the data is held.
Works with your storage
Designed to work with your existing cloud object storage.
ABOUT BITMECH
About
Bitmech.
Bitmech is a UK company building compression technology for large-scale machine data.
Logs, telemetry and event data are written around the clock and kept for years for operations, security and compliance, and the cost of storing them grows with every day they are held. Our aim is to make that data far cheaper to keep, without losing a single byte or giving up access to any individual file.
We benchmark Bitmech on public datasets against the strongest alternatives we can find, and we are open about where we are not yet ahead.
Bitmech has filed a patent application for its technology.
CONTACT
Interested in
the problem?
We are developing the technology and extending our benchmark. If your organisation stores large and growing volumes of logs or telemetry, and you would like to discuss the problem or see what Bitmech does with a sample of your own data, get in touch.
[email protected]↗