Time-Series Compression: Storing More with Less

Why it exists

Time-series data, common in system monitoring and IoT, accumulates rapidly because new measurements are always inserted instead of updated. This leads to massive datasets, often terabytes or larger, resulting in significant storage costs and slower query performance. These specialized compression algorithms were developed to make storing and querying this data feasible and cost-effective.

The mental model

Think of time-series compression like describing a walk. Instead of listing your exact GPS coordinates every second, you state your starting point and then just say "one step forward, one step forward, turn right." Because the changes are small and simple, the description is much shorter. Similarly, these algorithms store a full initial value, then encode the small, subsequent changes (deltas) using far fewer bits.

How it works

A suite of lossless algorithms is used, each tailored to specific data patterns. Delta encoding stores the difference between consecutive values. Delta-delta encoding takes it a step further, storing the difference of the differences, which is highly effective if the rate of change is constant. For floating-point numbers, XOR-based compression compares the bit patterns of adjacent values. Since they are often very similar, the result of a bitwise XOR operation has many zeros, which can be compressed efficiently.

When to use it

Use these algorithms for any data that is recorded sequentially over time and where values are often similar to their neighbors. This is ideal for IT infrastructure metrics (CPU, memory), IoT sensor readings (temperature, pressure), and financial data (stock prices). The primary benefit is a drastic reduction in storage costs (often over 90%), with a secondary benefit of faster queries since less data needs to be read from disk.

When not to use it

Avoid these techniques for data that is not sequential or ordered. They are ineffective on datasets like user account tables or product catalogs, where one record has no predictable numerical relationship to the next. The "delta" between two unrelated records would be large and random, offering no compression advantage and adding unnecessary CPU overhead.

One canonical example

An application's memory usage is recorded every minute: 100.1MB, 100.2MB, 100.3MB. Instead of storing three full floating-point numbers, a system can use delta-delta encoding. It stores the initial value (100.1), the first delta (+0.1), and the delta of the deltas (the change between subsequent deltas, which is 0). The sequence becomes 100.1, +0.1, 0, which can be stored using significantly fewer bits than the original values.