Data compression algorithms explained: lossless, lossy and how to choose

Lossless vs lossy compression

Lossless compression reverses exactly: decompress and you get the same bytes, bit for bit. It is the only option for text, source code, logs, databases, executables and anything a program will parse. Lossy compression is for signals meant for eyes and ears: photos, audio and video. It keeps what matters to perception and throws the rest away, which lets it shrink media far more than any lossless method, at the cost of never getting the original back.

Two limits apply before any algorithm is chosen. First, no lossless algorithm can compress every possible input. RFC 1951, the DEFLATE specification, calls this a simple counting argument (there are fewer short files than long ones, so not every input can map to a shorter output), and it bounds DEFLATE's worst case at 5 bytes of growth per 32 KiB block. Second, results depend on how predictable the data is. The same RFC notes that English text usually compresses by a factor of 2.5 to 3, executables somewhat less, and raster images can compress much more. Random or encrypted data barely compresses at all.

How lossless compression algorithms work

Almost every general-purpose compressor in use today does two jobs in sequence. It finds repeated strings and replaces them with short references, then it codes what is left so that common symbols take fewer bits. Some formats add a transform before both, to put the data into a shape that repeats more. The figure at the top of this page shows the pipeline.

Dictionary methods: LZ77 and its descendants

In May 1977 Ziv and Lempel published the method now called LZ77. The compressor keeps a sliding window of recent input. When the next bytes match something inside the window, it writes a pair of numbers instead of the bytes: how far back the match starts (the distance) and how many bytes to copy (the length). The decoder keeps the same window and copies the bytes from it. DEFLATE uses a 32 KiB window and matches of 3 to 258 bytes. Brotli allows windows up to 16 MiB, and the Zstandard format allows windows of up to 3.75 TB, although for interoperability it recommends that encoders stay within 8 MB.

The old Unix compress program used LZW, another member of the Lempel-Ziv family, and HTTP still registers it as the compress coding. A more useful variation today is a dictionary both sides hold before the data arrives. Brotli ships a built-in static dictionary of words, each usable in 121 transformed forms. Zstandard can load a dictionary trained on samples of your own data, which its documentation recommends for small records: a 1 KB record gives the compressor no history to learn from, and a dictionary trained on many similar records supplies one.

Entropy coding: Huffman, arithmetic coding and ANS

After matching, the output is a stream of literal bytes, lengths and distances, and some values occur far more often than others. Entropy coding turns that skew into savings. Huffman coding, published by D. A. Huffman in September 1952, builds a prefix code in which frequent symbols get short bit strings and rare ones longer strings. DEFLATE codes its literals, lengths and distances with Huffman codes and can rebuild the code tables for every block.

Huffman's limit is that every code is a whole number of bits, which amounts to rounding each symbol's probability to a power of two. Arithmetic coding gets very close to the theoretical limit, called Shannon entropy, but costs more computation. Asymmetric numeral systems (ANS), proposed by Jarek Duda in 2013, aim for arithmetic coding's ratio at Huffman's speed. Zstandard uses a form of ANS called Finite State Entropy (FSE) for match lengths, literal lengths and offsets, and Huffman coding for literals. Brotli stays with Huffman but adds context modelling: the code used for the next literal depends on a context computed from the two bytes before it.

Transforms that make data easier to compress

A transform shrinks nothing by itself. It rearranges or predicts values so that the matching and coding stages find more to work with:

• PNG passes each row of pixels through one of five filters that predict a byte from its neighbours (left, above, upper left) and store the difference, then compresses the filtered rows with DEFLATE.
• bzip2 runs each block through the Burrows-Wheeler block-sorting transform (BWT) before Huffman coding.
• Delta encoding stores the difference between consecutive values, so a column of timestamps or counters becomes a column of small numbers. Apache Parquet defines it for integer columns.
• FLAC, the lossless audio format standardized as RFC 9639 in December 2024, predicts each sample from the previous ones and stores only the residual (the prediction error) with Rice coding.

How lossy compression works for images, audio and video

Lossy codecs follow a common pattern: predict, transform, quantize, then entropy code. Google's documentation for lossy WebP walks through it. A block of pixels is predicted from the pixels already decoded around it, the difference (the residual) goes through a discrete cosine transform (DCT) that typically leaves many zero values, those values are quantized to a coarser scale, and the result is entropy coded. The same documentation points out that quantization is the only step where information is discarded; every other step can be inverted exactly.

The same pattern runs through the formats you meet every day:

• Images. JPEG's core coding system is a Huffman-coded, DCT-based lossy format. WebP adds block prediction and arithmetic coding, and Google reports that lossy WebP files are 25 to 34% smaller than comparable JPEGs at equivalent SSIM quality, and lossless WebP files 26% smaller than PNGs.
• Video. Codecs such as AV1 add inter prediction: a frame is predicted from frames already decoded, and only the residual, the difference between the prediction and the real frame, is coded.
• Audio. Opus (RFC 6716, September 2012) combines linear prediction with a modified DCT and covers everything from 6 kbit/s narrowband speech to 510 kbit/s stereo music. For lossless audio, FLAC uses the same prediction and residual idea without discarding anything.

Because quantization loses information each time, keep a lossless master and encode every lossy copy from it. Re-encoding a JPEG or an MP4 quantizes an already degraded signal again.

gzip vs Brotli vs Zstandard vs LZ4: the general-purpose algorithms compared

Brotli's authors describe it as similar in speed to DEFLATE with denser output. Zstandard covers a wide range: levels 1 to 19 by default (3 is the default level), 20 to 22 unlocked on request at a large cost in memory, and negative "fast" levels for speed. Its manual gives a rule of thumb that compression speed halves every two levels, while decompression speed stays roughly the same at every level, a property most LZ-family compressors share.

The Zstandard project publishes a benchmark of fast settings on the Silesia compression corpus, a public set of test files, run with lzbench on a Core i7-9700K:

Read it for the shape of the trade-off, not for your numbers. At level 1, zstd and Brotli reach nearly the same ratio, and zstd compresses nearly five times as fast as zlib. LZ4 gives up about a quarter of zstd's ratio for about 2.5 times its decompression speed, while Snappy, at a similar ratio to LZ4, decodes no faster than zstd. Higher levels move every tool toward better ratios and slower compression, and your own data will land somewhere else on the curve.

Where data compression is used

Compression is rarely a product you buy on its own. It is a setting inside the web server, the file format, the database, the file system and the backup tool, and each has its own defaults.

On the web: HTTP content encoding

HTTP negotiates compression per response. The browser lists the codings it can decode in the Accept-Encoding request header; the server compresses the body with one of them and names it in Content-Encoding; caches need Vary: Accept-Encoding so they do not serve a Brotli body to a client that asked for gzip. The registered codings for general use are gzip, br and zstd, plus the older deflate and compress.

As of September 2026, MDN's compatibility data lists Brotli support in all major browsers, and zstd in Chrome and Edge 123, Firefox 126 and Safari 26.3. RFC 9659 (September 2024) requires a zstd window of at most 8 MB for HTTP, because some browsers limit the memory they will spend on decoding.

In practice:

1. Compress text, not media. HTML, CSS, JavaScript, JSON, SVG and XML shrink well. JPEG, PNG, WebP, MP4 and ZIP files are already compressed, and MDN notes that compressing them again can make them larger.
2. Pre-compress static files at build time at the highest level, and serve them directly. Apache's mod_brotli (available since httpd 2.4.26) recompresses on every request unless you serve pre-compressed files, and nginx's gzip_static module (not built by default) serves a ready-made .gz file instead.
3. Compress dynamic responses at moderate levels. mod_brotli's default quality of 5 on a 0 to 11 scale is meant as a reasonable balance for dynamic content; nginx's gzip_comp_level defaults to 1.
4. Keep gzip enabled as the fallback for older clients and scripts.
5. Check what is served, from outside:

curl -s -o /dev/null -D - -H 'Accept-Encoding: zstd, br, gzip' https://example.com/ \
  | grep -i -E 'content-encoding|vary'

A typical nginx configuration compresses on the fly and serves a pre-compressed .gz file when one exists. gzip_types adds types to text/html, which is always compressed, and responses shorter than gzip_min_length are left alone:

gzip on;
gzip_comp_level 5;
gzip_min_length 1024;
gzip_types text/css application/javascript application/json image/svg+xml;
gzip_vary on;
gzip_static on;

Lighthouse checks for this in its "Enable text compression" audit, which flags any text response of 1.4 KiB or more, served without Brotli, gzip or deflate, that gzip would shrink by at least 10%. When a CDN sits in front of the site, compression usually happens there: Cloudflare, for example, serves gzip, Brotli or Zstandard to visitors depending on the browser's Accept-Encoding, the plan and any compression rules. Our hosting and maintenance service covers CDN caching and image and asset optimization as part of looking after a site.

In files, packages and archives

A .gz file is a DEFLATE stream with a header and a CRC-32 checksum. A Zstandard frame can end with an optional checksum taken from the XXH64 hash, while a Brotli stream carries no checksum at all, so its integrity check has to come from the container or the transport. ZIP records a method for every entry: method 8 is DEFLATE, and version 6.3.10 of the specification also defines methods for bzip2, LZMA, XZ and Zstandard. PNG, whose third edition became a W3C Recommendation in June 2025, always uses DEFLATE with a window of at most 32,768 bytes.

Package managers show the trade-off well because a package is compressed once and decompressed on every install. When Arch Linux moved its packages from xz to zstd at the end of December 2019, it reported that the total size of all packages grew by about 0.8% while decompression became about 1,300% faster.

In databases and columnar formats such as Parquet

Analytical data compresses best when values of one column are stored together, because they look alike. Apache Parquet works in two stages. It first encodes each column: dictionary encoding stores each distinct value once and replaces the column with small integer keys, run-length encoding and bit-packing shrink those keys, and delta encoding suits integers that change slowly, such as timestamps. It then compresses each page with a codec such as SNAPPY, GZIP, BROTLI, ZSTD or LZ4_RAW (the older LZ4 codec is deprecated because of an undocumented framing scheme). Where Parquet files sit in a full analytics stack is covered in our guide to building data analytics software.

Databases make similar choices. ClickHouse compresses columns with LZ4 by default in self-managed deployments and with zstd in ClickHouse Cloud, and it lets you chain a specialized codec before a general one, for example CODEC(Delta, ZSTD) on a numeric column. PostgreSQL compresses large column values (its TOAST mechanism) with its own pglz method by default, or with LZ4 when the server was built with LZ4 support and default_toast_compression or the column's setting asks for lz4.

If your warehouse or lakehouse bill grows faster than your data, file layout and codecs are among the first places to look. Our data platform service chooses warehouse or lakehouse storage for your volumes, skills and budget, with Apache Iceberg or Delta Lake tables, partitioning and cost controls.

In storage and backups

File systems compress block by block, transparently. In OpenZFS, compression=on selects LZ4 on pools with the lz4_compress feature enabled, zstd levels 1 to 19 are available when you want a better ratio, and changing the property affects only data written afterwards. Savings are rounded to whole disk sectors, so a block that saves less than one sector is stored uncompressed.

Backup tools add deduplication, which is a different thing: compression removes redundancy inside data, deduplication stores identical chunks only once across files and backups. BorgBackup deduplicates chunks of the source data first and then compresses each stored chunk, with LZ4 by default, zstd at levels 1 to 22 (3 if you do not choose), zlib, or lzma for low speed and high compression. Its auto mode tests each chunk with LZ4 and skips compression for incompressible data such as media files. For database dumps, pg_dump in PostgreSQL 18 accepts gzip, LZ4 or zstd, and its custom and directory formats compress with gzip by default.

Always compress before you encrypt. Encrypted output looks random, so a backup that is encrypted first will not compress afterwards.

How to choose a compression algorithm

Work through the questions in this order:

1. Can you lose information? For media delivered to people, use a lossy codec at a quality you have checked by eye or ear. For everything else, lossless.
2. Who pays, and how often? Data compressed once and read many times (static web files, packages, published datasets) justifies slow, high levels, as the Arch Linux example shows. Data compressed on every request or write (API responses, logs, messages, database pages) needs fast levels.
3. How much memory does the reader have? Large windows and high levels raise memory on both sides. Zstandard's levels 20 to 22 need more memory to decompress, the zstd command-line tool limits decompression to 128 MiB of memory unless told otherwise, and browsers expect 8 MB windows at most.
4. Who has to read it? Browsers accept gzip, Brotli and, now, zstd. Partners and old scripts may accept only gzip. A Parquet file is only as portable as its codec's support in the engines that will read it.
5. Measure on your data. Zstandard's benchmark mode tests a range of levels on a sample file in one command:

zstd -b1 -e19 sample.json

What is new in data compression

Claims of a "revolutionary" compression algorithm deserve a careful read, because nothing beats the counting argument: no lossless algorithm wins on all data, so real progress comes from exploiting specific kinds of data or specific delivery paths. As of September 2026, these are the developments worth knowing:

• Zstandard in every major browser. With Safari 26.3, zstd content encoding works in Chrome, Edge, Firefox and Safari, after Chrome shipped it in version 123.
• Compression dictionary transport. RFC 9842, a Standards Track RFC published in September 2025, lets a server mark a response as a dictionary for later requests. A new version of a JavaScript bundle can then travel as a small delta against the version the browser already holds, using the dcb (Brotli) or dcz (Zstandard) codings. MDN lists support in Chrome and Edge 130; Firefox has it in Nightly and, from version 145, behind a preference, and Safari not at all.
• Format-aware compression. Meta released OpenZL in October 2025. You describe the shape of your data (rows, columns, fields), a trainer searches for the transforms that compress it best, and one universal decoder reads every file by following the recipe embedded in it. When OpenZL does not understand the input, it falls back to zstd.
• Better encodings for floating-point columns. ALP (Adaptive Lossless floating-Point), adapted from a SIGMOD 2024 paper, is now in the Parquet specification and, as of August 1, 2026, marked Preview: the format is stable, but reader support is still being built, so enable it only when every reader you use supports it. It turns decimal-like floats such as prices and sensor readings into integers before packing them.
• Learned image compression. JPEG AI, published as ISO/IEC 6048-1:2025, is the first international image coding standard based on an end-to-end learning-based approach.

When you read about a new algorithm with remarkable ratios, ask three questions: on which data was it measured, against which baseline, and at what speed and memory cost.

Security risks of compression

Compression parses untrusted input and changes the size of what it outputs, and both create risks:

• Decompression bombs. A small file can expand into an enormous output. MITRE classifies the weakness as CWE-409, and its example is a small ZIP file that decompresses into a very large amount of data. Cap the output size and memory of every decompression of untrusted input, and treat uploaded archives like any other untrusted input, as our secure coding checklist describes.
• Compression side channels. When secret data and attacker-controlled data are compressed together and then encrypted, the compressed length leaks information. The CRIME attack used compression at the TLS layer, and TLS 1.3 removed compression from the protocol. BREACH uses HTTP body compression, and RFC 7457 notes there is no mitigation at the TLS level, so applications have to act. The RFC's example is randomizing CSRF tokens; more generally, stop compressing the responses that reflect user input next to secrets.
• Size leaks in stored data. Compressed sizes can reveal what a file is even when its content is encrypted. BorgBackup offers an obfuscate mode that obscures compressed chunk sizes for this reason.