# Data compression algorithms explained: lossless, lossy and how to choose

> Data compression algorithms remove redundancy to store data in fewer bits. How gzip, Brotli, Zstandard, LZ4 and lossy codecs work, and how to choose.

- URL: https://computese.com/revolutionary-data-compression-algorithm/
- Author: Duong Quan Nguyen, CEO, Computese
- Published: 2024-09-11
- Updated: 2026-09-25
- Topics: Data, Hosting

## In short
- Data compression algorithms remove redundancy. Lossless ones (DEFLATE, Brotli, Zstandard, LZ4) give back every byte; lossy codecs (JPEG, WebP, AV1, Opus) discard detail people are unlikely to notice.
- Most general-purpose lossless compressors pair LZ77-style matching of repeated strings with entropy coding (Huffman, or ANS in Zstandard). They differ mainly in window size, entropy coder and speed.
- No algorithm wins everywhere: choose by who decompresses and how often, the CPU and memory budget and what the reader supports, then measure on a sample of your own data.
- On the web, serve text with Brotli or Zstandard and keep gzip as the fallback. As of September 2026, zstd decodes in Chrome 123+, Firefox 126+ and Safari 26.3+.
- Columnar formats such as Apache Parquet encode each column first (dictionary, run-length, delta) and then compress the pages with a codec such as ZSTD or Snappy.

Data compression algorithms shrink data by finding redundancy and encoding it in fewer bits. Lossless algorithms such as DEFLATE (gzip), Brotli, Zstandard and LZ4 give back every original byte; lossy codecs such as JPEG, AV1 and Opus discard detail people are unlikely to notice. Choosing one means balancing compression ratio, speed and memory.

This guide explains how the main families work (dictionary methods, entropy coding and the transforms around them), compares the general-purpose algorithms you will meet in practice, shows where each one is used, from HTTP responses to Parquet files and backups, and ends with a way to choose, what is genuinely new, and the security traps.

## Lossless vs lossy compression

Lossless compression reverses exactly: decompress and you get the same bytes, bit for bit. It is the only option for text, source code, logs, databases, executables and anything a program will parse. Lossy compression is for signals meant for eyes and ears: photos, audio and video. It keeps what matters to perception and throws the rest away, which lets it shrink media far more than any lossless method, at the cost of never getting the original back.

|                             | Lossless                                       | Lossy                                                       |
| --------------------------- | ---------------------------------------------- | ----------------------------------------------------------- |
| After decoding              | Identical to the input                         | An approximation of the input                               |
| Typical data                | Text, code, logs, tables, backups, PNG, FLAC   | Photos (JPEG, lossy WebP), audio (Opus), video              |
| Where the savings come from | Repeated strings and skewed symbol frequencies | Detail below what people perceive, then the lossless tricks |
| What you tune               | Speed against ratio                            | Quality against size                                        |

Two limits apply before any algorithm is chosen. First, no lossless algorithm can compress every possible input. [RFC 1951, the DEFLATE specification](https://www.rfc-editor.org/rfc/rfc1951.html), calls this a simple counting argument (there are fewer short files than long ones, so not every input can map to a shorter output), and it bounds DEFLATE's worst case at 5 bytes of growth per 32 KiB block. Second, results depend on how predictable the data is. The same RFC notes that English text usually compresses by a factor of 2.5 to 3, executables somewhat less, and raster images can compress much more. Random or encrypted data barely compresses at all.

## How lossless compression algorithms work

Almost every general-purpose compressor in use today does two jobs in sequence. It finds repeated strings and replaces them with short references, then it codes what is left so that common symbols take fewer bits. Some formats add a transform before both, to put the data into a shape that repeats more. The figure at the top of this page shows the pipeline.

### Dictionary methods: LZ77 and its descendants

In May 1977 Ziv and Lempel published the method now called LZ77. The compressor keeps a sliding window of recent input. When the next bytes match something inside the window, it writes a pair of numbers instead of the bytes: how far back the match starts (the distance) and how many bytes to copy (the length). The decoder keeps the same window and copies the bytes from it. DEFLATE uses a 32 KiB window and matches of 3 to 258 bytes. Brotli allows [windows up to 16 MiB](https://www.rfc-editor.org/rfc/rfc7932.html), and the Zstandard format allows [windows of up to 3.75 TB](https://www.rfc-editor.org/rfc/rfc8878.html), although for interoperability it recommends that encoders stay within 8 MB.

![A strip of patterned tiles is copied into a shorter strip below, where a repeated group of tiles is replaced by one orange token whose arrow points back to the earlier copy.](https://computese.com/images/blog/revolutionary-data-compression-algorithm/back-reference.d0b9d01562-1536.webp)

*Repeats become pointers: the more often data repeats within the window, the smaller the output.*

The old Unix `compress` program used LZW, another member of the Lempel-Ziv family, and [HTTP still registers it](https://www.rfc-editor.org/rfc/rfc9110.html) as the `compress` coding. A more useful variation today is a dictionary both sides hold before the data arrives. Brotli ships a built-in static dictionary of words, each usable in 121 transformed forms. Zstandard can load a dictionary trained on samples of your own data, which [its documentation recommends](https://github.com/facebook/zstd) for small records: a 1 KB record gives the compressor no history to learn from, and a dictionary trained on many similar records supplies one.

### Entropy coding: Huffman, arithmetic coding and ANS

After matching, the output is a stream of literal bytes, lengths and distances, and some values occur far more often than others. Entropy coding turns that skew into savings. Huffman coding, published by D. A. Huffman in September 1952, builds a prefix code in which frequent symbols get short bit strings and rare ones longer strings. DEFLATE codes its literals, lengths and distances with Huffman codes and can rebuild the code tables for every block.

Huffman's limit is that every code is a whole number of bits, which amounts to rounding each symbol's probability to a power of two. Arithmetic coding gets very close to the theoretical limit, called Shannon entropy, but costs more computation. Asymmetric numeral systems (ANS), [proposed by Jarek Duda in 2013](https://arxiv.org/abs/1311.2540), aim for arithmetic coding's ratio at Huffman's speed. Zstandard uses a form of ANS called Finite State Entropy (FSE) for match lengths, literal lengths and offsets, and Huffman coding for literals. Brotli stays with Huffman but adds context modelling: the code used for the next literal depends on a context computed from the two bytes before it.

### Transforms that make data easier to compress

A transform shrinks nothing by itself. It rearranges or predicts values so that the matching and coding stages find more to work with:

- **PNG** passes each row of pixels through [one of five filters](https://www.w3.org/TR/png-3/) that predict a byte from its neighbours (left, above, upper left) and store the difference, then compresses the filtered rows with DEFLATE.
- **bzip2** runs each block through the [Burrows-Wheeler block-sorting transform (BWT)](https://sourceware.org/bzip2/manual/manual.html) before Huffman coding.
- **Delta encoding** stores the difference between consecutive values, so a column of timestamps or counters becomes a column of small numbers. [Apache Parquet defines it](https://parquet.apache.org/docs/file-format/data-pages/encodings/) for integer columns.
- **FLAC**, the lossless audio format standardized as [RFC 9639](https://www.rfc-editor.org/rfc/rfc9639.html) in December 2024, predicts each sample from the previous ones and stores only the residual (the prediction error) with Rice coding.

## How lossy compression works for images, audio and video

Lossy codecs follow a common pattern: predict, transform, quantize, then entropy code. [Google's documentation for lossy WebP](https://developers.google.com/speed/webp/docs/compression) walks through it. A block of pixels is predicted from the pixels already decoded around it, the difference (the residual) goes through a discrete cosine transform (DCT) that typically leaves many zero values, those values are quantized to a coarser scale, and the result is entropy coded. The same documentation points out that quantization is the only step where information is discarded; every other step can be inverted exactly.

![An image block passes through a gear that turns it into bars of different heights, then through an orange sieve that lets the tall bars through and drops the smallest ones before they are packed.](https://computese.com/images/blog/revolutionary-data-compression-algorithm/quantize.9352d44ed4-1536.webp)

*Only the quantizer throws information away; every other step of a lossy codec can be reversed exactly.*

The same pattern runs through the formats you meet every day:

- **Images.** [JPEG's core coding system](https://jpeg.org/jpeg/index.html) is a Huffman-coded, DCT-based lossy format. WebP adds block prediction and arithmetic coding, and Google reports that lossy WebP files are [25 to 34% smaller than comparable JPEGs](https://developers.google.com/speed/webp) at equivalent SSIM quality, and lossless WebP files 26% smaller than PNGs.
- **Video.** Codecs such as AV1 add [inter prediction](https://aomediacodec.github.io/av1-spec/): a frame is predicted from frames already decoded, and only the residual, the difference between the prediction and the real frame, is coded.
- **Audio.** Opus ([RFC 6716](https://www.rfc-editor.org/rfc/rfc6716.html), September 2012) combines linear prediction with a modified DCT and covers everything from 6 kbit/s narrowband speech to 510 kbit/s stereo music. For lossless audio, FLAC uses the same prediction and residual idea without discarding anything.

Because quantization loses information each time, keep a lossless master and encode every lossy copy from it. Re-encoding a JPEG or an MP4 quantizes an already degraded signal again.

## gzip vs Brotli vs Zstandard vs LZ4: the general-purpose algorithms compared

| Algorithm                 | Specification                                                                   | How it works                                                                        | Best at                                                           |
| ------------------------- | ------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
| DEFLATE (gzip, zlib, ZIP) | RFC 1951, May 1996                                                              | LZ77 with a 32 KiB window, Huffman coding                                           | Compatibility: the fallback everything reads                      |
| Brotli                    | RFC 7932, July 2016                                                             | LZ77, Huffman coding, context modelling, static dictionary, up to 16 MiB window     | Static web files compressed once at high quality                  |
| Zstandard (zstd)          | RFC 8878, February 2021                                                         | LZ77-style matching, Huffman and FSE, trained dictionaries                          | A general-purpose default across the speed range                  |
| LZ4                       | [LZ4 block format](https://github.com/lz4/lz4/blob/dev/doc/lz4_Block_format.md) | LZ77-type matching with no entropy stage                                            | Speed: [over 500 MB/s per core](https://lz4.org/), GB/s to decode |
| xz                        | .xz file format (XZ Utils)                                                      | LZMA-family compression; [XZ Utils builds on the LZMA SDK](https://tukaani.org/xz/) | Ratio, when compression time does not matter                      |
| bzip2                     | bzip2 reference implementation                                                  | Burrows-Wheeler transform, Huffman coding                                           | Existing `.bz2` archives                                          |

[Brotli's authors describe it](https://github.com/google/brotli) as similar in speed to DEFLATE with denser output. Zstandard covers a wide range: levels 1 to 19 by default (3 is the default level), 20 to 22 unlocked on request at a large cost in memory, and negative "fast" levels for speed. [Its manual gives a rule of thumb](https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md) that compression speed halves every two levels, while decompression speed stays roughly the same at every level, a property most LZ-family compressors share.

The Zstandard project [publishes a benchmark of fast settings](https://github.com/facebook/zstd) on the Silesia compression corpus, a public set of test files, run with lzbench on a Core i7-9700K:

| Compressor and level  | Ratio | Compression speed | Decompression speed |
| --------------------- | ----- | ----------------- | ------------------- |
| zstd 1.5.7, level 1   | 2.896 | 510 MB/s          | 1,550 MB/s          |
| Brotli 1.1.0, level 1 | 2.883 | 290 MB/s          | 425 MB/s            |
| zlib 1.3.1, level 1   | 2.743 | 105 MB/s          | 390 MB/s            |
| LZ4 1.10.0            | 2.101 | 675 MB/s          | 3,850 MB/s          |
| Snappy 1.2.1          | 2.089 | 520 MB/s          | 1,500 MB/s          |

Read it for the shape of the trade-off, not for your numbers. At level 1, zstd and Brotli reach nearly the same ratio, and zstd compresses nearly five times as fast as zlib. LZ4 gives up about a quarter of zstd's ratio for about 2.5 times its decompression speed, while Snappy, at a similar ratio to LZ4, decodes no faster than zstd. Higher levels move every tool toward better ratios and slower compression, and your own data will land somewhere else on the curve.

## Where data compression is used

Compression is rarely a product you buy on its own. It is a setting inside the web server, the file format, the database, the file system and the backup tool, and each has its own defaults.

### On the web: HTTP content encoding

HTTP negotiates compression per response. The browser lists the codings it can decode in the `Accept-Encoding` request header; the server compresses the body with one of them and names it in `Content-Encoding`; caches need `Vary: Accept-Encoding` so they do not serve a Brotli body to a client that asked for gzip. The registered codings for general use are `gzip`, `br` and `zstd`, plus the older `deflate` and `compress`.

As of September 2026, [MDN's compatibility data](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Encoding) lists Brotli support in all major browsers, and zstd in Chrome and Edge 123, Firefox 126 and Safari 26.3. [RFC 9659](https://www.rfc-editor.org/rfc/rfc9659.html) (September 2024) requires a zstd window of at most 8 MB for HTTP, because some browsers limit the memory they will spend on decoding.

In practice:

1. **Compress text, not media.** HTML, CSS, JavaScript, JSON, SVG and XML shrink well. JPEG, PNG, WebP, MP4 and ZIP files are already compressed, and MDN notes that compressing them again can make them larger.
2. **Pre-compress static files at build time** at the highest level, and serve them directly. [Apache's mod_brotli](https://httpd.apache.org/docs/2.4/mod/mod_brotli.html) (available since httpd 2.4.26) recompresses on every request unless you serve pre-compressed files, and [nginx's gzip_static module](https://nginx.org/en/docs/http/ngx_http_gzip_static_module.html) (not built by default) serves a ready-made `.gz` file instead.
3. **Compress dynamic responses at moderate levels.** mod_brotli's default quality of 5 on a 0 to 11 scale is meant as a reasonable balance for dynamic content; nginx's `gzip_comp_level` [defaults to 1](https://nginx.org/en/docs/http/ngx_http_gzip_module.html).
4. **Keep gzip enabled** as the fallback for older clients and scripts.
5. **Check what is served**, from outside:

```bash
curl -s -o /dev/null -D - -H 'Accept-Encoding: zstd, br, gzip' https://example.com/ \
  | grep -i -E 'content-encoding|vary'
```

A typical nginx configuration compresses on the fly and serves a pre-compressed `.gz` file when one exists. `gzip_types` adds types to `text/html`, which is always compressed, and responses shorter than `gzip_min_length` are left alone:

```nginx
gzip on;
gzip_comp_level 5;
gzip_min_length 1024;
gzip_types text/css application/javascript application/json image/svg+xml;
gzip_vary on;
gzip_static on;
```

Lighthouse checks for this in [its "Enable text compression" audit](https://developer.chrome.com/docs/lighthouse/performance/uses-text-compression), which flags any text response of 1.4 KiB or more, served without Brotli, gzip or deflate, that gzip would shrink by at least 10%. When a CDN sits in front of the site, compression usually happens there: Cloudflare, for example, [serves gzip, Brotli or Zstandard](https://developers.cloudflare.com/speed/optimization/content/compression/) to visitors depending on the browser's `Accept-Encoding`, the plan and any compression rules. Our [hosting and maintenance service](https://computese.com/services/hosting-maintenance/) covers CDN caching and image and asset optimization as part of looking after a site.

### In files, packages and archives

A `.gz` file is a DEFLATE stream with [a header and a CRC-32 checksum](https://www.rfc-editor.org/rfc/rfc1952.html). A Zstandard frame can end with an optional checksum taken from the XXH64 hash, while a Brotli stream carries no checksum at all, so its integrity check has to come from the container or the transport. ZIP records a method for every entry: method 8 is DEFLATE, and [version 6.3.10 of the specification](https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT) also defines methods for bzip2, LZMA, XZ and Zstandard. PNG, whose third edition became a W3C Recommendation in June 2025, always uses DEFLATE with a window of at most 32,768 bytes.

Package managers show the trade-off well because a package is compressed once and decompressed on every install. When Arch Linux [moved its packages from xz to zstd](https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/) at the end of December 2019, it reported that the total size of all packages grew by about 0.8% while decompression became about 1,300% faster.

### In databases and columnar formats such as Parquet

Analytical data compresses best when values of one column are stored together, because they look alike. Apache Parquet works in two stages. It first encodes each column: dictionary encoding stores each distinct value once and replaces the column with small integer keys, run-length encoding and bit-packing shrink those keys, and delta encoding suits integers that change slowly, such as timestamps. It then compresses each page with a codec such as SNAPPY, GZIP, BROTLI, ZSTD or LZ4_RAW ([the older LZ4 codec is deprecated](https://parquet.apache.org/docs/file-format/data-pages/compression/) because of an undocumented framing scheme). Where Parquet files sit in a full analytics stack is covered in our guide to [building data analytics software](https://computese.com/building-data-analytics-software/).

![A table of mixed rows is split into separate columns. One column of repeated shades becomes a small key of three swatches and a short strip of tokens, drawn in orange.](https://computese.com/images/blog/revolutionary-data-compression-algorithm/columns.d956036ef9-1536.webp)

*Values of one column look alike, so dictionary and run-length encoding shrink them before a general codec even starts.*

Databases make similar choices. [ClickHouse compresses columns with LZ4 by default](https://clickhouse.com/docs/sql-reference/statements/create/table) in self-managed deployments and with zstd in ClickHouse Cloud, and it lets you chain a specialized codec before a general one, for example `CODEC(Delta, ZSTD)` on a numeric column. PostgreSQL compresses large column values (its TOAST mechanism) with [its own pglz method by default](https://www.postgresql.org/docs/current/runtime-config-client.html), or with LZ4 when the server was built with LZ4 support and `default_toast_compression` or the column's setting asks for `lz4`.

If your [warehouse or lakehouse bill](https://computese.com/the-power-of-big-data-analytics/) grows faster than your data, file layout and codecs are among the first places to look. Our [data platform service](https://computese.com/services/data-platform/) chooses warehouse or lakehouse storage for your volumes, skills and budget, with Apache Iceberg or Delta Lake tables, partitioning and cost controls.

### In storage and backups

File systems compress block by block, transparently. In OpenZFS, `compression=on` [selects LZ4 on pools](https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html) with the `lz4_compress` feature enabled, zstd levels 1 to 19 are available when you want a better ratio, and changing the property affects only data written afterwards. Savings are rounded to whole disk sectors, so a block that saves less than one sector is stored uncompressed.

Backup tools add deduplication, which is a different thing: compression removes redundancy inside data, deduplication stores identical chunks only once across files and backups. BorgBackup [deduplicates chunks of the source data](https://borgbackup.readthedocs.io/en/stable/usage/help.html) first and then compresses each stored chunk, with LZ4 by default, zstd at levels 1 to 22 (3 if you do not choose), zlib, or lzma for low speed and high compression. Its `auto` mode tests each chunk with LZ4 and skips compression for incompressible data such as media files. For database dumps, `pg_dump` in PostgreSQL 18 [accepts gzip, LZ4 or zstd](https://www.postgresql.org/docs/current/app-pgdump.html), and its custom and directory formats compress with gzip by default.

Always compress before you encrypt. Encrypted output looks random, so a backup that is encrypted first will not compress afterwards.

## How to choose a compression algorithm

Work through the questions in this order:

1. **Can you lose information?** For media delivered to people, use a lossy codec at a quality you have checked by eye or ear. For everything else, lossless.
2. **Who pays, and how often?** Data compressed once and read many times (static web files, packages, published datasets) justifies slow, high levels, as the Arch Linux example shows. Data compressed on every request or write (API responses, logs, messages, database pages) needs fast levels.
3. **How much memory does the reader have?** Large windows and high levels raise memory on both sides. Zstandard's levels 20 to 22 need more memory to decompress, the `zstd` command-line tool limits decompression to 128 MiB of memory unless told otherwise, and browsers expect 8 MB windows at most.
4. **Who has to read it?** Browsers accept gzip, Brotli and, now, zstd. Partners and old scripts may accept only gzip. A Parquet file is only as portable as its codec's support in the engines that will read it.
5. **Measure on your data.** Zstandard's benchmark mode tests a range of levels on a sample file in one command:

```bash
zstd -b1 -e19 sample.json
```

| Situation                               | Good default                                                  | Why                                        |
| --------------------------------------- | ------------------------------------------------------------- | ------------------------------------------ |
| Static web files (CSS, JavaScript, SVG) | Brotli at quality 11, plus gzip; zstd where supported         | Compressed once, served many times         |
| Dynamic HTTP responses and APIs         | zstd at a low level or Brotli around 5; gzip fallback         | CPU is spent on every request              |
| Logs, events, messages between services | zstd at low levels, or LZ4                                    | Fast in both directions, streams well      |
| Analytics files (Parquet)               | ZSTD; Snappy or LZ4_RAW when scan speed dominates             | Column encodings do much of the work first |
| File systems                            | LZ4, or zstd at a low level                                   | Every read and write pays the cost         |
| Backups and archives                    | zstd at a mid to high level; xz when size is all that matters | Written once, restored rarely              |
| Many small, similar records             | zstd with a trained dictionary                                | Small inputs have no history to learn from |

> [!TIP]
> Published benchmarks show the shape of the trade-off, not your numbers. Before you standardize on a codec or level, run two or three candidates against a real sample of your own data and compare ratio, speed and memory.

## What is new in data compression

Claims of a "revolutionary" compression algorithm deserve a careful read, because nothing beats the counting argument: no lossless algorithm wins on all data, so real progress comes from exploiting specific kinds of data or specific delivery paths. As of September 2026, these are the developments worth knowing:

- **Zstandard in every major browser.** With Safari 26.3, zstd content encoding works in Chrome, Edge, Firefox and Safari, after Chrome shipped it in version 123.
- **Compression dictionary transport.** [RFC 9842](https://www.rfc-editor.org/rfc/rfc9842.html), a Standards Track RFC published in September 2025, lets a server mark a response as a dictionary for later requests. A new version of a JavaScript bundle can then travel as a small delta against the version the browser already holds, using the `dcb` (Brotli) or `dcz` (Zstandard) codings. MDN lists support in Chrome and Edge 130; Firefox has it in Nightly and, from version 145, behind a preference, and Safari not at all.
- **Format-aware compression.** [Meta released OpenZL](https://engineering.fb.com/2025/10/06/developer-tools/openzl-open-source-format-aware-compression-framework/) in October 2025. You describe the shape of your data (rows, columns, fields), a trainer searches for the transforms that compress it best, and one universal decoder reads every file by following the recipe embedded in it. When OpenZL does not understand the input, it falls back to zstd.
- **Better encodings for floating-point columns.** ALP (Adaptive Lossless floating-Point), adapted from a SIGMOD 2024 paper, is now in [the Parquet specification](https://parquet.apache.org/docs/file-format/data-pages/encodings/) and, as of August 1, 2026, marked Preview: the format is stable, but reader support is still being built, so enable it only when every reader you use supports it. It turns decimal-like floats such as prices and sensor readings into integers before packing them.
- **Learned image compression.** JPEG AI, [published as ISO/IEC 6048-1:2025](https://jpeg.org/jpegai/index.html), is the first international image coding standard based on an end-to-end learning-based approach.

When you read about a new algorithm with remarkable ratios, ask three questions: on which data was it measured, against which baseline, and at what speed and memory cost.

## Security risks of compression

Compression parses untrusted input and changes the size of what it outputs, and both create risks:

- **Decompression bombs.** A small file can expand into an enormous output. [MITRE classifies the weakness as CWE-409](https://cwe.mitre.org/data/definitions/409.html), and its example is a small ZIP file that decompresses into a very large amount of data. Cap the output size and memory of every decompression of untrusted input, and treat uploaded archives like any other untrusted input, as our [secure coding checklist](https://computese.com/best-practices-for-secure-coding/) describes.
- **Compression side channels.** When secret data and attacker-controlled data are compressed together and then encrypted, the compressed length leaks information. The CRIME attack used compression at the TLS layer, and [TLS 1.3 removed compression](https://www.rfc-editor.org/rfc/rfc8446.html) from the protocol. BREACH uses HTTP body compression, and [RFC 7457](https://www.rfc-editor.org/rfc/rfc7457.html) notes there is no mitigation at the TLS level, so applications have to act. The RFC's example is randomizing CSRF tokens; more generally, stop compressing the responses that reflect user input next to secrets.
- **Size leaks in stored data.** Compressed sizes can reveal what a file is even when its content is encrypted. BorgBackup offers an `obfuscate` mode that obscures compressed chunk sizes for this reason.

> [!WARNING]
> Do not turn off compression across a site to avoid BREACH. The risk sits in specific responses that mix secrets with reflected input; fix those, and keep compression on for everything else.

## Key terms
- **Lossless compression**: Compression that decodes to exactly the original bytes. Required for text, code, databases, logs, backups and anything a program parses.
- **Lossy compression**: Compression that decodes to an approximation of the input, discarding detail people are unlikely to notice. Used for photos, audio and video.
- **Compression ratio**: Original size divided by compressed size. A ratio of 2.9 means the compressed data takes about a third of the original space.
- **LZ77**: The dictionary method published by Ziv and Lempel in 1977: a repeated string is replaced by a back-reference giving how far back it appeared and how long it is.
- **Sliding window**: The span of recent data an LZ77-style compressor can refer back to. DEFLATE's window is 32 KiB; larger windows find more repeats but need more memory to decode.
- **Entropy coding**: Coding symbols so that frequent ones take fewer bits. Huffman coding, arithmetic coding and asymmetric numeral systems (ANS) are the main families.
- **Compression dictionary**: Data both sides hold in advance so that even the first bytes can refer to it: Brotli's built-in static dictionary, Zstandard's trained dictionaries or a dictionary sent over HTTP.
- **Quantization**: The step in a lossy codec that rounds transformed values to a coarser scale. It is where information is actually discarded.
- **Content-Encoding**: The HTTP header naming the compression applied to a response body (gzip, br, zstd), chosen from the codings the client lists in Accept-Encoding.
- **Decompression bomb**: A small compressed input built to expand into an enormous output and exhaust memory or disk. Classified as CWE-409.

## Common questions

### What is the best data compression algorithm?

There is no single best one. Zstandard is a strong general-purpose default across the speed range, Brotli suits static web files compressed once, LZ4 suits cases where speed matters more than size, and xz suits archives where only size matters. Photos, audio and video need lossy codecs instead.

### Which compression algorithm has the highest compression ratio?

Among common tools, the highest ratios come from the slowest settings: xz, Zstandard's levels 20 to 22 and Brotli at quality 11 all trade a lot of compression time (and, for zstd, memory) for size. Beyond that, gains come from knowing the data: columnar encodings, trained dictionaries or format-aware tools such as OpenZL.

### Is gzip still worth using?

Yes, as the fallback every client understands. For new work, Brotli or Zstandard usually give smaller output at the same speed, but keep gzip enabled for older clients, scripts and partner systems that only accept it.

### Can you compress data that is already compressed or encrypted?

Rarely with any benefit. Encrypted data looks random and does not compress, and compressing JPEG, MP4 or ZIP files again can even make them larger. Compress first, then encrypt.

### What is the difference between compression and deduplication?

Compression removes redundancy inside a piece of data. Deduplication stores identical chunks only once across files or backups. Backup tools such as BorgBackup do both: they deduplicate chunks of the source data, then compress each chunk they keep.

### Is there a new compression algorithm that beats Zstandard?

Not for all data, and none can be, because no lossless method shrinks every input. Recent work wins in specific cases: dictionaries shared over HTTP (RFC 9842), format-aware compression (OpenZL), floating-point encodings in Parquet (ALP) and learned image codecs (JPEG AI).

## Sources
1. [RFC 1951: DEFLATE Compressed Data Format Specification version 1.3](https://www.rfc-editor.org/rfc/rfc1951.html), IETF
2. [RFC 7932: Brotli Compressed Data Format](https://www.rfc-editor.org/rfc/rfc7932.html), IETF
3. [RFC 8878: Zstandard Compression and the 'application/zstd' Media Type](https://www.rfc-editor.org/rfc/rfc8878.html), IETF
4. [RFC 9110: HTTP Semantics](https://www.rfc-editor.org/rfc/rfc9110.html), IETF
5. [Zstandard README: benchmarks and dictionary compression](https://github.com/facebook/zstd), Meta (GitHub)
6. [Asymmetric numeral systems: entropy coding combining speed of Huffman coding with compression rate of arithmetic coding](https://arxiv.org/abs/1311.2540), arXiv (Jarek Duda)
7. [Portable Network Graphics (PNG) Specification (Third Edition)](https://www.w3.org/TR/png-3/), W3C
8. [bzip2 and libbzip2, version 1.0.8](https://sourceware.org/bzip2/manual/manual.html), Sourceware
9. [Parquet: Encodings](https://parquet.apache.org/docs/file-format/data-pages/encodings/), Apache Software Foundation
10. [RFC 9639: Free Lossless Audio Codec (FLAC)](https://www.rfc-editor.org/rfc/rfc9639.html), IETF
11. [WebP Compression Techniques](https://developers.google.com/speed/webp/docs/compression), Google for Developers
12. [JPEG 1](https://jpeg.org/jpeg/index.html), JPEG Committee
13. [WebP: An image format for the Web](https://developers.google.com/speed/webp), Google for Developers
14. [AV1 Bitstream & Decoding Process Specification](https://aomediacodec.github.io/av1-spec/), Alliance for Open Media
15. [RFC 6716: Definition of the Opus Audio Codec](https://www.rfc-editor.org/rfc/rfc6716.html), IETF
16. [LZ4: Extremely fast compression](https://lz4.org/), LZ4 project
17. [LZ4 Block Format Description](https://github.com/lz4/lz4/blob/dev/doc/lz4_Block_format.md), LZ4 project (GitHub)
18. [XZ Utils](https://tukaani.org/xz/), Tukaani Project
19. [Brotli README](https://github.com/google/brotli), Google (GitHub)
20. [zstd(1) manual](https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md), Meta (GitHub)
21. [Content-Encoding header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Encoding), MDN Web Docs
22. [RFC 9659: Window Sizing for Zstandard Content Encoding](https://www.rfc-editor.org/rfc/rfc9659.html), IETF
23. [Apache Module mod_brotli](https://httpd.apache.org/docs/2.4/mod/mod_brotli.html), Apache HTTP Server Project
24. [Module ngx_http_gzip_static_module](https://nginx.org/en/docs/http/ngx_http_gzip_static_module.html), nginx
25. [Module ngx_http_gzip_module](https://nginx.org/en/docs/http/ngx_http_gzip_module.html), nginx
26. [Enable text compression](https://developer.chrome.com/docs/lighthouse/performance/uses-text-compression), Chrome for Developers
27. [Content compression](https://developers.cloudflare.com/speed/optimization/content/compression/), Cloudflare Docs
28. [RFC 1952: GZIP file format specification version 4.3](https://www.rfc-editor.org/rfc/rfc1952.html), IETF
29. [APPNOTE.TXT: .ZIP File Format Specification 6.3.10](https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT), PKWARE
30. [Now using Zstandard instead of xz for package compression](https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/), Arch Linux
31. [Parquet: Compression](https://parquet.apache.org/docs/file-format/data-pages/compression/), Apache Software Foundation
32. [CREATE TABLE (column compression codecs)](https://clickhouse.com/docs/sql-reference/statements/create/table), ClickHouse Docs
33. [PostgreSQL 18: Client Connection Defaults (default_toast_compression)](https://www.postgresql.org/docs/current/runtime-config-client.html), PostgreSQL Global Development Group
34. [zfsprops.7: native properties of ZFS datasets](https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html), OpenZFS
35. [Borg documentation: borg help compression](https://borgbackup.readthedocs.io/en/stable/usage/help.html), BorgBackup
36. [PostgreSQL 18: pg_dump](https://www.postgresql.org/docs/current/app-pgdump.html), PostgreSQL Global Development Group
37. [RFC 9842: Compression Dictionary Transport](https://www.rfc-editor.org/rfc/rfc9842.html), IETF
38. [Introducing OpenZL: An Open Source Format-Aware Compression Framework](https://engineering.fb.com/2025/10/06/developer-tools/openzl-open-source-format-aware-compression-framework/), Engineering at Meta
39. [JPEG AI](https://jpeg.org/jpegai/index.html), JPEG Committee
40. [CWE-409: Improper Handling of Highly Compressed Data (Data Amplification)](https://cwe.mitre.org/data/definitions/409.html), MITRE
41. [RFC 7457: Summarizing Known Attacks on TLS and DTLS](https://www.rfc-editor.org/rfc/rfc7457.html), IETF
42. [RFC 8446: The Transport Layer Security (TLS) Protocol Version 1.3](https://www.rfc-editor.org/rfc/rfc8446.html), IETF