Cloudflare prototypes Zstandard compression to expand cache capacity

Cloudflare has prototyped a system called Cache Transcoding that uses Zstandard compression inside its Pingora proxy to expand effective cache capacity. The approach encodes eligible assets with zstd before writing them to disk, then decodes them before serving to clients, trading a small CPU increase for significant storage and bandwidth savings.

In initial testing, eligible assets shrank to about a third of their original on-disk size on average. The prototype only compresses compressible text responses of at least 4 KiB that arrive uncompressed, avoiding already-compressed media. The estimated extra CPU cost was a few percent under the traffic and reuse assumptions tested.

The smaller representation also reduces data transferred between data centres via Tiered Cache. Cloudflare said the encoding cost is paid once when an asset enters the cache, while storage and bandwidth savings continue on every reuse.