EIP-706 DEVp2p Snappy compression
Final · Networking & nodes · EIP status Final · created 2017-09-07
Compresses all network messages exchanged between Ethereum nodes using the Snappy algorithm to reduce traffic. Data traffic drops by 60 to 80 percent for initial synchronization.
Why it is needed
The basic network protocol Ethereum nodes use to communicate with each other, DEVp2p, originally applied no compression at all. Block and transaction data compress well, yet exchanging them uncompressed wasted enormous bandwidth across the network, slowing both initial sync, when a node first starts up and downloads the blockchain, and normal operation. In practice, syncing the main network (as of block 4,248,000) with the Geth client via fast sync required 1.01GB upload and 33.59GB download. Enabling compression at the message content level cuts the download for the same sync to 13.46GB, and on the Rinkeby test network the download drops from 2.51GB to 463.65MB. Rather than implementing compression separately in each subprotocol (eth, les, bzz), handling it once at the DEVp2p layer below lets every protocol benefit without modification.
What changes
Today
- Messages between nodes are transmitted raw, without compression
- Mainnet fast sync requires 33.59GB of download (as of block 4,248,000)
- Any subprotocol wanting compression, such as eth, must implement it separately
After EIP-706
- After the handshake, all message content is compressed with Snappy before transmission
- The mainnet download under the same conditions drops to 13.46GB
- All subprotocols, including eth, les, and bzz, get the compression benefit without modification
- Compatibility is preserved by communicating with older (v4) nodes using the existing uncompressed scheme
Who is affected
- Node operators: Network usage for both initial sync and normal operation drops significantly, with download traffic in particular falling by 60 to 80 percent.
- Client teams: The DEVp2p version must be raised to 5, compression and decompression steps must be added, and nodes must fall back to the existing scheme when communicating with peers that only support v4.
- Subprotocol developers: Protocols such as eth, les, and bzz gain compression at the transport layer with no changes of their own, reducing the need for each to separately optimize its traffic.
- Ordinary users: Protocol behavior stays the same, so there is no perceptible change, but the lower barrier to running a node benefits overall network health.
In depth
Version negotiation: v4 to v5
The advertised DEVp2p version number is raised from 4 to 5. If a peer announces support for only version 4 during the handshake, the existing uncompressed protocol continues to run; if the peer announces version 5 or higher, compression is enabled. The handshake message itself, the means by which the common version is negotiated, is never compressed. This structure provides full backward compatibility, so the network does not break even when upgraded and legacy clients are mixed together.
The compression step inserted into the send and receive path
On send, a Snappy compression step is inserted immediately before the DEVp2p message is encrypted. A message consists of {Code, Size, Payload}; the original Payload is compressed with Snappy and stored in the same field, Size is updated to the compressed length, and the message is then encrypted and transmitted exactly as before. On receive, a symmetric decompression step is inserted immediately after decryption, restoring the Payload and resetting Size to the original length. Because the encryption and decryption logic remains entirely oblivious to whether compression occurred, changes to the existing transport code are minimized. Since DEVp2p is already a message-oriented protocol, the Snappy framing format is not used.
DoS prevention: the decompressed size is also capped at 16MB
DEVp2p message length is limited to 24 bits, for a maximum of 16MB. Once compression is introduced, a small compressed payload can expand to something far larger when decompressed, so blindly decompressing any received message would expose nodes to memory exhaustion attacks. The Snappy stream encodes the decompressed length as a little-endian varint at the very start (up to 2^32 - 1), so the resulting size can be known without actually decompressing it into memory. The proposal applies the same existing 16MB limit to the decompressed size and discards any message that exceeds it. This preserves the guarantees DEVp2p already provided, so higher-level application protocols see no surprises.
Why the DEVp2p layer instead of eth
Adding compression to an individual protocol like eth was considered, since it would allow finer optimization of when to compress, but it was discarded because it mixes transport-layer encoding into application-layer logic, complicates message specifications with compression details, and requires repeated coordination among clients for each of eth, les, shh, and bzz. Advertising separate compressed protocol variants, like xyz-compressed, would avoid coordination but was also discarded because it would lead to a proliferation of client-specific protocol advertisements on the network and would still require an EIP specification for interoperability. Limiting only the pre-compression size while leaving the decompressed size unbounded was likewise rejected, since it would force higher-level protocols to inspect large messages themselves and require deferred decompression.
Benchmarks, implementation, and test vectors
On the main network (block 4,248,000), usage dropped from 1.01GB upload / 33.59GB download to 1.01GB / 13.46GB, and on the Rinkeby testnet (block 852,000) from 55.89MB / 2.51GB to 46.21MB / 463.65MB. The reference implementation is in go-ethereum pull request 15106. Because Snappy allows more than one valid encoding for the same input, compression output can differ slightly between implementations, but they must remain cross-decompressible; test vectors are provided showing the RLP of Rinkeby block #272621 (about 3MB) compressed to roughly 70KB by both the Go and Python Snappy libraries. Snappy also supports uncompressed literals up to 4GB, leaving room for the optimization of storing already-compressed or encrypted data as is.
Status: Final
This EIP was proposed on September 7, 2017, and has been finalized with status Final. It is a completed specification already in effect on the network as part of DEVp2p version 5, and its contents will not change further.