ERC-3668 CCIP Read: offchain data lookup (CCIP Read—Secure offchain data retrieval)
Final · Other · created 2020-07-19
Defines a common procedure in which a contract signals by an error that the data it needs is offchain, the client fetches the data from a specified server and passes it back to the contract, and the contract verifies it.
Why it is needed
Because storing data and sending transactions on Ethereum is expensive, contract developers have moved data offchain using methods such as hashes, Merkle trees, and L2s. The methods differ, but they share one trait: enough information remains on chain to verify the data kept outside. The problem so far is that each app has built its own method. With a single storage method this is workable, but in an environment where each user picks a different storage or L2, clients must support every method separately, which quickly becomes unmanageable. If there is a common procedure for a contract to fetch offchain data, clients do not need to know which storage is used, and new apps can be built without knowing the contract's internals. The original document gives examples such as a contract that keeps an airdrop list in a Merkle trie, viewing token information held on an L2 as if it were an L1 token, and delegating ENS domains to multiple L2s.
What to implement
error OffchainLookup(address sender, string[] urls, bytes callData, bytes4 callbackFunction, bytes extraData): An error a contract reverts with when a function that needs offchain data is called. It carries which server to query, what to ask, and which function to return the result to.function balanceOf(address addr) public view returns(uint balance);: The example function from the original document. Instead of answering directly, it throws an OffchainLookup error to start a gateway lookup.function balanceOfWithProof(bytes calldata response, bytes calldata extraData) public view returns(uint balance);: The example callback from the original document. It receives the gateway response and extraData, verifies them, and returns the result in the same format as the original function.
In depth
The five fields of the OffchainLookup error
A function that needs off-chain data must revert with the error OffchainLookup(address sender, string[] urls, bytes callData, bytes4 callbackFunction, bytes extraData). sender is the address of the contract that threw the error, and the client uses it to tell whether the contract it called threw the error or whether it came up from a nested call. urls is a list of gateway URL templates, and if it is the empty array [], the client must choose the URL template itself. The order of attempts is up to the client, but the contract should list URLs in order of importance. callData is the data to send to the gateway and is a byte sequence with no meaning to the client. ABI encoding is usually used, but the contract and the gateway decide the format. callbackFunction is the 4-byte selector of the original contract function that will receive the result. extraData is additional data the callback needs, and the client must keep this value and pass it to the callback unchanged.
Callback function requirements
The contract must also implement a callback function that parses and verifies the gateway response. The name is free, but the parameters must be (bytes response, bytes extraData) and the return type must match that of the original function that threw OffchainLookup. When the client succeeds in calling the gateway, it calls the callback with the value the gateway returned as response and the value carried in the error as extraData. The callback can throw OffchainLookup again to continue the lookup, but note that the limit on repetitions differs between clients. For read calls, the callback's return value is returned to the user as if it were the return value of the function first called. The reason the source example puts addr into both callData and extraData is that trusting only the value the gateway returns in its response would let it slip in an answer to a different question.
URL templates and GET/POST rules
URL templates can use two substitution variables, {sender} and {data}. {sender} is replaced with the lowercase 0x-prefixed hex address, and {data} with the 0x-prefixed hex callData. If the template contains {data}, the client sends a GET request. If not, it sends a POST request with Content-Type application/json and a JSON body containing the two fields data and sender. The gateway must respond with application/json, with the 0x-prefixed hex result in the data field. On failure it must use an appropriate HTTP status code. For example, 404 for an unsupported sender, 400 for a malformed callData, and 500 for an internal server error. If a 4xx or 5xx response is application/json, it must carry a human-readable error message in the message field. The client must support both GET and POST, and the gateway may implement only the one it needs. This specification defines handling of HTTPS URLs only.
Client lookup procedure
The client first calls the contract as usual and, if the call succeeds, returns the result unchanged. Errors other than OffchainLookup are passed through as usual. If the error is OffchainLookup, it decodes the five fields and ends with an error if sender differs from the address of the contract it called. It builds the URLs and makes requests, preferring earlier entries in the list. If the response is 400 to 499, it ends with an error, and if it is 500 to 599, it tries the next URL and stops when there are no more. On success, it replaces the call data with the callbackFunction selector followed by the ABI-encoded response data and extraData, and repeats the procedure from the first step. If the Content-Type of a 4xx or 5xx response is not application/json, the client must not try to interpret the body as JSON. Because a single call can trigger several lookups, the client must set a limit on the number of lookups, and that limit must be at least 4.
Handling OffchainLookup in nested calls
When a CCIP Read-aware contract calls another contract that may also implement CCIP Read, it must catch every OffchainLookup thrown by the other contract and revert with a different error if sender differs from the other contract's address. It may also replace every OffchainLookup with a different error, which avoids implementing nested support but makes nested lookups impossible. The default nested-call error propagation in Solidity must not be left as is. If A calls B, which does not know CCIP, and B calls A back so that A throws OffchainLookup, then B and A propagate it in turn, and an error whose sender matches A and looks valid reaches the client, but in reality only the inner call completes. The NestedLookup example in the source shows how to catch the other contract's error, check sender, and then rethrow it bundled with its own aCallback selector, the other contract's address, the other contract's callback, and the other contract's extraData.
It can be used for transactions too
The specification is written with read calls such as eth_call in mind, but it applies as is to transactions that need off-chain data. If OffchainLookup is received while pre-executing a transaction with eth_estimateGas or eth_call, the client goes through the same lookup procedure and then sends a transaction instead of a call in the final step. The source explains that this suits uses such as on-chain claims backed by off-chain proof data. However, the security considerations recommend that client libraries disable CCIP Read for transactions by default and have the caller enable it explicitly.
Design rationale: revert, sender, extraData, GET and POST
The error-revert approach was chosen because an existing function can be converted to support off-chain data without changing its name or ABI. Marking the call type in the ABI would either give the same name a different return value and cause decoding errors, or require duplicating whole functions such as offchainBalanceOf. The sender address is passed to the gateway so that generic gateways can be built. The gateway can query the original contract to get the information it needs, so one gateway can serve many contracts that use the same interface. extraData exists because contracts do not keep state between calls, so it carries context across the two calls and is used to check that the question the gateway answered is the original question. GET is simple enough that a gateway can be built with a static-file HTTP server or IPFS, but URL length is limited to 2 kilobytes, so POST is offered as an option for more complex uses.
Compatibility and status
Existing contracts that do not use this specification are unaffected. Clients can add CCIP Read support to all contract calls with no extra burden or incompatibility. Conversely, a contract that needs CCIP Read does not work with a client that has not implemented it, and when such a client calls it, the exception thrown by the contract is passed to the user as is. The status of this proposal is Final, meaning its content is settled.
Things to watch out for
- Verifying the gateway response: To prevent a malicious gateway from causing wrong results or side effects, the contract must put enough information in extraData to check the relevance and validity of the response. If the lookup used the address received in the original call, it must put that address in extraData and check in the callback that the response is not an answer to a different question. The response itself must also be verified. The method varies by application, such as verifying an inclusion proof against L2 or Merkleized state, or verifying a signature from a trusted signer.
- Re-validating extraData in the callback: A malicious client can tamper with extraData in a transaction, so the permission and consistency checks done in the first call must be repeated in the callback. If a transaction can be executed only by certain accounts, that permission check must be done in the callback. Checking in the first call and putting the authorized address in extraData is not enough.
- Identifying users through HTTP requests: CCIP Read makes the user's browser send HTTP requests to addresses chosen by the contract, so it can be abused to link a wallet address to an IP address. When resolving an ENS name, the attacker learns only which IP is looking up which name, so the impact is small, but an NFT transfer transaction gives them everything needed to find out the wallet's IP address.
- Recommendations for libraries, apps, and wallets: Client libraries should provide hooks that can proxy or block CCIP Read calls and make allow and block lists easy to manage. CCIP Read for transactions should be off by default and enableable per contract, per domain, or globally. App developers should not include a from address in read calls that may run untrusted code. Wallets that fetch user information through token lookups and similar should disable transaction CCIP Read and call without from, or operate a proxy on behalf of the user. The source says that following these precautions makes it safe, so CCIP Read itself should not be disabled by default.