⛓️ Fast & resilient blockchain calls
Written Nov 25, 2022.
As part of the Web 3 project I'm building, I needed reliable and fast RPC access to 10+ blockchains, parsing in realtime all the blocks 24/7.
The idea is to offer realtime, multi-chain monitoring:
I need to parse tx, logs and contract data for each block.
On some chains like Arbitrum there's ~ 10 blocks/s & ~200 tx/block.
That's a lot of requests, hitting the rate limit for public nodes and going well above the free plan on Alchemy, Infura & co.
I could've ran my own nodes, but I want flexibility to add more chains w/o server overheads.
So I built this meta provider for Ethers.js:
https://github.com/vincelwt/super-provider
I feed it with a lot of rpc endpoints (via WebSocketProvider & JsonRpcBatchProvider).
Then it:
- benchmark providers speed regularly
- keep only the fastestt
- split requests between them to prevent overload
- handles rate-limiting and errors with smart retries
It handles my ~5k req/s beautifully.
Now I get:
- ✅ Super fast queries
- ✅ 0 data lost due to rate limiting
- ✅ No need to run servers or pay $$ for dedicated nodes
That's how I do near-realtime block scanning on ~10 chains.
Other benefits of decentralising the providers:
- you get more privacy as requests are split between node providers
- resilience, if Alchemy goes down tomorrow the platform will run as if nothing happened.
Check it out if you ever run in limitations with public/shared RPC's or want to make your dapp more resilient.
Also, I added a 🐇 'parallel' mode when absolute speed is needed:
Instead of cycling through providers, it sends requests in parallel to the fastest ones and returns the first reply that comes back. (Tradeoff is higher network usage)