High Performance Ethereum Indexer
Syncs events from Ethereum and indexes them for fast retrieval. This is useful and often essential for production Dapps.
- Support for 6 different persistence stores
- Includes "single-node friendly" stores such as flat-file JSON
- Comprehensive error retry logic when communicating with Ethereum node
- Maximum throughput via JSON RPC using batched parallel fetching
Todo
- Keep track of blockNumbers in persistence for fast-resume
- Benchmark documentation
- Support going forward syncing for Ethereum
- NPM package
- Improved failsafes for data consistency
- Split up stores into separate dependencies
Supported Indexing Stores and Benchmarks
The following indexing implementations have been benchmarked on an Ethereum blockchain section with a relatively large number of events (EtherDelta exchange blocks 4800000
- 5002718
) on a single 8-core 2017 Macbook Pro:
Store Name | Store ID | Events Per Second | Blocks Per Second |
---|---|---|---|
Local Memory | memory |
4072.7/s | 464.5/s |
Redis | redis |
2049.55/s | 235.25/s |
Local Flat File | file |
1798.75/s | 201.75/s |
MongoDB | mongodb |
795.5/s | 94.6/s |
Elasticsearch | elasticsearch |
452.25/s | 57.1/s |
LevelDB | level |
207.35/s | 26.95/s |
Install Dependencies
yarn install
Testing & Linting
yarn test
yarn lint
Integration tests (requires synced ethereum node at localhost:8545):
yarn test:integration
Example: Sync trading and balance events from decentralized trading contract: EtherDelta
Contract address: https://etherscan.io/address/0x8d12a197cb00d4747a1fe03395095ce2a5cc6819
See examples/etherdelta/sync.js
.
Define which events to index on which keys:
const indexing = events: Withdraw: keys: 'user' Trade: keys: 'tokenGive' 'tokenGet' 'get' 'give' ;
Then create a store:
const store = indexing '/tmp/etherdelta.db';
Now boot up the indexer:
const indexer = store EtherdeltaABI '0x8d12a197cb00d4747a1fe03395095ce2a5cc6819';await indexer;
Once indexing is complete events can be retrieved as follows:
const events = await store;
Directory Structure
package.json
- Configure dependenciesdist/*
- Files generated by babelsrc
- All source codesrc/*/__tests__
- Unit testsintegration/__tests__
- Integration tests