Datawaves
Search…
Architecture

Exporting the Blockchain

Datawaves extracts blockchain data from blockchain network through node RPC. Datawaves will load the exported the raw data into an internal message queue for further processing. See the data models for the raw data Datawaves exported from the blockchain network..

Transforming and Normalizing Web3 Data

Raw data are challenging to understand and use. To simplify downstream user-defined pipelines, Datawaves employs internal pipelines to create higher-level structured semantic data streams.
For example, Datawaves can use contract ABI to decode events emitted by a particular smart contract or decode events according some token standards like ERC20 transfer.

Self-service Transformation and Integration

Datawaves provides two fundamental concepts, pipeline, and connection, to help you build pipelines with SQL and integrate them with your databases. A pipeline is a streaming SQL query that processes data from streams and writes the results to an output stream. A connection allows data to flow between a Datawaves stream and an external system like a database, messaging system, or storage system.

Idempotent and Resilient Data Processing

The above stages are processed in parallel and are decoupled by stream. When a connector fails to load data due to transient errors or unavailability, Datawaves doesn't have to re-ingest data from the blockchain or recompute the data pipeline. This limits the impact of destination outages and improves Datawaves reliability.
Copy link
On this page
Exporting the Blockchain
Transforming and Normalizing Web3 Data
Self-service Transformation and Integration
Idempotent and Resilient Data Processing