Data & ML · Advanced
Vectorized Columnar Query Engine
Build a small analytical query engine that parses SQL, plans and optimizes it, and executes vectorized scans and joins over columnar Parquet files with predicate pushdown.
You build a miniature analytical query engine in Rust that reads columnar Parquet files, parses SQL into a logical plan, applies rule-based optimizations such as predicate pushdown and projection pruning, and executes the plan using vectorized Arrow record batches. The project gives you a ground-level understanding of how systems like DuckDB and DataFusion achieve sub-second query times on hundreds of millions of rows without loading entire files into memory. It is worth building because almost every fast data pipeline ultimately bottoms out at these same primitives, and implementing them yourself demystifies the "magic" of modern OLAP engines.
What you build
- Parse a subset of SQL (SELECT, WHERE, GROUP BY, ORDER BY, LIMIT) into a typed AST using sqlparser-rs
- Convert the AST into a logical plan tree (Scan, Filter, Project, Aggregate, Sort, Limit nodes)
- Apply at least two optimizer rules: predicate pushdown past projections and projection pruning to drop unused columns before scan
- Execute plans over Apache Arrow RecordBatches using columnar, SIMD-friendly kernels for filter, hash aggregate, and sort-merge join
- Read Parquet files with column and row-group pruning so unneeded data is never deserialized from disk
- Print a human-readable EXPLAIN plan showing the logical and physical plan trees
- Measure and display per-node row counts and elapsed time so you can see where query time is spent
What it teaches
- How columnar storage (Parquet row groups, column chunks) differs from row-oriented storage and why it matters for analytical workloads
- The logical plan to physical plan pipeline: parsing, planning, rule-based optimization, and physical operator selection
- Predicate pushdown and projection pruning as concrete optimization rules you implement and test
- Vectorized execution over Arrow RecordBatches: applying filters and aggregations as bulk array operations rather than row-by-row interpretation
- How the Rust ownership and lifetime system interacts with zero-copy Arrow buffer sharing across operators
- Reading Parquet metadata without deserializing data: using column statistics and row-group skipping to minimize I/O
How it works
- 1
SQL Text
- Raw SQL string input to the CLI
- 2
Parse
- sqlparser-rs produces a typed AST
- LogicalPlanner converts AST to LogicalPlan nodes
- 3
Optimize
- Predicate pushdown moves filters toward scans
- Projection pruning drops unused columns
- 4
Execute
- ParquetScan reads row groups with column pruning
- Filter, Aggregate, Sort operate on Arrow RecordBatches
- 5
Result
- Printed table output + EXPLAIN plan with per-node timing
Sign in to open the build guide
Free account. Get the step-by-step build and every resource link.
Take it further
- Add a parallel execution layer using Rayon to process Parquet row groups concurrently and measure the speedup on a multi-core machine
- Implement a hash join executor that builds an in-memory hash table from the smaller side and probes it with Arrow record batches from the larger side, enabling two-table joins
- Add basic statistics-based optimization: read Parquet row-group min/max statistics and skip entire row groups whose ranges cannot satisfy the predicate
