Skip to content

Introduction ​

Arneb is a distributed SQL query engine built in Rust — a Trino alternative designed for federated queries across heterogeneous data sources.

Key Features ​

  • Arrow-Native: All intermediate data uses Apache Arrow columnar format. No row-by-row processing.
  • PostgreSQL Compatible: Full Simple and Extended Query wire protocol (v3) and a queryable pg_catalog. Connect with psql, DBeaver, JDBC drivers, psycopg2, psycopg 3, SQLAlchemy or dbt-postgres — no special client needed.
  • Trino Client Compatible: Trino's client REST protocol for the trino CLI, Trino JDBC, trino-python-client, Superset's SQLAlchemy dialect and dbt-trino. See Trino Clients, and Client Compatibility for what CI checks and what is not supported yet.
  • Federated Queries: Query data across CSV files, Parquet files, S3/GCS/Azure object stores, Hive Metastore catalogs (including Iceberg tables) and PostgreSQL databases from a single SQL interface, and join across them in one statement. See Cross-source joins.
  • Distributed Execution: Coordinator-worker architecture with Apache Arrow Flight RPC for high-throughput data exchange between nodes.
  • Async Streaming: Operators return async record batch streams, enabling pipelined execution without materializing full intermediate results.
  • Pushdown Optimization: Filters, projections, and limits are pushed into connectors when supported.
  • Low Memory: No JVM and no committed heap; an idle three-node cluster holds about 70 MB, and TPC-H peaks average 0.14× of Trino's at SF10.
  • Secure: TLS and password authentication on pgwire and the Trino listener, and a shared secret plus TLS on the Flight RPC between nodes. See Authentication and Securing the Cluster.
  • Access Control: Per-user read rules on catalogs, schemas and tables, in Trino's file-based access control format. See Access Control.
  • Admission Control: A cap on running queries with a queue behind it, a per-query memory limit on every node, statement_timeout, and cancellation that reaches the workers. See Admission Control.
  • Worker Failure Handling: A worker that crashes or hangs fails its queries with an error naming it, instead of hanging them; the rest of the cluster keeps serving. With retry_policy = "query" (off by default), a read-only query re-runs on the remaining workers. See When a Worker Fails.
  • Rolling Upgrades and Kubernetes: Workers drain on SIGTERM and the coordinator refuses workers on another release. Multi-arch images and a Helm chart are published with every release. See Stopping a Worker and Kubernetes.
  • Observable: Prometheus metrics on the Web UI port, a slow-query log, an example Grafana dashboard, and the query history as system.runtime.queries. See Observability.

Supported Data Sources ​

SourceFormatDescription
Local filesCSV, ParquetRead from local filesystem paths
S3CSV, ParquetAmazon S3 and S3-compatible stores (MinIO, LocalStack)
GCSCSV, ParquetGoogle Cloud Storage
AzureCSV, ParquetAzure Blob Storage
Hive MetastoreParquet, ORCHMS 4.x catalog with automatic table discovery; partitioned tables read the partitions registered in HMS. See Hive
Iceberg (via HMS)ParquetRead-only, current snapshot, through the Hive catalog. See Iceberg
PostgreSQLTables and viewsRead-only catalog with filter, projection and limit pushdown and parallel key-range reads. See PostgreSQL

Current Status ​

The latest release is v0.2.1 (release notes, changelog). Releases are source releases; see Quickstart.

All 22 TPC-H queries return results cell-identical to Trino, single-node and distributed. Against Trino on the same hardware, Arneb is faster on all 22 at SF10 (2.3× warm, geomean) at about a seventh of the peak memory, and faster on all 22 at SF30 cold (3.5× at 0.22× the memory); see Performance. With dimensions in PostgreSQL and facts in Hive, all 22 are still cell-identical to Trino on the same layout. SQL support includes SELECT, DDL/DML, CTEs, window functions, subqueries, set operations and 90 Trino-compatible scalar functions.

Next Steps ​