Apache DataFusion Driver v0.26.0¶

Driver Version v0.26.0 Release Date 2026-07-07 Tested With Apache DataFusion 53

This driver provides access to Apache DataFusion .

Note

This project is not part of the Apache Software Foundation.

Installation & Quickstart¶

The driver can be installed with dbc :

dbc install datafusion

Connecting¶

DataFusion is an in-process query engine and does not need a connection string or URI.

from adbc_driver_manager import dbapi

dbapi.connect(driver="datafusion")

You can provide the URI datafusion://, but no other URI is currently accepted:

from adbc_driver_manager import dbapi

dbapi.connect("datafusion://")

Note: The example above is for Python using the adbc-driver-manager package but the process will be similar for other driver managers. See adbc-quickstarts .

Feature & Type Support¶

The DataFusion driver supports many of the extensions to the SQL dialect that the DataFusion CLI implements, including SHOW ALL, SHOW, SET <OPTION> TO <VALUE>, CREATE EXTERNAL TABLE, and scanning local and remote files/directories, including over HTTP and on S3.

For example:

SELECT `Breed Name`, `Lifespan`
  FROM 'https://hyperparam-public.s3.amazonaws.com/bunnies.parquet'
  ORDER BY `Lifespan` DESC
  LIMIT 5;

-- Result:
-- ┌──────────────────┬──────────┐
-- │ Breed Name       │ Lifespan │
-- ├──────────────────┼──────────┤
-- │ French Angora    │ 12       │
-- │ English Angora   │ 10       │
-- │ Netherland Dwarf │ 10       │
-- │ Mini Lop         │ 9        │
-- │ Lionhead         │ 9        │
-- └──────────────────┴──────────┘

Feature		Apache DataFusion
Bulk Ingestion	Create	✅
	Append	✅
	Create/Append	✅
	Replace	✅
	Temporary Table	❌
	Target Catalog	✅
	Target Schema	✅
	Non-nullable fields are marked NOT NULL	❌
Catalog (GetObjects)	depth=catalogs	✅
	depth=db_schemas	✅
	depth=tables	✅
	depth=columns (all)	✅
Get Parameter Schema		✅
Get Table Schema		✅
Prepared Statements		✅
Transactions		❌

Types¶

Database to Arrow¶

Database Type	Apache DataFusion
BIGINT	int64
BOOLEAN	bool
BYTEA	binary
DATE	date32[day]
DECIMAL	decimal128
DOUBLE	double
FLOAT	float
INT	int32
SMALLINT	int16
TIME	time64[ns]
TIMESTAMP	timestamp[ns] [1]
TIMESTAMP WITH TIME ZONE	timestamp[ns] [1] [2]
VARCHAR	string, string_view ⚠️ [3]

Arrow to Database¶

Arrow Type	Apache DataFusion Type
	Bind	Ingest
	binary	BYTEA
binary_view	❌	BYTEA
bool	BOOLEAN
date32[day]	DATE
decimal128	NUMERIC	DECIMAL, NUMERIC
double	DOUBLE PRECISION	DOUBLE
fixed_size_binary	❌	BYTEA
float	REAL	FLOAT
int16	SMALLINT
int32	INT
int64	BIGINT
large_binary	❌	BYTEA
large_string	❌	VARCHAR
string	❌	VARCHAR
string_view	❌	VARCHAR
time32[ms]	(NA/not tested)	TIME
time32[s]	(NA/not tested)	TIME
time64[ns]	TIME
time64[us]	(NA/not tested)	TIME
timestamp[ms]	TIMESTAMP(3)	TIMESTAMP
timestamp[ms] (with time zone)	❌	TIMESTAMP WITH TIME ZONE
timestamp[ns]	TIMESTAMP(9)	TIMESTAMP
timestamp[ns] (with time zone)	❌	TIMESTAMP WITH TIME ZONE
timestamp[s]	TIMESTAMP(0)	TIMESTAMP
timestamp[s] (with time zone)	❌	TIMESTAMP WITH TIME ZONE
timestamp[us]	TIMESTAMP(6)	TIMESTAMP
timestamp[us] (with time zone)	❌	TIMESTAMP WITH TIME ZONE

Partitioned Execution¶

This driver supports ADBC’s partitioned execution; one partition is generated per output partition. Each partition contains the physical plan, and can be distributed across CPUs or physical machines. If the plan contains a shuffle, however, distributed execution will likely be slower, as each partition must read the entire output.

The statement option datafusion.partition_mode controls what to do if a shuffle is detected:

Value	Behavior
`auto` (default)	Collapse to a single partition when the plan shuffles, otherwise natural partitions
`multi`	One descriptor per natural output partition; logs a warning if the plan shuffles
`single`	Always collapse to a single partition (effectively the same as regular execution)

Extending DataFusion¶

It is possible to customize the embedded DataFusion by depending on this crate, then using hooks to customize the DataFusion SessionContext and register PhysicalExtensionCodecs. The customized driver can then be built into a shared library and distributed as an ADBC driver. For more, see:

DataFusionDriver::new_with_context_init, which accepts a callback to modify the SessionContext
DataFusionDriver::with_codec, which accepts a PhysicalExtensionCodec used to serialize/deserialize plans in partitioned execution
The adbc_ffi crate, which provides helpers to export a Rust ADBC driver as a shared library

Compatibility¶

This driver was tested on:

Apache DataFusion 53.1.0

Previous Versions¶

To see documentation for previous versions of this driver, see the following: