Databricks Driver v0.1.2¶
Driver Version v0.1.2 Tested With Databricks 17
This driver provides access to Databricks , a cloud-based platform for data analytics.
Note
This is an early version of the driver, and contributors are still working with Databricks to expand the featureset and improve performance.
Installation¶
The Databricks driver can be installed with dbc :
dbc install databricks
Connecting¶
To connect, edit the uri option below to match your environment and run the following:
from adbc_driver_manager import dbapi
conn = dbapi.connect(
driver="databricks",
db_kwargs = {
"uri": "databricks://token:[email protected]:443/sql/protocolv1/o/1234567890123456/1234-567890-abcdefgh"
}
)
Note: The example above is for Python using the adbc-driver-manager package but the process will be similar for other driver managers.
Connection String Format¶
Databricks’s URI syntax supports three primary forms:
Databricks personal access token authentication:
databricks://token:<personal-access-token>@<server-hostname>:<port-number>/<http-path>?<param1=value1>&<param2=value2>
Components:
scheme:databricks://(required)<personal-access-token>: (required) Databricks personal access token.<server-hostname>: (required) Server Hostname value.port-number: (required) Port value, which is typically 443.http-path: (required) HTTP Path value.Query params: Databricks connection attributes. For complete list of optional parameters, see Databricks Optional Parameters
OAuth user-to-machine (U2M) authentication:
databricks://<server-hostname>:<port-number>/<http-path>?authType=OauthU2M&<param1=value1>&<param2=value2>
Components:
scheme:databricks://(required)<server-hostname>: (required) Server Hostname value.port-number: (required) Port value, which is typically 443.http-path: (required) HTTP Path value.authType=OauthU2M: (required) Specifies OAuth user-to-machine authentication.Query params: Additional Databricks connection attributes. For complete list of optional parameters, see Databricks Optional Parameters
OAuth machine-to-machine (M2M) authentication:
databricks://<server-hostname>:<port-number>/<http-path>?authType=OAuthM2M&clientID=<client-id>&clientSecret=<client-secret>&<param1=value1>&<param2=value2>
Components:
scheme:databricks://(required)<server-hostname>: (required) Server Hostname value.port-number: (required) Port value, which is typically 443.http-path: (required) HTTP Path value.authType=OAuthM2M: (required) Specifies OAuth machine-to-machine authentication.<client-id>: (required) Service principal’s UUID or Application ID value.<client-secret>: (required) Secret value for the service principal’s OAuth secret.Query params: Additional Databricks connection attributes. For complete list of optional parameters, see Databricks Optional Parameters
This follows the Databricks SQL Driver for Go format with the addition of the databricks:// scheme.
Note
Reserved characters in URI elements must be URI-encoded. For example, @ becomes %40.
Examples:
databricks://token:dapi1234abcd5678efgh@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/protocolv1/o/1234567890123456/1234-567890-abcdefghdatabricks://myworkspace.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456?authType=OauthU2Mdatabricks://myworkspace.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456?authType=OAuthM2M&clientID=12345678-1234-1234-1234-123456789012&clientSecret=mysecret123
Feature & Type Support¶
| Feature | Support | |
|---|---|---|
| Bulk Ingestion | Create | ✅ |
| Append | ✅ | |
| Create/Append | ✅ | |
| Replace | ✅ | |
| Temporary Table | ❌ | |
| Specify target catalog | ❌ | |
| Specify target schema | ❌ | |
| Non-nullable fields are marked NOT NULL | ❌ | |
| Catalog (GetObjects) | depth=catalogs | ✅ |
| depth=db_schemas | ✅ | |
| depth=tables | ✅ | |
| depth=columns (all) | ✅ | |
| Get Parameter Schema | ❌ | |
| Get Table Schema | ❌ | |
| Prepared Statements | ✅ | |
| Transactions | ❌ | |
Types¶
Databricks to Arrow¶
Databricks Type |
Arrow Type |
|---|---|
BIGINT |
int64 |
BOOLEAN |
bool |
DATE |
date32[day] |
DOUBLE PRECISION |
double |
INT |
int32 |
NUMERIC |
❌ |
REAL |
float |
SMALLINT |
int16 |
TIMESTAMP |
timestamp[us] (with time zone) |
TIMESTAMP_NTZ |
timestamp[us] |
VARBINARY |
binary |
VARCHAR |
string |
Arrow to Databricks¶
| Arrrow Type | Databricks Type | |
|---|---|---|
| Bind | Ingest | |
| binary |
❌ |
VARBINARY |
| binary_view |
❌ |
VARBINARY |
| bool |
❌ |
BOOLEAN |
| date32[day] |
❌ |
DATE |
| decimal128 |
❌ |
|
| double |
❌ |
DOUBLE PRECISION |
| fixed_size_binary |
❌ |
VARBINARY |
| float |
❌ |
REAL |
| halffloat |
❌ |
(not tested) |
| int16 |
❌ |
SMALLINT |
| int32 |
❌ |
INT |
| int64 |
❌ |
BIGINT |
| large_binary |
❌ |
VARBINARY |
| large_string |
❌ |
VARCHAR |
| string |
❌ |
VARCHAR |
| string_view |
❌ |
VARCHAR |
| time32[ms] |
❌ |
(not tested) |
| time32[s] |
❌ |
(not tested) |
| time64[ns] |
❌ |
(not tested) |
| time64[us] |
❌ |
(not tested) |
| timestamp[ms] |
❌ |
TIMESTAMP_NTZ |
| timestamp[ms] (with time zone) |
❌ |
TIMESTAMP |
| timestamp[ns] |
❌ |
TIMESTAMP_NTZ |
| timestamp[ns] (with time zone) |
❌ |
TIMESTAMP |
| timestamp[s] |
❌ |
TIMESTAMP_NTZ |
| timestamp[s] (with time zone) |
❌ |
TIMESTAMP |
| timestamp[us] |
❌ |
TIMESTAMP(6) |
| timestamp[us] (with time zone) |
❌ |
TIMESTAMP |
Compatibility¶
This driver was tested on the following versions of Databricks:
2025.35