BigQuery¶
Driver Version v1.10.0 Tested With BigQuery 1.72.0
This driver provides access to BigQuery , a data warehouse offered by Google Cloud.
Note
This project is not affiliated with Google.
Installation¶
The BigQuery driver can be installed with dbc :
dbc install bigquery
Pre-requisites¶
Using the BigQuery driver requires some setup before you can connect:
Create a Google Cloud account
Install the Google Cloud CLI (for managing credentials)
Authenticate with Google Cloud
Run
gcloud auth application-default login
Create, find, or reuse a project and dataset (record these for later)
Connecting¶
To connect, replace my-gcp-project and my-gcp-dataset below with the appropriate values for your situation and run the following:
from adbc_driver_manager import dbapi
conn = dbapi.connect(
driver="bigquery",
db_kwargs={
"adbc.bigquery.sql.project_id": "my-gcp-project",
"adbc.bigquery.sql.dataset_id": "my-gcp-dataset"
}
)
Note: The example above is for Python using the adbc-driver-manager package but the process will be similar for other driver managers.
The driver supports connecting with individual options or connection strings.
Connection String Format¶
BigQuery URI syntax:
bigquery://[Host]:[Port]/ProjectID?OAuthType=[AuthValue]&[Key]=[Value]&[Key]=[Value]...
The format follows a similar approach to the Simba BigQuery JDBC Connection String Format .
Components:
Scheme:bigquery://(required)Host: BigQuery API endpoint (optional, defaults tobigquery.googleapis.com)Port: TCP port (optional, defaults to 443)ProjectID: Google Cloud Project ID (required)OAuthType: Authentication type number (optional, defaults to0)0- Application Default Credentials (default)1- Service Account JSON File2- Service Account JSON String3- User Authentication (OAuth)
Additional Parameters: Connection configuration as key-value pairs. For a complete list of available parameters, see the Simba BigQuery JDBC Connector Configuration Options .
Note
Reserved characters in URI elements must be URI-encoded. For example, @ becomes %40.
Examples:
bigquery:///my-project-123(uses Application Default Credentials)bigquery://bigquery.googleapis.com/my-project-123?OAuthType=1&AuthCredentials=/path/to/key.jsonbigquery:///my-project-123?OAuthType=3&AuthClientId=123.apps.googleusercontent.com&AuthClientSecret=secret&AuthRefreshToken=tokenbigquery://bigquery.googleapis.com/my-project-123?OAuthType=0&DatasetId=analytics&Location=USbigquery:///my-project-123?OAuthType=2&AuthCredentials=%7B%22type%22%3A%22service_account%22...%7D&DatasetId=data_warehouse
Feature & Type Support¶
| Feature | Support | |
|---|---|---|
| Bulk Ingestion | Create | ✅ |
| Append | ✅ | |
| Create/Append | ✅ | |
| Replace | ✅ | |
| Temporary Table | ❌ | |
| Specify target catalog | ❌ | |
| Specify target schema | ✅ | |
| Non-nullable fields are marked NOT NULL | ✅ | |
| Catalog (GetObjects) | depth=catalogs | ✅ |
| depth=db_schemas | ✅ | |
| depth=tables | ✅ | |
| depth=columns (all) | ✅ | |
| Get Parameter Schema | ❌ | |
| Get Table Schema | ✅ | |
| Prepared Statements | ✅ | |
| Transactions | ✅ | |
Types¶
BigQuery to Arrow¶
BigQuery Type |
Arrow Type |
|---|---|
ARRAY |
list [1] |
BIGNUMERIC |
decimal256(76, 38) |
BOOL |
bool |
BOOLEAN |
bool |
BYTES |
binary |
DATE |
date32[day] |
DATETIME |
timestamp[us] [2] |
FLOAT64 |
double |
GEOGRAPHY |
extension<geoarrow.wkt> |
INT64 |
int64 |
INTERVAL |
month_day_nano_interval |
JSON |
extension<arrow.json> |
NUMERIC |
|
RANGE<DATE> |
struct<start: date32[day], end: date32[day]> |
RANGE<DATETIME> |
struct<start: timestamp[us], end: timestamp[us]> |
RANGE<TIMESTAMP> |
struct<start: timestamp[us, tz=UTC], end: timestamp[us, tz=UTC]> |
STRING |
string |
STRUCT |
struct |
TIME |
time64[us] |
TIMESTAMP |
timestamp[us] (with time zone) [6] |
Arrow to BigQuery¶
| Arrrow Type | BigQuery Type | |
|---|---|---|
| Bind | Ingest | |
| binary |
BYTES |
|
| binary_view |
BYTES |
❌ [9] |
| bool |
BOOLEAN |
|
| date32[day] |
DATE |
|
| decimal128 |
NUMERIC |
|
| double |
FLOAT64 |
|
| fixed_size_binary |
BYTES |
|
| float |
(not tested) |
FLOAT64 |
| int16 |
(not tested) |
INT64 |
| int32 |
(not tested) |
INT64 |
| int64 |
INT64 |
|
| large_binary |
BYTES |
|
| large_string |
STRING |
|
| list |
(not tested) |
|
| string |
STRING |
|
| string_view |
STRING |
❌ [13] |
| struct |
(not tested) |
STRUCT |
| time32[ms] |
TIME |
|
| time32[s] |
TIME |
|
| time64[ns] |
❌ [7] |
❌ [14] |
| time64[us] |
TIME |
|
| timestamp[ms] |
DATETIME |
❌ [15] |
| timestamp[ms] (with time zone) |
TIMESTAMP |
❌ [15] |
| timestamp[ns] |
DATETIME [8] |
❌ [15] |
| timestamp[ns] (with time zone) |
TIMESTAMP |
❌ [15] |
| timestamp[s] |
DATETIME |
❌ [15] |
| timestamp[s] (with time zone) |
TIMESTAMP |
❌ [15] |
| timestamp[us] |
DATETIME |
❌ [15] |
| timestamp[us] (with time zone) |
TIMESTAMP |
|
Previous Versions¶
To see documentation for previous versions of this driver, see the following: