Apache Spark Driver v0.1.0¶

Connection String Format¶

• The URI scheme is “spark://”.

• The host and port should be provided.

• If not specified, the api defaults to thrift+binary (URI-encoded: thrift%2Bbinary).

• Options can be specified as query parameters or as driver options.

Connection Options¶

These parameters can be specified in the URI as query parameters, or as connection parameters:

spark.api (query parameter: api)

Values: connect, livy, thrift+binary, or thrift+http.

The protocol used to connect to Spark.

Value	Backend
connect	Spark Connect
livy	Apache Livy
thrift+binary	HiveServer2 Thrift (over TCP)
thrift+http	HiveServer2 Thrift (over HTTP)

spark.auth_type (query parameter: auth_type)

Values: sql, spark, or pyspark.

How to authenticate to Spark.

Auth Type	Applicable Backends	Description
aws_sigv4	livy	Use AWS SDK
basic	livy	Username/password
ldap	thrift+binary, thrift+http	Not yet implemented
kerberos	thrift+binary, thrift+http	Not yet implemented
none	connect, livy	No authentication
nosasl	thrift+binary, thrift+http	No authentication
plain	thrift+binary, thrift+http	Username/password
token	connect	Username/password (token)

spark.livy.session_kind (query parameter: livy.session_kind)

Values: sql, spark, or pyspark.

For the Livy backend, what kind of session to create.

Warning

Currently only sql is tested/supported.

spark.connect.session_id (query parameter: connect.session_id)

Type: string.

For the Spark Connect backend, reuse this client session.

spark.connect.release_session (query parameter: connect.release_session)

Type: boolean. Default: true.

For the Spark Connect backend, whether to call ReleaseSession when the connection is closed. Set to false to keep a session available after closing the connection.

spark.livy.session_id (query parameter: livy.session_id)

Type: string.

For the Livy backend, reuse this client session.

spark.livy.release_session (query parameter: livy.release_session)

Type: boolean. Default: true.

For the Livy backend, whether to delete the session when the connection is closed. Set to false to keep a session available after closing the connection.

spark.tls (query parameter: tls)

Type boolean. Default: false.

Whether to use TLS for connecting. Only applies to connect, livy, and thrift+http.

spark.validate_server_certificate (query parameter: validateservercertificate)

Type boolean. Default: true.

Whether to validate the server’s TLS certificate. Should only be disabled for development/testing.

adbc.connection.catalog (query parameter: catalog)

Type: string.

This is a standard ADBC option; it can additionally be specified during connection to select which catalog to use instead of the server default.

HiveServer2/Thrift Protocol¶

• In Spark 3.x, binary data that does not happen to be valid UTF-8 will be corrupted.

• The client cannot tell whether a timestamp carries a time zone or not; all timestamps are assumed to be in UTC as a result.

Apache Livy¶

• Only the first 1000 rows of a result set can be fetched. This can be tuned by configuring Spark with spark.sql.repl.eagerEval.maxNumRows.

• In general, we have found that performance is worse than with Spark Connect or HiveServer2.

• Connecting to an Amazon EMR (Serverless) cluster via Livy requires setting the emr-serverless.session.executionRoleArn session config option to an appropriate role ARN. This can be set via the ADBC option spark.opt.emr-serverless.session.executionRoleArn.

• By default, the driver will attempt to start a new Livy session, which tends to take some time (~a few minutes), especially when using Amazon EMR. To amortize this time across multiple connections, the option spark.livy.session_id can be used to fetch the session ID, and to provide it upon connection, bypassing creating a new session.

• By default, the driver will close the session when the connection is closed. Setting spark.livy.release_session to false on connection will avoid this, making it easier to reuse the session.

Spark Connect¶

• To connect to Amazon EMR, the connection URI should look like this:

  spark://:<AUTH TOKEN>@<SESSION ID>.s.emr-serverless-services.<REGION>.amazonaws.com:443?tls=true&auth_type=token&api=connect
  

  The full hostname can be obtained from the AWS API, e.g. via the CLI:

  aws emr-serverless get-session-endpoint --application-id <APPLICATION ID> --session-id <SESSION ID>
  

  This command will also give you the auth token.

Amazon EMR (Serverless)¶

• Amazon EMR is not currently enabled in our automated integration testing.

• To use bulk ingest, set spark.ingest.location to a path on S3 where the table data will be stored.

• In our testing, once 25 sessions have been created via Spark Connect (note that this is distinct from an EMR “session”), the server will reject further connections with Maximum allowed sessions (25) exceeded, even if prior sessions have been released. (In other words, the maximum session limit is not a maximum concurrent session limit.) A new EMR session must be created to continue.

Also see the above caveats for specific ways to connect to EMR.

Azure Synapse Analytics¶

• Only Livy is supported.

• The hostname/port to use in the connection URI is <WORKSPACE NAME>.dev.azuresynapse.net:443. This can be found as the “Development endpoint” in the Azure portal.

• The connection URI must contain a path component like this:

  /livyApi/versions/2024-09-20/sparkpools/<NAME OF SPARK POOL>
  

• Your user or service principal must have the “Synapse Compute Operator” role. Synapse roles can only be assigned from Synapse Studio, not Azure Portal. See the Azure documentation for more details.

Types¶