Databricks

Adding and configuring a Databricks connection within Qualytics empowers the platform to build a symbolic link with your database to perform operations like data discovery, visualization, reporting, syncing, profiling, scanning, anomaly surveillance, and more.

This documentation provides a step-by-step guide on how to add Databricks as a source datastore in Qualytics. It covers the entire process, from initial connection setup to testing and finalizing the configuration. A Databricks datastore can also be the place Qualytics writes to; see Use as an Enrichment Datastore.

By following these instructions, enterprises can ensure their Databricks environment is properly connected with Qualytics, unlocking the platform's potential to help you proactively manage your full data quality lifecycle.

Let's get started 🚀

Databricks Setup Guide

This guide provides a comprehensive walkthrough for setting up Databricks. It highlights the distinction between SQL Warehouses and All-Purpose Compute, the functionality of node pools, and the enhancements they offer.

Additionally, it details the process for attaching compute resources to node pools and explains the minimum requirements for effective operation.

Understanding SQL Warehouses and All-Purpose Compute

SQL Warehouses (Serverless)

SQL Warehouses (Serverless) in Databricks utilize serverless SQL endpoints for running SQL queries.

All-Purpose Compute

All-purpose compute in Databricks refers to clusters that are not optimized for specific tasks. While they offer flexibility, they may not provide the best performance or cost-effectiveness for certain workloads.

Node Pool and Its Usage

A node pool in Databricks is a set of homogeneous virtual machines (VMs) within a cluster. It allows you to have a fixed set of instances dedicated to specific tasks, ensuring consistent performance and resource isolation.

Improving All-Purpose Compute with Node Pools

To improve the performance of all-purpose compute using node pools, you can follow these steps:

Step 1: Configure details for Qualytics Node Pool.

Step 2: Attach Compute details with the Node Pool.

Databricks Privileges and Permissions

Qualytics connects to Databricks via JDBC using either a Personal Access Token (PAT) or OAuth M2M authentication. It uses the Hive metastore or Unity Catalog metadata APIs to discover catalogs, schemas, tables, columns, and partition information.

Minimum Databricks Permissions (Source Datastore)

Additional Permissions for an Enrichment Datastore

When Databricks is used as an enrichment datastore, the following additional permissions are required so Qualytics can write its outputs, which are named from the linked source's prefix (for example _tpch_failed_checks):

Example: Source Datastore Permissions (Read-Only)

Replace <catalog_name>, <schema_name>, and <user_or_group> with your actual values.

-- Grant catalog and schema access
GRANT USAGE ON CATALOG <catalog_name> TO <user_or_group>;
GRANT USAGE ON SCHEMA <catalog_name>.<schema_name> TO <user_or_group>;

-- Grant read access to all tables in the schema
GRANT SELECT ON SCHEMA <catalog_name>.<schema_name> TO <user_or_group>;

Example: Enrichment Datastore Permissions (Read-Write)

-- Grant catalog and schema access
GRANT USAGE ON CATALOG <catalog_name> TO <user_or_group>;
GRANT USAGE ON SCHEMA <catalog_name>.<schema_name> TO <user_or_group>;

-- Grant read-write access and table creation
GRANT SELECT, MODIFY ON SCHEMA <catalog_name>.<schema_name> TO <user_or_group>;
GRANT CREATE TABLE ON SCHEMA <catalog_name>.<schema_name> TO <user_or_group>;

Troubleshooting Common Errors

Detailed Troubleshooting Notes

Authentication Errors

The error INVALID_CREDENTIALS indicates that the Personal Access Token (PAT) or OAuth credentials are invalid.

Common causes:

• Expired PAT: Personal Access Tokens have a configurable expiration date. The token may have expired since the connection was created.
• Revoked PAT: the token was manually revoked from the Databricks workspace.
• Wrong workspace: the PAT was generated in a different Databricks workspace than the one being connected to.
• OAuth secret expired: when using OAuth M2M, the Service Principal's secret has expired.

Permission Errors

The error User does not have USAGE permission or User does not have SELECT permission means the user authenticated successfully but lacks Unity Catalog grants.

Common causes:

• Missing USAGE on catalog: the user cannot access the catalog. This is the most common issue: USAGE must be granted on both the catalog and the schema.
• Missing SELECT on schema: the user has catalog access but cannot read tables in the specific schema.
• Unity Catalog not enabled: the workspace uses the legacy Hive metastore instead of Unity Catalog, and permissions are managed differently.

Compute Errors

The error Cluster is not running or HTTP Path is invalid indicates a problem with the compute resource, not with data permissions.

Common causes:

• Cluster stopped: All-Purpose Clusters do not auto-resume by default. The cluster must be started manually or switched to a SQL Warehouse with auto-resume.
• Wrong HTTP Path: the HTTP Path was copied incorrectly or the warehouse/cluster has been recreated with a new path.
• No compute access: the user lacks CAN USE permission on the SQL Warehouse or cluster.

Retrieve the Connection Details

This section explains how to retrieve the connection details that you need to connect to Databricks.

Credentials to Connect with Qualytics

To configure Databricks, you need the following credentials:

Get Connection Details for the SQL Warehouse

Follow the given steps to get the connection details for the SQL warehouse:

1. Click on the SQL Warehouses in the sidebar.
2. Choose a warehouse to connect to.
3. Navigate to the Connection Details tab.
4. Copy the connection details.

Get Connection Details for the Cluster

Follow the given steps to get the connection details for the cluster:

1. Click on the Compute in the sidebar.
2. Choose a cluster to connect to.
3. Navigate to the Advanced Options.
4. Click on the JDBC/ODBC tab.
5. Copy the connection details.

Get the Access Token

Step 1: In your Databricks workspace, click your Databricks username in the top bar, and then select User Settings from the dropdown menu.

Step 2: In the Settings page, select the Developer option in the User section.

Step 3: In the Developer page, click on Manage in Access Tokens.

Step 4: In the Access Tokens page, click on the Generate new token button.

Step 5: You will see a modal to add a description and validation time (in days) for the token.

Step 6: After adding the contents, click on Generate, and it will show the token.

Step 7: You can see the new token on the Access Tokens page.

You can also revoke a token on the Access Tokens page by clicking on the Revoke token button.

Add a Source Datastore

A source datastore is a storage location Qualytics connects to so it can profile, scan, and monitor data. Adding Databricks as a source lets Qualytics query the tables in your workspace through the Databricks JDBC driver and run quality operations on the tables it discovers.

Before you start, gather the connection details and the credential described in Retrieve the Connection Details above.

Field reference

The Add Datastore page shows the sections below when Databricks is selected. When reusing an existing connection, the Connection Properties, Authentication, and Secrets Management sections come already filled in and read-only: Qualytics has already validated those credentials, so you fill in only the Location and the General fields. To change a saved connection's credentials, edit the connection through the Manage Connections page; edits there apply to every datastore that reuses the connection.

Connection Properties

These fields define the Databricks endpoint Qualytics connects to. They belong to the connection: when reusing an existing connection, they come already filled in and read-only.

Authentication

Choose how Qualytics authenticates to the workspace. Setting Type changes the credential fields shown below it, so pick the tab that matches your choice. These fields also belong to the connection: already filled in and read-only when reusing one.

Secrets Management

This group is optional: use it only if you want Qualytics to pull credentials from a secrets manager instead of typing them into the form. Turn on HashiCorp Vault to show the fields below. Despite the label, any secrets manager that exposes a compatible REST API works, not only HashiCorp Vault; see Secrets Management. It also belongs to the connection: read-only when reusing an existing connection.

Location

Pick the catalog and database(s) Qualytics should read from. You fill these in on both flows.

General

Common fields for every source datastore, shown below the Location section. You fill these in on both flows.

Below the form, an information banner lists the Public addresses and Private addresses your datastore connections originate from. Allow the ones that match your network setup through your security groups or firewall rules; see How Connections Work for where the addresses come from.

Steps

There are two ways to set up the connection: reuse a connection you already saved (Existing Connection) or create a new one from scratch (New Connection). The tabs below walk through each option; pick the one you want to follow. Each field is described in the Field reference above.

Use as an Enrichment Datastore

An enrichment datastore is where Qualytics writes its findings: scan results, source record examples, remediation snapshots, materialized copies, and exports. Keep it on a dedicated schema or storage path, separate from the source data you govern. When a source datastore is linked to it, that source's writes land there. The destination can share the source's connection when those credentials have the required write access.

To use a Databricks datastore as a destination, its connection user needs the write grants listed under Additional Permissions for an Enrichment Datastore. Then link it from the datastore that will write to it: open its Settings menu, click Enrichment , and either pick the Databricks datastore under Use existing or create it there under Add new. The same choice is offered in the second step of the Add Datastore page. The flow is the same for every connector and is documented once, in Link an Enrichment Destination and Link Enrichment on Datastore Creation.

The rest of this section covers only what is specific to Databricks: the fields you fill in when creating the destination on Databricks from inside that dialog. For what Qualytics writes and how the outputs are named, see Enrichment Outputs.

Field reference

The Enrichment Destination step shows the sections below when Databricks is selected. When reusing an existing connection, the Connection Properties, Authentication, and Secrets Management sections come already filled in and read-only.

Connection Properties

These fields define the Databricks endpoint Qualytics writes to. They are the same fields as on the source datastore flow, repeated here so this section stands on its own.

Authentication

Pick the tab that matches the credential you use. The account behind it needs write access, not just read.

Secrets Management

This group is optional: use it only if you want Qualytics to pull credentials from a secrets manager instead of typing them into the form. Turn on HashiCorp Vault to show the fields below. Despite the label, any secrets manager that exposes a compatible REST API works, not only HashiCorp Vault; see Secrets Management.

Location

Where Qualytics writes the enrichment tables inside the workspace.

General

Write Behavior

Whether a Scan also writes a snapshot of the anomalous records, and whether Qualytics syncs and profiles the destination after each write.

Steps

Fill in these fields under Add new, either in the Enrichment Destination dialog of the datastore that will write here or in the second step of the Add Datastore page, then click Test connection and save. The steps are the same for every connector: see Link an Enrichment Destination or Link Enrichment on Datastore Creation.

API Payload Examples

This section provides detailed examples of API payloads to guide you through the process of creating and managing datastores using Qualytics API. Each example includes endpoint details, sample payloads, and instructions on how to replace placeholder values with actual data relevant to your setup.

Creating a Source Datastore

This section provides sample payloads for creating a Databricks datastore. Replace the placeholder values with actual data relevant to your setup.

Endpoint: /api/datastores (post)

{
    "name": "your_datastore_name",
    "teams": ["Public"],
    "database": "databricks_catalog",
    "schema": "databricks_database",
    "enrichment_only": false,
    "trigger_sync": true,
    "connection": {
        "name": "your_connection_name",
        "type": "databricks",
        "host": "databricks_host",
        "password": "databricks_token",
        "parameters": {
            "path": "databricks_http_path"
        }
    }
}

{
    "name": "your_datastore_name",
    "teams": ["Public"],
    "database": "databricks_catalog",
    "schema": "databricks_database",
    "enrichment_only": false,
    "trigger_sync": true,
    "connection_id": 123
}

{
    "name": "your_datastore_name",
    "teams": ["Public"],
    "database": "databricks_catalog",
    "schema": "databricks_database",
    "enrichment_only": false,
    "trigger_sync": true,
    "connection": {
        "name": "your_connection_name",
        "type": "databricks",
        "host": "databricks_host",
        "username": "service_principal_application_id",
        "password": "service_principal_oauth_secret",
        "parameters": {
            "path": "databricks_http_path",
            "authentication_type": "OAUTH_M2M"
        }
    }
}

# Step 1: Create a Connection
qualytics connections create \
    --type databricks \
    --name "your_connection_name" \
    --host ${DATABRICKS_HOST} \
    --password ${DATABRICKS_TOKEN} \
    --parameters '{"path": "databricks_http_path"}'

# Step 2: Create a Source Datastore
qualytics datastores create \
    --name "your_datastore_name" \
    --connection-name "your_connection_name" \
    --database hive_metastore \
    --schema default

Create an Enrichment Datastore

This section provides sample payloads for creating an enrichment datastore. Replace the placeholder values with actual data relevant to your setup.

Endpoint: /api/datastores (post)

{
    "name": "your_datastore_name",
    "teams": ["Public"],
    "database": "databricks_enrichment_catalog",
    "schema": "databricks_enrichment_database",
    "enrichment_only": true,
    "connection": {
        "name": "your_connection_name",
        "type": "databricks",
        "host": "databricks_host",
        "password": "databricks_token",
        "parameters": {
            "path": "databricks_http_path"
        }
    }
}

{
    "name": "your_datastore_name",
    "teams": ["Public"],
    "database": "databricks_enrichment_catalog",
    "schema": "databricks_enrichment_database",
    "enrichment_only": true,
    "connection_id": 123
}

# Step 1: Create a Connection
qualytics connections create \
    --type databricks \
    --name "your_connection_name" \
    --host ${DATABRICKS_HOST} \
    --password ${DATABRICKS_TOKEN} \
    --parameters '{"path": "databricks_http_path"}'

# Step 2: Create an Enrichment Datastore
qualytics datastores create \
    --name "your_datastore_name" \
    --connection-name "your_connection_name" \
    --database hive_metastore \
    --schema your_enrichment_schema \
    --enrichment-only

Link an Enrichment Destination

Use the provided endpoint to link a datastore to its enrichment destination:

Endpoint Details: /api/datastores/{datastore-id}/enrichment/{enrichment-id} (patch)