Databricks

Adding and configuring a Databricks connection within Qualytics empowers the platform to build a symbolic link with your database to perform operations like data discovery, visualization, reporting, syncing, profiling, scanning, anomaly surveillance, and more.

This documentation provides a step-by-step guide on how to add Databricks as both a source and enrichment datastore in Qualytics. It covers the entire process, from initial connection setup to testing and finalizing the configuration.

By following these instructions, enterprises can ensure their Databricks environment is properly connected with Qualytics, unlocking the platform's potential to help you proactively manage your full data quality lifecycle.

Let's get started 🚀

Databricks Setup Guide

This guide provides a comprehensive walkthrough for setting up Databricks. It highlights the distinction between SQL Warehouses and All-Purpose Compute, the functionality of node pools, and the enhancements they offer.

Additionally, it details the process for attaching compute resources to node pools and explains the minimum requirements for effective operation.

Understanding SQL Warehouses and All-Purpose Compute

SQL Warehouses (Serverless)

SQL Warehouses (Serverless) in Databricks utilize serverless SQL endpoints for running SQL queries.

All-Purpose Compute

All-purpose compute in Databricks refers to clusters that are not optimized for specific tasks. While they offer flexibility, they may not provide the best performance or cost-effectiveness for certain workloads.

Node Pool and Its Usage

A node pool in Databricks is a set of homogeneous virtual machines (VMs) within a cluster. It allows you to have a fixed set of instances dedicated to specific tasks, ensuring consistent performance and resource isolation.

Improving All-Purpose Compute with Node Pools

To improve the performance of all-purpose compute using node pools, you can follow these steps:

Step 1: Configure details for Qualytics Node Pool.

Step 2: Attach Compute details with the Node Pool.

Databricks Privileges and Permissions

Qualytics connects to Databricks via JDBC using either a Personal Access Token (PAT) or OAuth M2M authentication. It uses the Hive metastore or Unity Catalog metadata APIs to discover catalogs, schemas, tables, columns, and partition information.

Minimum Databricks Permissions (Source Datastore)

Additional Permissions for Enrichment Datastore

When using Databricks as an enrichment datastore, the following additional permissions are required for Qualytics to write metadata tables (e.g., _qualytics_*):

Example: Source Datastore Permissions (Read-Only)

Replace <catalog_name>, <schema_name>, and <user_or_group> with your actual values.

-- Grant catalog and schema access
GRANT USAGE ON CATALOG <catalog_name> TO <user_or_group>;
GRANT USAGE ON SCHEMA <catalog_name>.<schema_name> TO <user_or_group>;

-- Grant read access to all tables in the schema
GRANT SELECT ON SCHEMA <catalog_name>.<schema_name> TO <user_or_group>;

Example: Enrichment Datastore Permissions (Read-Write)

-- Grant catalog and schema access
GRANT USAGE ON CATALOG <catalog_name> TO <user_or_group>;
GRANT USAGE ON SCHEMA <catalog_name>.<schema_name> TO <user_or_group>;

-- Grant read-write access and table creation
GRANT SELECT, MODIFY ON SCHEMA <catalog_name>.<schema_name> TO <user_or_group>;
GRANT CREATE TABLE ON SCHEMA <catalog_name>.<schema_name> TO <user_or_group>;

Troubleshooting Common Errors

Detailed Troubleshooting Notes

Authentication Errors

The error INVALID_CREDENTIALS indicates that the Personal Access Token (PAT) or OAuth credentials are invalid.

Common causes:

• Expired PAT: Personal Access Tokens have a configurable expiration date. The token may have expired since the connection was created.
• Revoked PAT: the token was manually revoked from the Databricks workspace.
• Wrong workspace: the PAT was generated in a different Databricks workspace than the one being connected to.
• OAuth secret expired: when using OAuth M2M, the Service Principal's secret has expired.

Permission Errors

The error User does not have USAGE permission or User does not have SELECT permission means the user authenticated successfully but lacks Unity Catalog grants.

Common causes:

• Missing USAGE on catalog: the user cannot access the catalog. This is the most common issue: USAGE must be granted on both the catalog and the schema.
• Missing SELECT on schema: the user has catalog access but cannot read tables in the specific schema.
• Unity Catalog not enabled: the workspace uses the legacy Hive metastore instead of Unity Catalog, and permissions are managed differently.

Compute Errors

The error Cluster is not running or HTTP Path is invalid indicates a problem with the compute resource, not with data permissions.

Common causes:

• Cluster stopped: All-Purpose Clusters do not auto-resume by default. The cluster must be started manually or switched to a SQL Warehouse with auto-resume.
• Wrong HTTP Path: the HTTP Path was copied incorrectly or the warehouse/cluster has been recreated with a new path.
• No compute access: the user lacks CAN USE permission on the SQL Warehouse or cluster.

Retrieve the Connection Details

This section explains how to retrieve the connection details that you need to connect to Databricks.

Credentials to Connect with Qualytics

To configure Databricks, you need the following credentials:

Get Connection Details for the SQL Warehouse

Follow the given steps to get the connection details for the SQL warehouse:

1. Click on the SQL Warehouses in the sidebar.
2. Choose a warehouse to connect to.
3. Navigate to the Connection Details tab.
4. Copy the connection details.

Get Connection Details for the Cluster

Follow the given steps to get the connection details for the cluster:

1. Click on the Compute in the sidebar.
2. Choose a cluster to connect to.
3. Navigate to the Advanced Options.
4. Click on the JDBC/ODBC tab.
5. Copy the connection details.

Get the Access Token

Step 1: In your Databricks workspace, click your Databricks username in the top bar, and then select User Settings from the dropdown menu.

Step 2: In the Settings page, select the Developer option in the User section.

Step 3: In the Developer page, click on Manage in Access Tokens.

Step 4: In the Access Tokens page, click on the Generate new token button.

Step 5: You will see a modal to add a description and validation time (in days) for the token.

Step 6: After adding the contents, click on Generate, and it will show the token.

Step 7: You can see the new token on the Access Tokens page.

You can also revoke a token on the Access Tokens page by clicking on the Revoke token button.

Add a Source Datastore

A source datastore is a storage location Qualytics connects to so it can profile, scan, and monitor data. Adding Databricks as a source lets Qualytics query the tables in your workspace through the Databricks JDBC driver and run quality operations on the tables it discovers.

Before you start, gather the connection details and the credential described in Retrieve the Connection Details above.

Field reference

The Add Datastore page shows the sections below when Databricks is selected. When reusing an existing connection, the Connection Properties, Authentication, and Secrets Management sections come already filled in and read-only: Qualytics has already validated those credentials, so you fill in only the Datastores Extraction and the Datastore Properties. To change a saved connection's credentials, edit the connection through the Manage Connections page; edits there apply to every datastore that reuses the connection.

Connection Properties

These fields define the Databricks endpoint Qualytics connects to. They belong to the connection: when reusing an existing connection, they come already filled in and read-only.

Authentication

Choose how Qualytics authenticates to the workspace. Setting Type changes the credential fields shown below it, so pick the tab that matches your choice. These fields also belong to the connection: already filled in and read-only when reusing one.

Secrets Management

This group is optional: use it only if you want Qualytics to pull credentials from a secrets manager instead of typing them into the form. Turn on HashiCorp Vault to show the fields below. Despite the label, any secrets manager that exposes a compatible REST API works, not only HashiCorp Vault; see Secrets Management. It also belongs to the connection: read-only when reusing an existing connection.

Datastores Extraction

Pick the catalog and database(s) Qualytics should read from. You fill these in on both flows.

Datastore Properties

Common fields for every source datastore, shown below the Datastores Extraction section. You fill these in on both flows.

Steps

There are two ways to set up the connection: reuse a connection you already saved (Existing Connection) or create a new one from scratch (New Connection). The tabs below walk through each option; pick the one you want to follow. Each field is described in the Field reference above.

Add Enrichment Datastore

An enrichment datastore is where Qualytics writes what it finds: anomalies, remediation tables, and record enrichment. Databricks is supported for this role, so the same workspace can hold both the data you monitor and the results.

Linking one during datastore creation is recommended, because scan results are then persisted from the very first operation. The flow itself is the same for every connector and is documented once, in Link Enrichment on Datastore Creation: reach it by clicking Next instead of Finish on the source datastore you just configured. To link one to a datastore that already exists, see Link Enrichment Datastore instead.

The rest of this section covers only what is specific to Databricks: the fields you fill in when the enrichment datastore is a Databricks one.

Field reference

The Enrichment Datastore step shows the sections below when Databricks is selected. When reusing an existing connection, the Connection Properties, Authentication, and Secrets Management sections come already filled in and read-only.

Connection Properties

These fields define the Databricks endpoint Qualytics writes to. They are the same fields as on the source datastore flow, repeated here so this section stands on its own.

Authentication

Pick the tab that matches the credential you use. The account behind it needs write access, not just read.

Secrets Management

This group is optional: use it only if you want Qualytics to pull credentials from a secrets manager instead of typing them into the form. Turn on HashiCorp Vault to show the fields below. Despite the label, any secrets manager that exposes a compatible REST API works, not only HashiCorp Vault; see Secrets Management.

Enrichment Extraction

Where Qualytics writes the enrichment tables inside the workspace.

Enrichment Properties

Advanced Options

Collapsed by default. Expand it to change how anomalous source records are replicated.

Steps

A Databricks enrichment datastore can be created from two places: as the second step of creating a source datastore, or on its own from the Enrichment Datastores page. Either way you choose between creating a connection from scratch (New Connection) or reusing a saved one (Existing Connection). The tabs below cover both entry points for each option; each field is described in the Field reference above.

API Payload Examples

This section provides detailed examples of API payloads to guide you through the process of creating and managing datastores using Qualytics API. Each example includes endpoint details, sample payloads, and instructions on how to replace placeholder values with actual data relevant to your setup.

Creating a Source Datastore

This section provides sample payloads for creating a Databricks datastore. Replace the placeholder values with actual data relevant to your setup.

Endpoint: /api/datastores (post)

{
    "name": "your_datastore_name",
    "teams": ["Public"],
    "database": "databricks_catalog",
    "schema": "databricks_database",
    "enrichment_only": false,
    "trigger_sync": true,
    "connection": {
        "name": "your_connection_name",
        "type": "databricks",
        "host": "databricks_host",
        "password": "databricks_token",
        "parameters": {
            "path": "databricks_http_path"
        }
    }
}

{
    "name": "your_datastore_name",
    "teams": ["Public"],
    "database": "databricks_catalog",
    "schema": "databricks_database",
    "enrichment_only": false,
    "trigger_sync": true,
    "connection_id": 123
}

{
    "name": "your_datastore_name",
    "teams": ["Public"],
    "database": "databricks_catalog",
    "schema": "databricks_database",
    "enrichment_only": false,
    "trigger_sync": true,
    "connection": {
        "name": "your_connection_name",
        "type": "databricks",
        "host": "databricks_host",
        "username": "service_principal_application_id",
        "password": "service_principal_oauth_secret",
        "parameters": {
            "path": "databricks_http_path",
            "authentication_type": "OAUTH_M2M"
        }
    }
}

# Step 1: Create a Connection
qualytics connections create \
    --type databricks \
    --name "your_connection_name" \
    --host ${DATABRICKS_HOST} \
    --password ${DATABRICKS_TOKEN} \
    --parameters '{"path": "databricks_http_path"}'

# Step 2: Create a Source Datastore
qualytics datastores create \
    --name "your_datastore_name" \
    --connection-name "your_connection_name" \
    --database hive_metastore \
    --schema default

Creating an Enrichment Datastore

This section provides sample payloads for creating an enrichment datastore. Replace the placeholder values with actual data relevant to your setup.

Endpoint: /api/datastores (post)

{
    "name": "your_datastore_name",
    "teams": ["Public"],
    "database": "databricks_enrichment_catalog",
    "schema": "databricks_enrichment_database",
    "enrichment_only": true,
    "connection": {
        "name": "your_connection_name",
        "type": "databricks",
        "host": "databricks_host",
        "password": "databricks_token",
        "parameters": {
            "path": "databricks_http_path"
        }
    }
}

{
    "name": "your_datastore_name",
    "teams": ["Public"],
    "database": "databricks_enrichment_catalog",
    "schema": "databricks_enrichment_database",
    "enrichment_only": true,
    "connection_id": 123
}

# Step 1: Create a Connection
qualytics connections create \
    --type databricks \
    --name "your_connection_name" \
    --host ${DATABRICKS_HOST} \
    --password ${DATABRICKS_TOKEN} \
    --parameters '{"path": "databricks_http_path"}'

# Step 2: Create an Enrichment Datastore
qualytics datastores create \
    --name "your_datastore_name" \
    --connection-name "your_connection_name" \
    --database hive_metastore \
    --schema your_enrichment_schema \
    --enrichment-only

Link an Enrichment Datastore to a Source Datastore

Use the provided endpoint to link an enrichment datastore to a source datastore:

Endpoint Details: /api/datastores/{datastore-id}/enrichment/{enrichment-id} (patch)