Dremio

Adding and configuring a Dremio connection within Qualytics empowers the platform to build a symbolic link with your schema to perform operations like data discovery, visualization, reporting, syncing, profiling, scanning, anomaly surveillance, and more.

This documentation provides a step-by-step guide on adding Dremio as a source datastore in Qualytics. It covers the entire process, from initial connection setup to testing and finalizing the configuration.

By following these instructions, enterprises can ensure their Dremio environment is properly connected with Qualytics, unlocking the platform’s potential to help you proactively manage your full data quality lifecycle.

Let’s get started 🚀

Dremio Setup Guide

Qualytics connects to Dremio through the Dremio JDBC driver, using a direct connection to the endpoint. It uses standard SQL queries for data profiling and scanning. Dremio permissions are managed through its own privilege model, which controls access to sources, spaces, and datasets.

Minimum Dremio Permissions (Source Datastore)

Authentication Methods

Dremio supports two authentication methods:

Example: Granting Permissions in Dremio

In the Dremio UI, navigate to the target source or dataset and grant SELECT access to the Qualytics user:

GRANT SELECT ON TABLE <source>.<schema>.<dataset> TO USER qualytics_read;
-- Or grant access to all datasets in a schema:
GRANT SELECT ON SCHEMA <source>.<schema> TO USER qualytics_read;

Troubleshooting Common Errors

Detailed Troubleshooting Notes

Authentication Errors

The error Authentication failed indicates that the credentials are incorrect.

Common causes:

• Incorrect password: the password does not match the Dremio user account.
• Invalid PAT: the Personal Access Token is expired, revoked, or was generated in a different Dremio instance.
• PAT permissions: the PAT does not have sufficient permissions for the requested operation.

Permission Errors

The error Permission denied on dataset means the user authenticated successfully but lacks access to the target dataset.

Common causes:

• Missing SELECT on dataset: the user does not have SELECT permission on the specific dataset.
• Source not shared: in Dremio, datasets derived from a source must be explicitly shared with users.
• Project scope (Cloud only): the dataset exists in a different project than the one configured in the connection.

Connection Errors

The error Connection refused or SSL handshake failed indicates a connectivity issue.

Common causes:

• Wrong port: Dremio Cloud listens on 443, while a self-managed coordinator listens on 31010. The form's pre-filled 32010 belongs to a different Dremio driver and will not connect.
• SSL not configured: the Dremio server requires SSL but the connection is not configured for it.
• Project ID missing (Cloud only): Dremio Cloud requires a Project ID to scope queries.

Add the Source Datastore

A source datastore is a storage location Qualytics connects to so it can profile, scan, and monitor data. Adding Dremio as a source lets Qualytics query your datasets through the Dremio JDBC driver and run quality operations on the tables it discovers.

Before you start, review the Minimum Dremio Permissions the connecting account needs and the available Authentication Methods.

Field reference

The Add Datastore page shows the sections below when Dremio is selected. When reusing an existing connection, the Connection Properties, Authentication, and Secrets Management sections come already filled in and read-only: Qualytics has already validated those credentials, so you fill in only the Datastores Extraction and the Datastore Properties. To change a saved connection's credentials, edit the connection through the Manage Connections page; edits there apply to every datastore that reuses the connection.

Connection Properties

These fields define the Dremio endpoint Qualytics connects to. They belong to the connection: when reusing an existing connection, they come already filled in and read-only.

Authentication

Choose how Qualytics authenticates to Dremio. Setting Type changes the credential fields shown below it, so pick the tab that matches your choice. These fields also belong to the connection: already filled in and read-only when reusing one. See Authentication Methods above for how to obtain each credential.

Secrets Management

This group is optional: use it only if you want Qualytics to pull credentials from a secrets manager instead of typing them into the form. Turn on HashiCorp Vault to show the fields below. Despite the label, any secrets manager that exposes a compatible REST API works, not only HashiCorp Vault; see Secrets Management. It also belongs to the connection: read-only when reusing an existing connection.

Datastores Extraction

Pick the schema(s) Qualytics should read from. You fill these in on both flows.

Datastore Properties

Common fields for every source datastore, shown below the Datastores Extraction section. You fill these in on both flows.

Steps

There are two ways to set up the connection: reuse a connection you already saved (Existing Connection) or create a new one from scratch (New Connection). The tabs below walk through each option; pick the one you want to follow. Each field is described in the Field reference above.

API Payload Examples

This section provides detailed examples of API payloads to guide you through the process of creating and managing datastores using Qualytics API. Each example includes endpoint details, sample payloads, and instructions on how to replace placeholder values with actual data relevant to your setup.

Creating a Source Datastore

This section provides sample payloads for creating a Dremio datastore. Replace the placeholder values with actual data relevant to your setup.

Endpoint (Post): /api/datastores (post)

{
    "name": "your_datastore_name",
    "teams": ["Public"],
    "schema": "dremio_schema",
    "enrichment_only": false,
    "trigger_sync": true,
    "connection": {
        "name": "your_connection_name",
        "type": "dremio",
        "host": "sql.dremio.cloud",
        "port": 443,
        "password": "your_personal_access_token",
        "parameters": {
            "project_id": "7c4d9f21-3b8e-4a6c-9d51-2f8b6e04a1c3",
            "ssl": true,
            "authentication_type": "PAT"
        }
    }
}

{
    "name": "your_datastore_name",
    "teams": ["Public"],
    "schema": "dremio_schema",
    "enrichment_only": false,
    "trigger_sync": true,
    "connection_id": 123
}

# Step 1: Create a Connection
qualytics connections create \
    --type dremio \
    --name "your_connection_name" \
    --host sql.dremio.cloud \
    --port 443 \
    --username ${DREMIO_USER} \
    --password ${DREMIO_PASSWORD}

# Step 2: Create a Source Datastore
qualytics datastores create \
    --name "your_datastore_name" \
    --connection-name "your_connection_name" \
    --schema your_schema