Hive

Adding and configuring a Hive connection within Qualytics empowers the platform to build a symbolic link with your schema to perform operations like data discovery, visualization, reporting, syncing, profiling, scanning, anomaly surveillance, and more.

This documentation provides a step-by-step guide on how to add Hive as a source datastore in Qualytics. It covers the entire process, from initial connection setup to testing and finalizing the configuration.

By following these instructions, enterprises can ensure their Hive environment is properly connected with Qualytics, unlocking the platform's potential to help you proactively manage your full data quality lifecycle.

Let’s get started 🚀

Hive Setup Guide

Qualytics connects to Hive through the Hive JDBC driver (HiveServer2). It accesses the Hive metastore for schema and table discovery, and reads table data via HiveQL SELECT queries for profiling and scanning operations. Qualytics also identifies partition columns from Hive metastore metadata for optimized data reading.

Minimum Hive Permissions (Source Datastore)

Example: Source Datastore User (Read-Only)

Replace <database_name> with your actual value.

-- Grant read access to the target database and all its tables
GRANT SELECT ON DATABASE <database_name> TO USER qualytics_read;
GRANT SELECT ON ALL TABLES IN DATABASE <database_name> TO USER qualytics_read;

Troubleshooting Common Errors

Detailed Troubleshooting Notes

Authentication Errors

The error GSS initiate failed (Kerberos) or Could not open client transport indicates an authentication or transport problem.

Common causes:

• Stale keytab: the credential was changed on the cluster and the keytab on the connection is the previous one. Upload the current keytab; Qualytics renews tickets from it on its own, so there is nothing to run by hand.
• Wrong principal: the Client Principal does not match the identity in the keytab, or the HiveServer2 Service Principal does not match the one configured in the Hive server.
• KDC unreachable: the Key Distribution Center named in the uploaded krb5.conf is not reachable from your Qualytics deployment.
• Realm already claimed: another connection uses the same realm name with a different KDC. One realm name cannot mean two clusters, so give each a distinct realm name or connect them from separate deployments.
• HiveServer2 not running: the HiveServer2 process is not started or has crashed.

Permission Errors

The error Permission denied: user does not have SELECT privilege means the user authenticated successfully but lacks the necessary Hive grants.

Common causes:

• Missing SELECT on database: the user does not have SELECT on the target database.
• Missing SELECT on table: the user has database-level access but not table-level access.
• Ranger/Sentry policy: if Apache Ranger or Sentry is enabled, permissions are managed through policies rather than Hive GRANT statements.

Connection Errors

The error Could not open client transport or User is not allowed to impersonate indicates a transport or proxy user issue.

Common causes:

• HiveServer2 not reachable: the host or port (default 10000) is incorrect.
• Proxy user not allowed: the hadoop.proxyuser.<hive_user>.users property in core-site.xml does not include the Qualytics user.
• SSL required: HiveServer2 requires SSL but the connection is not configured for it.

Add a Source Datastore

A source datastore is a storage location Qualytics connects to so it can profile, scan, and monitor data. Adding Hive as a source lets Qualytics query it through the Hive JDBC driver and run quality operations on the tables it discovers.

Before you start, review the Minimum Hive Permissions the connecting user needs.

Field reference

The Add Datastore page shows the sections below when Hive is selected. When reusing an existing connection, the Connection Properties and Secrets Management sections come already filled in and read-only: Qualytics has already validated those credentials, so you fill in only the Location and the General fields. To change a saved connection's credentials, edit the connection through the Manage Connections page; edits there apply to every datastore that reuses the connection.

Connection Properties

These fields define the HiveServer2 endpoint Qualytics connects to. They belong to the connection: when reusing an existing connection, they come already filled in and read-only.

Authentication

Choose how Qualytics authenticates to HiveServer2. Setting Type changes the credential fields shown below it, so pick the tab that matches your choice. These fields also belong to the connection: already filled in and read-only when reusing one.

Secrets Management

This group is optional: use it only if you want Qualytics to pull credentials from a secrets manager instead of typing them into the form. Turn on HashiCorp Vault to show the fields below. Despite the label, any secrets manager that exposes a compatible REST API works, not only HashiCorp Vault; see Secrets Management. It also belongs to the connection: read-only when reusing an existing connection.

Location

Pick the schema or schemas Qualytics should read from. You fill these in on both flows.

General

Common fields for every source datastore, shown below the Location section. You fill these in on both flows.

Below the form, an information banner lists the Public addresses and Private addresses your datastore connections originate from. Allow the ones that match your network setup through your security groups or firewall rules; see How Connections Work for where the addresses come from.

Steps

There are two ways to set up the connection: reuse a connection you already saved (Existing Connection) or create a new one from scratch (New Connection). The tabs below walk through each option; pick the one you want to follow. Each field is described in the Field reference above.

API Payload Examples

This section provides detailed examples of API payloads to guide you through the process of creating and managing datastores using Qualytics API.

Each example includes endpoint details, sample payloads, and instructions on how to replace placeholder values with actual data relevant to your setup.

Creating a Datastore

This section provides a sample payload for creating a datastore. Replace the placeholder values with actual data relevant to your setup.

Endpoint (Post)

/api/datastores (post)

    {
        "name": "your_datastore_name",
        "teams": ["Public"],
        "schema": "hive_schema",
        "enrichment_only": false,
        "trigger_sync": true,
        "connection": {
            "name": "your_connection_name",
            "type": "hive",
            "host": "hive_host",
            "port": 10000,
            "username": "hive_username",
            "password": "hive_password",
            "parameters": {
                "zookeeper": false,
                "authentication_type": "BASIC"
            }
        }
    }

    {
        "name": "your_datastore_name",
        "teams": ["Public"],
        "schema": "hive_schema",
        "enrichment_only": false,
        "trigger_sync": true,
        "connection": {
            "name": "your_connection_name",
            "type": "hive",
            "host": "hive_host",
            "port": 10000,
            "username": "qualytics@EXAMPLE.COM",
            "password": "<base64-encoded keytab>",
            "parameters": {
                "zookeeper": false,
                "authentication_type": "KERBEROS",
                "hive.metastore.kerberos.principal": "hive/_HOST@EXAMPLE.COM",
                "krb5_base64": "<krb5.conf contents>"
            }
        }
    }

    {
        "name": "your_datastore_name",
        "teams": ["Public"],
        "schema": "hive_schema",
        "enrichment_only": false,
        "trigger_sync": true,
        "connection_id": 123
    }

# Step 1: Create a Connection
qualytics connections create \
    --type hive \
    --name "your_connection_name" \
    --host ${HIVE_HOST} \
    --port 10000 \
    --username ${HIVE_USER} \
    --password ${HIVE_PASSWORD}

# Step 2: Create a Source Datastore
qualytics datastores create \
    --name "your_datastore_name" \
    --connection-name "your_connection_name" \
    --schema default

Link an Enrichment Destination

Endpoint (Patch)

/api/datastores/{datastore-id}/enrichment/{enrichment-id} (patch)