Hive Native Troubleshooting

This page documents the known problems with a Hive Native datastore and the steps to resolve them. A Hive Native connection depends on three things in sequence, reaching the metastore, authenticating to it, and reaching the storage the metastore points at, and each section below covers one of them in that order. A failure at the storage step often reads like a broken connection even though the connection itself is fine, so work through the sections top to bottom.

Reaching the Metastore

The Connection Test Reports the Metastore as Unreachable

The Test connection button fails before anything else has been attempted. Qualytics has not authenticated and has not looked at storage yet.

Cause: The Metastore URI is wrong, port 9083 is blocked between your Qualytics deployment and the metastore, or the metastore is not running.

Resolution: Confirm the address, in the form thrift://host:9083, listing every address separated by commas for a highly available metastore. Then allow the Public addresses or Private addresses shown in the Add Datastore form through your security groups or firewall rules, and confirm the metastore service is up.

The connection test passes, but the Database dropdown offers nothing, or the first Sync completes with no tables.

Cause: Qualytics reached the metastore, but the identity it presented cannot see the databases. On a Kerberized metastore that identity comes from the keytab on the connection, or from the deployment's credentials when the connection carries none.

Resolution: Grant that identity read access to the Hive databases you want to monitor, then test again. See Permissions.

Authenticating

Authentication can fail before the metastore is contacted. Check that the realm is reachable, the keytab is current, the Client Principal matches an identity in that keytab, and the Metastore Service Principal names the metastore. See Authentication for the distinction between the two identities.

Cause: The Client Principal does not match an identity in the keytab. Older connections may have a metastore principal such as hive/_HOST@EXAMPLE.COM stored in the client field.

Resolution: Set Client Principal to the account in the keytab, for example qualytics@EXAMPLE.COM. Put hive/_HOST@EXAMPLE.COM in Metastore Service Principal, or leave that field empty when its default matches your cluster. Test the connection again.

If an older connection uses deployment credentials and has no keytab of its own, its saved service/_HOST@REALM value is not used as the metastore service identity. Qualytics uses hive/_HOST@ followed by the realm of the deployment's login principal instead, which fails when the metastore is in a different realm. Move the saved value to Metastore Service Principal and enter the deployment's login principal as Client Principal.

The Connection Is Refused Because a Realm Is Already Registered

Saving or testing the connection fails with a message that its realm is already registered by another datastore.

Cause: Another connection on this deployment uses the same realm name with a different Key Distribution Center. One realm name cannot mean two clusters, so the second connection is refused rather than quietly reading from the wrong one.

Resolution: Confirm which cluster the realm belongs to. Two clusters that share a realm name need distinct names, or separate deployments.

A Kerberos Connection Fails Right After a Credential Change

A connection that worked yesterday fails today, and a credential was changed on the cluster in between.

Cause: The keytab stored on the connection no longer matches the key the Key Distribution Center holds. Qualytics renews tickets from the keytab on its own, so there is nothing to run by hand, but a keytab from before the change cannot renew anything.

Resolution: Upload the current keytab on the connection and test again. The change reaches every datastore that shares the connection.

The Keytab Upload Is Rejected

The Keytab field refuses the file you picked.

Cause: The file is not a keytab, or it is not named with a .keytab extension. Qualytics reads the upload according to its extension, and a keytab is binary. Uploading a krb5.conf into the Keytab field produces the same rejection.

Resolution: Upload the keytab itself, renamed to end in .keytab, and put the krb5.conf in its own field.

The Connection Fails With a Protection Level Error

The connection test or the Sync fails with a Kerberos or SASL error from the metastore, even though the principals and the keytab are correct.

Cause: The metastore requires a different RPC protection level from the one the connection uses. When RPC Protection is empty, Qualytics uses Authentication only.

Resolution: Ask your Hadoop administrator which protection level the metastore requires, set RPC Protection to match (Integrity (signed) or Privacy (encrypted)), and test the connection again. See RPC protection.

An Error Names a Missing Server Principal

Authentication to the metastore succeeds, but reading data fails with an error about a server principal Qualytics could not find.

Cause: Qualytics works out the storage service's principal from the realm of the Metastore Service Principal. Your cluster names its storage service differently from that default.

Resolution: Ask your Hadoop administrator for the storage service principal used on the cluster.

Reaching Storage

This is the step that most often looks like something else. Reading table information never touches storage, so a datastore whose storage is unreachable connects cleanly, syncs cleanly, lists every table, and then fails on the first operation that reads records.

Reading Table Files Fails With UnknownHostException

The Sync lists the tables, and the first Profile or Scan fails with an UnknownHostException naming a host that is not a real machine.

Cause: The table locations use a highly available HDFS nameservice, such as hdfs://<nameservice>/..., and Qualytics has no HDFS client configuration to resolve it.

Resolution: Upload the cluster's hdfs-site.xml in the hdfs-site.xml field of the connection. If the upload is refused with The uploaded hdfs-site.xml could not be read as a Hadoop configuration file, check that the file is valid XML. If it is refused with The uploaded hdfs-site.xml contains no dfs.* properties, upload the cluster's HDFS client file rather than another configuration file. See HDFS client configuration.

The Sync Succeeds but Profiling or Scanning Fails

The connection test and the Sync both pass. The first Profile or Scan fails.

Cause: Table information can be read from the metastore while the data files cannot be reached. Two causes account for almost all of these: the NameNode or DataNode ports are not open to your Qualytics deployment, or the hostnames recorded in the table locations do not resolve outside the cluster. Qualytics follows those locations exactly as the metastore reports them.

Resolution: Allow the NameNode and DataNode ports listed in Permissions, add the storage hostnames to the DNS your deployment uses if they resolve only inside the cluster, and confirm the identity Qualytics presents can read the files, not only list the tables.

A Read Fails and the Message Names a Missing HDFS Token

A Profile or Scan on a Kerberized cluster fails while reading table files, and the message says the read had no valid HDFS delegation token, followed by the reason.

Cause: Qualytics reads HDFS with short-lived tokens it requests for the identity on the connection, hands them to every worker it adds while an operation runs, and requests new ones several times a day, well inside the time a cluster normally allows a token. The message says why the worker had no valid token, for example: the token request failed, with the error the cluster returned, neither the connection nor the deployment has a keytab to request one with, no token for that filesystem had reached the worker that read the file, or the token presented was rejected because it had expired or been cancelled. When Qualytics had already tried to replace a rejected token, the message also says why that failed.

Resolution: Act on the reason given. A failed request, or a replacement not obtained, carries the error from the cluster: when it says the Key Distribution Center or the NameNode could not be reached, check the ports listed in Permissions, and when it says the login from the keytab was refused, see A Kerberos Connection Fails Right After a Credential Change. When no keytab is available, upload one on the connection, or leave the field empty only when the deployment has its own Kerberos login. A rejected token with no further reason either expired or was cancelled, and the message does not say which. Qualytics keeps a token until its scheduled replacement, so the same error can repeat for a while and running the operation again does not help. Ask the cluster administrator which it was: a token that expired points at a cluster token lifetime set to a few hours or less, and a cancelled one at an action taken on the cluster. Neither means the keytab is outdated. Reads no longer fail on their own while the cluster adds workers during an operation.

A Sync Takes Much Longer Than Expected

The Sync completes, but far more slowly than the size of the database suggests.

Cause: The tables have a very large number of partitions, or storage is responding slowly.

Resolution: Confirm storage is reachable and responsive, then narrow the datastore to the databases you actually need.

Reading Tables

A table Qualytics declines is reported individually, and the rest of the database syncs normally. This is a refusal rather than a failure. Reading an ACID table's files directly would ignore its delete records, and reading a Delta, Hudi, Iceberg, or Paimon table's files directly would return rows those formats no longer consider current. Refusing is safer than returning data that quietly disagrees with what Hive itself reports.

A Table Is Reported as Unsupported

The Sync completes, but one or more tables appear as unsupported rather than as containers.

Cause: The table is transactional (ACID), a view, backed by a storage handler such as HBase or Kudu, a Delta, Hudi, Iceberg, or Paimon table registered in Hive, registered with SymlinkTextInputFormat, uses an input format or SerDe that is not available to Qualytics, or was created with a USING data source other than Parquet, ORC, CSV, or JSON. The reason is reported with each table. The full list is in Supported tables and formats.

Resolution: Read that table through the Hive connector. Both connectors can point at the same cluster at the same time.

A File Does Not Match the Table's Schema

A Profile or Scan fails with a message naming the table, a column, the type the metastore declares for it, and the type one of the files stores it as, ending with The table's files don't match its schema. Fix the table definition or rewrite the mismatched files.

Cause: The table definition says one thing and a data file says another, for example a column declared as a string in Hive while a Parquet file stores it as a timestamp. Qualytics reads the files as the metastore describes them, so the two have to agree.

Resolution: Correct the table definition in Hive, or rewrite the files that disagree with it. The operation stops at the first such file instead of retrying it, so the message names the file to fix.

A File Is Not Valid Parquet

A Profile or Scan fails with a message naming the table and a file, ending with not a valid Parquet file (truncated, or not Parquet). Fix or remove the file.

Cause: The file is truncated, was written by a job that did not finish, or is not a Parquet file at all. The operation stops at that file instead of retrying it.

Resolution: Fix or remove the file, then run the operation again. Files too short to hold any data, such as a leftover marker of a few bytes, are skipped rather than failing the read, and the operation reports them in a warning that names them. Empty files are skipped without a warning.

A Scan Is Refused over a Column Name That Differs Only in Case

The Scan stops with a message that a column's name in the data files does not match the one Hive records, apart from its case.

Cause: Hive records column names in lower case while the data files keep the case they were written with. Qualytics confirms the two agree before reading, and refuses the Scan when they differ rather than returning a column full of empty values.

Resolution: Rewrite the affected files with the column names Hive records, or read that table through the Hive connector.

Limitations

File-based tables only. Parquet, ORC, delimited text, JSON, Avro, and other SerDe-based tables are read. Views, transactional tables, storage-handler tables, Delta, Hudi, Iceberg, and Paimon tables registered in Hive, and the other cases in Supported tables and formats are declined rather than read incorrectly.
Read-only. Hive Native cannot be used as an enrichment datastore. Link a separate enrichment datastore as its destination for the anomalies and metadata Qualytics produces.
Files are read as the metastore describes them. A file whose columns disagree with the table definition, or a corrupt file, fails the operation with a message naming the file rather than being skipped. Only files too short to hold data are skipped.
Kerberos only. The metastore protocol has no username and password authentication. To connect to Hive as a specific named user, use the Hive connector.
One realm name per cluster. Two clusters that use the same realm name with different Key Distribution Centers cannot coexist on one deployment.