How Hive Native Works¶

Hive Native reads table information from the metastore and then reads the data files straight from storage. This page follows one read from start to finish on a Kerberized HDFS cluster: which service Qualytics talks to at each step, on which port, and where your cluster's access policies are applied.

Every connection in this flow is opened by your Qualytics deployment toward your cluster. Nothing on the cluster opens a connection back to Qualytics.

Services involved¶

flowchart TD
    Q["Qualytics deployment"]
    subgraph Cluster["Your Hive cluster"]
        KDC["Kerberos KDC"]
        HMS["Hive metastore"]
        NN["HDFS NameNodes"]
        DN["HDFS DataNodes"]
        KMS["Key Management Server<br/>(encryption zones only)"]
    end
    Q -->|"88"| KDC
    Q -->|"9083"| HMS
    Q -->|"8020"| NN
    Q -->|"1004 or 9866"| DN
    Q -.->|"9494 HTTPS"| KMS

The ports shown are the common defaults, and your cluster may use different ones. Permissions lists each service and how to check its port.

A read, step by step¶

The first four steps prepare the connection and are reused across operations. The last three run for every data file an operation opens.

sequenceDiagram
    participant Q as Qualytics
    participant KDC as Kerberos KDC
    participant HMS as Hive metastore
    participant NN as NameNodes
    participant KMS as Key Management Server
    participant DN as DataNodes

    rect rgba(127, 127, 127, 0.08)
    Note over Q,DN: Setup, reused across operations
    Q->>KDC: 1. Log in with the keytab (88)
    Q->>HMS: 2. Read tables, columns, and partition locations (9083)
    Q->>NN: 3. Get an HDFS delegation token (8020)
    Q->>KMS: 4. Get a KMS delegation token (9494)
    end

    rect rgba(127, 127, 127, 0.16)
    Note over Q,DN: For each data file
    Q->>NN: 5. Open the file: permission check and block locations (8020)
    Q->>KMS: 6. Decrypt the file's key (9494)
    Q->>DN: 7. Read the file's blocks (1004 or 9866)
    end

    Note over Q: Decrypt in memory and compute results

1. Log in. Qualytics authenticates to the KDC as the connection's Client Principal, using the keytab stored on the connection. A connection with no keytab of its own logs in with the deployment's principal and keytab instead, which only older self-hosted deployments have. See Hive Native Authentication.
2. Read the metastore. Databases, tables, columns, and the location of every table and partition come from the metastore. Partitions are taken from the metastore rather than from listing storage.
3. Get an HDFS delegation token. The token lets the processes that read files act as the same principal without each one logging in to Kerberos. Qualytics requests a fresh one every few hours, well before it expires.
4. Get a KMS delegation token. When the cluster runs a Key Management Server, a KMS token is requested alongside the HDFS token. If the KMS cannot be reached, reads outside encryption zones still work, while reads inside one fail with a decryption error.
5. Open the file. The NameNode checks that the principal may read the file and returns where its blocks are. This is where your HDFS permissions and Ranger HDFS policies apply.
6. Decrypt the file's key. Only for files in an HDFS encryption zone. The NameNode hands back the file's key in encrypted form, and Qualytics asks the KMS to decrypt it. This is where your Ranger KMS policies apply. See Encryption zones.
7. Read the blocks. The DataNodes serve the file's blocks. For an encrypted file, the blocks travel as ciphertext.

Steps 4 and 6 are the only ones that reach the KMS. Step 4 runs whenever the NameNode reports a KMS for the cluster, and step 6 only for files in an encryption zone. Tables in Amazon S3, Azure Data Lake Storage, or Google Cloud Storage skip steps 3 to 7, because their files are read from object storage with the credentials your deployment is already configured with.

Large partitioned tables¶

A partitioned Hive table can hold thousands of partitions and billions of rows. Qualytics reads such a table so that an operation costs about as much as the data it needs, not as much as the whole table.

Newest partitions first¶

A preview, and a profile or scan with a record limit, read the newest partitions first. Newest means the highest partition values, compared as the type each partition column is declared with, so on an integer hour column 23 is newer than 9.

• A profile or scan with a record limit on a partitioned table that holds more rows than the limit reads whole partitions, newest first, until it has enough rows, and cuts only the oldest of them short. It never returns more rows than the limit, and can return fewer, for example when Hive's statistics for those partitions are out of date. Partitions with no row to read are skipped. On a table whose first partition column holds dates as text (see below), partitions whose value is not a date are read last.
• A preview shows rows from the newest partitions. A preview with a filter keeps searching older partitions until it has the rows it needs or has searched every partition, so a filter that matches few rows, or none, can take about as long as reading the whole table.

A limited operation therefore checks the newest data rather than a sample spread across the whole table. A check inferred from a profile of older data can raise anomalies the first time a limited scan reads only the newest partitions.

Row counts¶

A profile does not count the rows of a table, and a scan counts them only when it runs a Min Partition Size or Max Partition Size check, which need the record count. That count covers the records the scan selects: only the new ones on an Incremental scan, and only the matching ones when the scan has a filter. On a very large table, counting every row can take longer than reading the records the operation needs.

The total an operation reports comes from the rows it read, when it read every row, or from the row count in Hive's table statistics. When Hive's statistics are out of date, that total can differ from an exact count. When neither is available, the operation reports no total. See Keep Hive's table statistics current.

Incremental scans on a text partition column¶

Many Hive tables are partitioned by a date or a batch number stored as text, such as datepart='20261112'. Text compares character by character, which puts 9 after 10, so a field the source stores as text normally cannot be an incremental field. On Hive Native the table's first partition column is the exception. Qualytics reads its values from the partition names in the metastore, puts them in order as dates or as numbers, and an incremental scan reads only the partitions after the last one it scanned, without listing or opening the older ones.

Dates as text. Qualytics recognizes a first partition column declared as string when more than half of its values are real dates in one of these formats, with years from 1900 to 2200:

Only formats whose text order is also their time order are recognized. Formats such as dd-MM-yyyy or MM/dd/yyyy, and month or year values such as yyyyMM or yyyy, are not, because short numeric IDs could be mistaken for them. Values are read as UTC.

When Sync finds such a column, it sets the column as the table's Last Modified field, both when it creates the container and when it refreshes one that has no incremental field. It never replaces an incremental field that is already set. A table whose incremental field is another column is read by that column instead, and you can switch it to the partition column in the table's settings.

An incremental scan then reads the partitions after the newest one it already scanned, to the unit of the format: whole days for yyyyMMdd, whole hours for yyyyMMddHH. The newest partition read becomes the next scan's starting point. Rows added later to a partition that was already scanned are not read by the next incremental scan. To cover them, run an Incremental scan with a Greater Than Time Starting Threshold before that partition, or a Full scan.

Batch numbers as text. Set the column as the table's Batch Value field. Qualytics reads its values as whole numbers, so batch 10 follows batch 9, and an incremental scan reads only the batches greater than the last one it scanned. The number is only an order and is never read as a time. A value counts as a number when it is plain digits, at most 16 of them, no larger than 9,007,199,254,740,992. An epoch in nanoseconds, for example, is too large.

There is no limit on how many new batches one incremental scan reads. When more than 1,000 batches follow the last one scanned, for example after a long pause between scans, the scan still reads all of them, and only them, in that one scan. To find them, though, Qualytics fetches the details of every partition of the table from the metastore, as a Full scan does, so on a table with many partitions that scan takes longer to start. No warning is shown.

On Hive Native, any field stored as text can be picked as the Batch Value field, but a scan fails with a configuration error when that field is not the table's first partition column. See A Scan Fails Over a Text Incremental Field.

Values that are not dates or numbers. A partition whose value is not a date, or not a whole number for a batch column, such as a stray datepart='test', is left out of incremental scans. A Full scan still reads it. The operation reports a warning on the table, with the code non_date_partitions_skipped or non_numeric_partitions_skipped, that names the column and up to five of the values. See A Warning Says Partitions Were Left Out of Incremental Scans.

Record limits and incremental scans¶

A scan with a record limit can leave some of the records it selected unread, and the next Incremental scan does not always go back for them. Where that scan starts depends on the table's incremental field:

• The first partition column, holding dates or batch numbers as text. The next scan starts after the newest partition of the table, even when the limit stopped the scan before it reached the older ones. Partitions the limit left out are not read by later Incremental scans.
• Any other incremental field, such as a timestamp or numeric column. The next scan starts after the highest value among the records the scan read. Records the limit left out are read by a later Incremental scan only when their value is higher than that.
• Any other incremental field, on a scan that counts its records (one that runs a Min Partition Size or Max Partition Size check, see Row counts), or when the field is a computed field or one of the table's excluded fields. The next scan starts after the highest value among all the records the scan selected, including those the limit left out, so later Incremental scans do not read them.

To check records that a limited scan left out, run a Full scan, or an Incremental scan with a Starting Threshold before them, without a record limit.

Encryption zones¶

On clusters that use HDFS transparent encryption, such as Cloudera CDP with Ranger KMS, the files of an encryption zone are stored encrypted, and only the client that reads them decrypts them. The NameNode holds each file's key in encrypted form and the DataNodes hold ciphertext, so nothing on the cluster can hand plaintext to a reader that goes straight to the files. That is why Hive Native, unlike the Hive connector, needs to reach the KMS.

That is the only KMS permission involved:

• The zone key itself never leaves the KMS.
• Decrypt EEK gives no access to key material and no ability to list, create, roll, or delete keys.
• Every decrypt call is recorded in the KMS audit log under the principal Qualytics logs in as in step 1.
• Revoking that one permission stops all reads from the zone, even while HDFS access is still granted.

Decryption happens in memory inside your Qualytics deployment, on the data being profiled or scanned.

What Qualytics does not do¶

• It does not write to your cluster. The connector cannot create, alter, or drop databases or tables through the metastore, and it does not write files to your HDFS.
• It does not apply Hive's fine-grained policies. Because the data files are read directly, Ranger Hive column masking and row filters are not applied. Access is decided by the HDFS and KMS checks in steps 5 and 6. Use the Hive connector for tables that depend on Hive-level policies.
• It does not need administrative access. No HDFS web UIs, WebHDFS, HttpFS, Cloudera Manager, Ranger admin, or SSH access is required.

See Also¶