Introduction to the Hive Native Connector

Apache Hive organizes data held in HDFS or cloud object storage into databases and tables, and records in its metastore where every table and partition lives. Most Hive environments expose that data through HiveServer2, which compiles each query, runs it on the cluster, and returns the rows.

The Qualytics Hive Native connector takes a shorter path. It reads table information from the metastore and then reads the underlying Parquet and ORC files directly, leaving the cluster's query engine out of the read path, which is considerably faster on large tables. Partitions come from the metastore rather than from listing storage, so Qualytics profiles tables, runs scheduled scans, and surfaces record- and schema-level anomalies on results that match what Hive itself reports.

Notes

Choosing between the two Hive connectors. Qualytics keeps both, and they are good at different things. Use Hive Native for the Parquet and ORC tables that make up most large Hive environments, and the Hive connector for anything the native path declines to read. Both can be connected against the same cluster at the same time.

Supported tables and formats. The connector reads tables stored as Parquet or ORC, managed or external, partitioned or not. Anything else is declined rather than read incorrectly, and each declined table is reported on its own while the rest of the database syncs normally. Use the Hive connector for the tables on this list.

What the form does not ask for. There is no Catalog field of the kind Athena asks for, because Hive has no level above the database, only databases and tables. Qualytics works the rest out from the metastore address, which also makes it impossible for two datastores to collide while pointing at different clusters. There is no question about reaching storage by hostname either, because that depends on where your Qualytics deployment runs rather than on the datastore, so Qualytics handles it.

Row counts and column names. When Hive's own table statistics are current, Qualytics uses the row count Hive recorded instead of counting the rows itself, which makes profiling a very large table noticeably quicker; when the statistics are stale or absent, Qualytics counts the rows. Hive also records column names in lower case while the data files keep the case they were written with, so Qualytics confirms the two agree before a scan reads any data and refuses the scan if they differ, rather than returning a column full of empty values.

Deep Dive

How-tos

Reference