Introduction to the Hive Native Connector

Apache Hive organizes data held in HDFS or cloud object storage into databases and tables, and records in its metastore where every table and partition lives. Most Hive environments expose that data through HiveServer2, which compiles each query, runs it on the cluster, and returns the rows.

The Qualytics Hive Native connector takes a shorter path. It reads table information from the metastore and then reads the table's data files directly, leaving the cluster's query engine out of the read path, which is considerably faster on large tables. Partitions come from the metastore rather than from listing storage, so Qualytics profiles tables, runs scheduled scans, and surfaces record- and schema-level anomalies on results that match what Hive itself reports.

Notes

Choosing between the two Hive connectors. Qualytics keeps both, and they are good at different things. Use Hive Native for the file-based tables that make up most large Hive environments, whether stored as Parquet, ORC, delimited text, JSON, or Avro, and the Hive connector for the tables the native path declines to read. Both can be connected against the same cluster at the same time.

Supported tables and formats. The connector reads managed and external tables, partitioned or not, in these formats:

Text and JSON tables are read with the SerDe settings the metastore records for them, including Hive's defaults where a setting is absent. A LazySimpleSerDe table with no field delimiter set is split on Ctrl-A, as Hive does, not on commas, and \N is treated as the null marker. When a text table uses a setting that cannot be carried over exactly, such as an escape character, a line delimiter other than a newline, more than one header line, footer lines, or complex columns, Qualytics reads it through Hive's own SerDe instead, so the results still agree with what Hive returns.

Anything else is declined rather than read incorrectly, and each declined table is reported on its own, with the reason, while the rest of the database syncs normally. Use the Hive connector for the tables on this list.

What the form does not ask for. There is no Catalog field of the kind Athena asks for, because Hive has no level above the database, only databases and tables. Qualytics works the rest out from the metastore address, which also makes it impossible for two datastores to collide while pointing at different clusters. There is no question about reaching storage by hostname either, because that depends on where your Qualytics deployment runs rather than on the datastore, so Qualytics handles it.

Row counts and column names. When Hive's own table statistics are current, Qualytics uses the row count Hive recorded instead of counting the rows itself, which makes profiling a very large table noticeably quicker; when the statistics are stale or absent, Qualytics counts the rows. Hive also records column names in lower case while the data files keep the case they were written with, so Qualytics confirms the two agree before a scan reads any data and refuses the scan if they differ, rather than returning a column full of empty values.

Deep Dive

How-tos

Reference