Skip to content

AWS Glue Native Troubleshooting

This page documents the known problems with an AWS Glue Native datastore and the steps to resolve them. A connection depends on three things in sequence, establishing the AWS identity, reading the Glue Data Catalog with it, and reading the S3 objects the catalog points at, and each section below covers one of them in that order. A failure at the S3 step often reads like a broken connection even though the connection itself is fine, so work through the sections top to bottom.

Establishing the AWS Identity

The Connection Test Fails When Assuming the Role

With Assumed Role authentication, the Test connection button fails with an access error from AWS STS.

Cause: The role's trust policy does not allow the AWS identity your Qualytics deployment uses, the External ID does not match the one the trust policy requires, or the Role ARN has a typo.

Resolution: Check the Role ARN, then check the trust policy against IAM Role Authentication. If the trust policy requires an External ID, enter exactly the same value on the connection; if it does not, leave the field empty.

The Assumed Role Option Is Missing

The Type selector is not shown, and the form asks only for an Access Key and Secret Key.

Cause: Assumed Role authentication is available on AWS and local deployments only.

Resolution: Use Access Key authentication on Azure and GCP deployments. See Authentication.

A Connection Stops Working After Keys Are Changed

A connection that worked before fails, and the IAM user's keys were changed in AWS in between.

Cause: The connection still holds the old Access Key ID and Secret Access Key.

Resolution: Edit the connection through Manage Connections and enter the new keys. The change reaches every datastore that shares the connection.

Reading the Glue Data Catalog

A failure here names the Glue call that was refused, such as GetDatabases or GetTables, together with the database or table it was reading.

The Connection Test Reports Access Denied from Glue

The test fails with an access error on a Glue call.

Cause: The identity is missing a Glue permission, or the policy does not cover the catalog, database, and table levels together. Glue checks every call against the resource and each level above it.

Resolution: Grant the Glue actions listed in Permissions on the catalog ARN, the database ARN, and the tables in that database.

The Database Dropdown Is Empty

The connection test passes, but the Database dropdown offers nothing, or not the database you expected.

Cause: The Region or Catalog ID points at a different catalog from the one holding your databases, or the identity is not allowed to see the databases. On a catalog governed by Lake Formation, databases the identity has no Lake Formation permission on are not returned.

Resolution: Confirm the region, and leave Catalog ID empty unless the catalog belongs to another account. Then check the Glue permissions and, where Lake Formation governs the catalog, the identity's Lake Formation permissions. See Lake Formation.

A Cross-Account Catalog Cannot Be Read

With Catalog ID set to another account, Glue calls fail with an access error even though the identity's own IAM policy allows them.

Cause: Cross-account access needs both accounts to allow the call. The account that owns the catalog has not granted your identity access in its Glue Data Catalog resource policy.

Resolution: Ask the owning account to add your identity to its catalog resource policy and to its S3 bucket policies. See Cross-account catalogs.

Reading Amazon S3

This is the step that most often looks like something else. The connection test reads only the catalog, so an identity that cannot read the S3 objects passes the test and lists every table, then fails when Qualytics reads the files.

The Connection Test Passes but Profiling or Scanning Fails

The connection test passes. The first Profile or Scan fails with an access error.

Cause: The identity can read the Glue Data Catalog but not the S3 objects behind a table. Common reasons are a missing s3:ListBucket or s3:GetObject grant, a table or partition stored in a bucket the policy does not cover, a bucket policy that denies the identity, or an SSE-KMS key the identity cannot use.

Resolution: Grant the S3 permissions in Permissions on every bucket and prefix the tables point to, including partitions registered outside the table's own path, and add kms:Decrypt on the key for encrypted buckets.

A Table Returns Fewer Rows Than Expected

A Profile or Scan succeeds, but some recent data is missing.

Cause: Qualytics reads the partitions registered in the Glue Data Catalog. Files written under a new partition directory are not read until that partition is registered.

Resolution: Register the new partitions, for example with a Glue crawler, MSCK REPAIR TABLE, or by having the writing job add them, then scan again.

Reading Tables

A declined table is skipped, and the rest of the database syncs normally. This is a refusal rather than a failure: reading those files directly would return something different from what the catalog describes, and refusing is safer than returning data that quietly disagrees.

A Table in Glue Does Not Appear as a Container

The Sync completes, but one or more tables registered in Glue do not become containers.

Cause: The table is one of the kinds AWS Glue Native declines: a view, a transactional (ACID) Hive table, a Hudi table, an Athena partition-projection table, a table backed by a storage handler or a Glue JDBC connection, a table or partition stored outside Amazon S3, or a table whose file format Qualytics does not ship. The full list is in the introduction.

Resolution: Read that table through the Athena connector. Both connectors can point at the same catalog at the same time.

An Iceberg Table Is Not Read

An Iceberg table registered in Glue does not become a container, or fails when it is read.

Cause: The Glue entry does not record where the table's current metadata file is, or that metadata file cannot be read from S3.

Resolution: Confirm that the table's Glue properties include metadata_location, which Iceberg's own Glue integration sets, and that the identity can read the table's metadata/ prefix in S3.

Limitations

  • Amazon S3 only. Tables and partitions stored anywhere else are declined. This is permanent, not a gap to be filled later.
  • Read-only. AWS Glue Native cannot be used as an enrichment datastore. Link a separate enrichment datastore as its destination for the anomalies and metadata Qualytics produces.
  • No Lake Formation filtering or credential vending. The identity reads with its own Glue and S3 permissions, and Lake Formation row, column, and cell filters are not applied.
  • No views, Hudi tables, partition-projection tables, or transactional Hive tables. Read them through the Athena connector.
  • Current version only for Iceberg and Delta. Qualytics reads the version the table currently points to; earlier versions are not read.