Skip to content

Enrichment Outputs

Every output Qualytics writes into an enrichment destination belongs to one purpose, and the purpose decides its name. This page lists the physical names and columns of Scan outputs, remediation snapshots, and materialized copies. Export outputs are documented on the Export Outputs page.

Purpose Physical name One row is
Scan <prefix>_check_metrics, <prefix>_failed_checks, <prefix>_source_records, <prefix>_scan_operations See each output below
Remediation <prefix>_remediation_<container_id> One anomalous record of the source container, with the anomaly it belongs to
Materialize <prefix>_mat_<container_name>, or <prefix>_materialize_<container_id> when the container name cannot be used One record of the source container as of the run
Export _<datastore_name>_export_anomalies, _<datastore_name>_export_checks, _<datastore_name>_export_field_profiles, _export_check_templates See Export Outputs

<prefix> is the enrichment prefix of the source datastore that wrote the output. Exports instead use the normalized datastore name, as explained in Export Outputs. A JDBC destination stores these outputs as database tables. A DFS destination stores them in Delta format in cloud object storage. After Sync, Qualytics represents both as containers.

Display names

In Qualytics these containers appear under a readable name built from their purpose and their source, such as Failed Checks (TPCH) or ORDERS / Remediation (TPCH). The physical name is what you query in the destination itself. See How Enrichment Works.

Scan outputs

A Scan can write four kinds of outputs. Rows are appended as results become available. Failed-check and source-record outputs are only written when there are failed checks or source examples to record, so a first Scan does not necessarily create all four outputs.

_CHECK_METRICS output

Captures and logs detailed metrics for every data quality check performed within the Qualytics Platform, providing insights into asserted and anomalous records across datasets.

Columns

Name Data Type Description
OPERATION_ID NUMBER Identifier of the Scan operation that produced the metric.
CONTAINER_ID NUMBER Identifier for the container associated with the check metric.
SOURCE_DATASTORE STRING Datastore where the source data resides.
SOURCE_CONTAINER STRING Name of the source data container.
SOURCE_PARTITION STRING Partition of the source data.
ASSERTION_RESULT STRING Result of the check assertion: one of passed, failed, or unasserted.
ASSERTION_DETAILS STRING Text description explaining any warnings, errors, or notes from the check.
QUALITY_CHECK_ID NUMBER Unique identifier for the quality check performed.
ASSERTED_RECORDS_COUNT NUMBER Count of records expected or asserted in the source.
ANOMALOUS_RECORDS_COUNT NUMBER Count of records identified as anomalous.

_FAILED_CHECKS output

Acts as an associative entity that consolidates information on failed checks, associating anomalies with their respective quality checks.

Columns

Name Data Type Description
QUALITY_CHECK_ID NUMBER Unique identifier for the quality check.
ANOMALY_UUID STRING UUID for the anomaly detected.
QUALITY_CHECK_MESSAGE STRING Message describing the quality check outcome.
SUGGESTED_REMEDIATION_FIELD STRING Field suggesting remediation.
SUGGESTED_REMEDIATION_VALUE STRING Suggested value for remediation.
SUGGESTED_REMEDIATION_SCORE FLOAT Score indicating confidence in remediation.
QUALITY_CHECK_RULE_TYPE STRING Type of rule applied for quality check.
QUALITY_CHECK_TAGS STRING Tags associated with the quality check.
QUALITY_CHECK_PARAMETERS STRING Parameters used for the quality check.
QUALITY_CHECK_DESCRIPTION STRING Description of the quality check.
QUALITY_CHECK_FIELDS STRING Names of the fields targeted by the quality check.
QUALITY_CHECK_METADATA STRING Optional JSON string containing additional check metadata.
OPERATION_ID NUMBER Identifier for the operation detecting anomaly.
DETECTED_TIME TIMESTAMP Timestamp when the anomaly was detected.
SOURCE_CONTAINER STRING Name of the source data container.
SOURCE_PARTITION STRING Partition of the source data.
SOURCE_DATASTORE STRING Datastore where the source data resides.
FINGERPRINT INTEGER Unique identifier created when Reactivate Recurring Anomalies is enabled.
ANOMALOUS_RECORDS_COUNT NUMBER The total number of records that triggered this anomaly. For row-level checks, this is 1 per anomaly row. For shape-level checks, this is the total count of anomalous records, independent of any source record sampling limit.

Info

This output is not characterized by unique ANOMALY_UUID or QUALITY_CHECK_ID values alone. Instead, the combination of ANOMALY_UUID and QUALITY_CHECK_ID serves as a composite key that uniquely identifies each record.

_SOURCE_RECORDS output

Stores source records in JSON format, primarily to enable the preview source record feature in the Qualytics App.

Columns

Name Data Type Description
SOURCE_CONTAINER STRING Name of the source data container.
SOURCE_PARTITION STRING Partition of the source data.
ANOMALY_UUID STRING UUID for the associated anomaly.
CONTEXT STRING Contextual information for the anomaly.
RECORD STRING JSON representation of the source record.

_SCAN_OPERATIONS output

Captures and stores the results of every scan operation conducted on the Qualytics Platform.

Schema Change (December 2024)

The CONTAINER_SCAN_ID column was removed from this output. Outputs created after this change do not contain this column. The container and datastore can be identified using the CONTAINER_ID and DATASTORE_ID columns respectively.

Columns

Name Data Type Description
OPERATION_ID NUMBER Unique identifier for the scan operation.
DATASTORE_ID NUMBER Identifier for the source datastore associated with the operation.
CONTAINER_ID NUMBER Identifier for the container associated with the operation.
PARTITION_NAME STRING Name of the source partition on which the scan operation is performed.
INCREMENTAL BOOLEAN Boolean flag indicating whether the scan operation is incremental.
RECORDS_PROCESSED NUMBER Total number of records processed during the scan operation.
ENRICHMENT_SOURCE_RECORD_LIMIT NUMBER Maximum number of source records written to the destination for each anomaly detected.
MAX_RECORDS_ANALYZED NUMBER Maximum number of records analyzed in the scan operation.
ANOMALY_COUNT NUMBER Total number of anomalies identified in the scan operation.
START_TIME TIMESTAMP Timestamp marking the start of the scan operation.
END_TIME TIMESTAMP Timestamp marking the end of the scan operation.
RESULT STRING Textual representation of the scan operation's status.
MESSAGE STRING Detailed message regarding the process of the scan operation.

Remediation outputs

When anomalies are detected in a container, a Scan can also write a remediation output: a snapshot of the anomalous records of that container, with two extra columns for the anomaly. Whether it does depends on the source datastore's remediation strategy, which is a datastore setting rather than a per-Scan choice:

  • None: no remediation outputs are written, regardless of anomaly detection.
  • Append: anomalous records are added after every Scan, building a history.
  • Overwrite: the output is replaced with the anomalous records of the latest Scan.

Note

Remediation outputs are named <enrichment_prefix>_remediation_<container_id>, where <enrichment_prefix> is the source datastore's prefix and <container_id> is the identifier of the source container the records came from.

Illustrative output

<enrichment_prefix>_remediation_<container_id>

This remediation output is an illustrative snapshot of the "Orders" container for reference purposes.

Name Data Type Description
_QUALYTICS_SOURCE_PARTITION STRING The partition from the source data container.
ANOMALY_UUID STRING Unique identifier of the anomaly.
ORDERKEY NUMBER Unique identifier of the order.
CUSTKEY NUMBER The customer key related to the order.
ORDERSTATUS CHAR The status of the order (e.g., 'F' for 'finished').
TOTALPRICE FLOAT The total price of the order.
ORDERDATE DATE The date when the order was placed.
ORDERPRIORITY STRING Priority of the order (e.g., 'urgent').
CLERK STRING The clerk who took the order.
SHIPPRIORITY INTEGER The priority given to the order for shipping.
COMMENT STRING Comments related to the order.

Note

In addition to capturing the original container fields, the platform includes two metadata columns designed to assist in the analysis and remediation process.

  • _QUALYTICS_SOURCE_PARTITION
  • ANOMALY_UUID

Remediation outputs vs. source record outputs

Both hold anomalous source data, in different shapes for different jobs.

Remediation outputs capture the affected records with one column per source field, plus the two metadata columns above. They exist only when the remediation strategy is Append or Overwrite. Their structure makes them suitable for corrective work and data workflows.

Source record outputs (<prefix>_source_records) store each record as JSON in a single RECORD column, next to SOURCE_CONTAINER, SOURCE_PARTITION, ANOMALY_UUID, and CONTEXT. Qualytics reads them to show source records next to an anomaly in the app. A Scan writes available examples up to the effective Maximum Source Examples per Anomaly, which can be overridden for the operation.

For querying and audit snapshots, use the remediation outputs. For reviewing individual anomalies inside Qualytics, the app uses the source record outputs.

Materialized outputs

A Materialize operation copies a container into the destination. A normal Materialize snapshot carries the source container's columns, with no metadata columns added, and holds the selected records as of the run. Masked fields remain masked unless Reveal Masked Values is enabled for the operation. Its name is <enrichment_prefix>_mat_<container_name>, or <enrichment_prefix>_materialize_<container_id> when the source container name cannot be used. See the Materialize Operation page for when each form is used.

Materialized and remediation outputs are also linked back to the source container they came from, at container and field level. See How Enrichment Works.

How the outputs join

The outputs a Scan and an Export write are designed to be joined: failed checks to source records by anomaly, check metrics to scan operations by operation, and so on. The Relationship Diagram draws those joins, colored by key, so you can trace how to get from one output to another. It opens from the How-tos on an enrichment datastore as Join Enrichment Tables. See View the Relationship Diagram.