Enrichment Outputs

Every output Qualytics writes into an enrichment destination belongs to one purpose, and the purpose decides its name. This page lists the physical names and columns of Scan outputs, remediation snapshots, source records audit snapshots, and materialized copies. Export outputs are documented on the Export Outputs page.

<prefix> is the enrichment prefix of the source datastore that wrote the output. Exports instead use the normalized datastore name, as explained in Export Outputs. A JDBC destination stores these outputs as database tables. A DFS destination stores them in Delta format in cloud object storage. After Sync, Qualytics represents both as containers.

Scan outputs

A Scan can write four kinds of outputs. Rows are appended as results become available. Failed-check and source-record outputs are only written when there are failed checks or source examples to record, so a first Scan does not necessarily create all four outputs.

_CHECK_METRICS output

Captures and logs detailed metrics for every data quality check performed within the Qualytics Platform, providing insights into asserted and anomalous records across datasets.

Columns

_FAILED_CHECKS output

Acts as an associative entity that consolidates information on failed checks, associating anomalies with their respective quality checks.

Columns

_SOURCE_RECORDS output

Stores source records in JSON format, primarily to enable the preview source record feature in the Qualytics App.

Columns

_SCAN_OPERATIONS output

Captures and stores the results of every scan operation conducted on the Qualytics Platform.

Columns

Remediation outputs

When anomalies are detected in a container, a Scan can also write a remediation output: a snapshot of the anomalous records of that container, with two extra columns for the anomaly. Whether it does depends on the source datastore's remediation strategy, which is a datastore setting rather than a per-Scan choice:

• None: no remediation outputs are written, regardless of anomaly detection.
• Append: anomalous records are added after every Scan, building a history.
• Overwrite: the output is replaced with the anomalous records of the latest Scan.

Illustrative output

<enrichment_prefix>_remediation_<container_id>

This remediation output is an illustrative snapshot of the "Orders" container for reference purposes.

Remediation outputs vs. source record outputs

Both hold anomalous source data, in different shapes for different jobs.

Remediation outputs capture the affected records with one column per source field, plus the two metadata columns above. They exist only when the remediation strategy is Append or Overwrite. Their structure makes them suitable for corrective work and data workflows.

Source record outputs (<prefix>_source_records) store each record as JSON in a single RECORD column, next to SOURCE_CONTAINER, SOURCE_PARTITION, ANOMALY_UUID, and CONTEXT. Qualytics reads them to show source records next to an anomaly in the app. A Scan writes available examples up to the effective Maximum Source Examples per Anomaly, which can be overridden for the operation.

For querying and audit snapshots, use the remediation outputs. For reviewing individual anomalies inside Qualytics, the app uses the source record outputs.

Source records audit outputs

When a Scan runs with Persist Source Records for Audit turned on, it writes a snapshot of the source data it read: for each container it loads, every record it loaded, as it existed when the Scan ran. Use these outputs when an audit requires evidence of the exact data a Scan evaluated, not only the records that failed.

Each output carries the source container's own columns, one row per record loaded, plus a _qualytics_source_partition column naming the partition the record was read from. What it holds follows what the Scan loaded, not the anomaly settings:

• It is not limited by Maximum Source Examples per Anomaly, Maximum Record Anomalies per Check, or any other anomaly limit.
• An Incremental Scan writes only the incremental slice it read. A check's own filter does not narrow the output: it limits what that check evaluates, not what the Scan loads, so rows that no check looked at are written too.
• The Record Limit (max_records_analyzed_per_partition in the API) still bounds what is loaded, and therefore what is written.
• A container the Scan does not load, for example one with no checks that need to read its records, gets no audit output.

Every new Scan with the option turned on creates a new output for each container it loads, without touching the outputs of other scans. Resume keeps the same operation and retries only the containers that did not finish, replacing each retried container's earlier partial output with the data read on that attempt. Copy an earlier partial snapshot before resuming if you need it. These outputs accumulate and can be large, since each one is a copy of the data read. Qualytics does not delete them: manage their retention directly in the enrichment datastore.

The snapshot is written while the container is scanned, one partition at a time. If writing one partition fails, that container's scan fails, the rest of its snapshot is not written, and the output is not reported as written, so an incomplete snapshot never passes for a complete one. Run the Scan again once the destination accepts writes.

In Qualytics, these outputs carry the Source Records Audit purpose.

Materialized outputs

A Materialize operation copies a container into the destination. A normal Materialize snapshot carries the source container's columns, with no metadata columns added, and holds the selected records as of the run. Masked fields remain masked unless Reveal Masked Values is enabled for the operation. Its name is <enrichment_prefix>_mat_<container_name>, or <enrichment_prefix>_materialize_<container_id> when the source container name cannot be used. See the Materialize Operation page for when each form is used.

Materialized and remediation outputs are also linked back to the source container they came from, at container and field level. See How Enrichment Works.

How the outputs join

The outputs a Scan and an Export write are designed to be joined: failed checks to source records by anomaly, check metrics to scan operations by operation, and so on. The Relationship Diagram draws those joins, colored by key, so you can trace how to get from one output to another. It opens from the How-tos on an enrichment datastore as Join Enrichment Tables. See View the Relationship Diagram.

Joining scan outputs

_check_metrics records every check evaluation, passes included, but it carries no timestamp of its own. Join it to _scan_operations on the operation, the container, and the partition to get the scan's start and end times. The result is one row per check, per partition, per scan: an audit dataset of every execution.

SELECT m.QUALITY_CHECK_ID,
       m.ASSERTION_RESULT,
       m.ASSERTED_RECORDS_COUNT,
       m.ANOMALOUS_RECORDS_COUNT,
       m.ASSERTION_DETAILS,
       s.START_TIME,
       s.END_TIME,
       s.RESULT,
       s.INCREMENTAL,
       m.OPERATION_ID,
       m.SOURCE_CONTAINER,
       m.SOURCE_PARTITION
FROM   <prefix>_check_metrics m
JOIN   <prefix>_scan_operations s
       ON  s.OPERATION_ID   = m.OPERATION_ID
       AND s.CONTAINER_ID   = m.CONTAINER_ID
       AND s.PARTITION_NAME = m.SOURCE_PARTITION;

Anomalies join the same way: _failed_checks carries OPERATION_ID and DETECTED_TIME, so it links to _scan_operations through the operation and to _source_records through ANOMALY_UUID. START_TIME, END_TIME, and DETECTED_TIME are written as UTC instants; a timezone-naive column in the destination shows the UTC wall-clock time.

To confirm which definition a check had when a scan ran, pass the scan's START_TIME to GET /api/quality-checks/{id}/version.