How Enrichment Works¶

This page is the reference for the mechanics of enrichment: the settings a source datastore carries, what each operation writes into the destination, what Qualytics does on its own after a write, and what happens when you change or remove the link. For the concepts, start with the Introduction; for what the outputs look like once written, see Enrichment Purposes.

Enrichment settings¶

The settings live on the datastore whose results are being written. Open Settings > Enrichment to choose its destination, prefix, remediation strategy, and whether written outputs are profiled automatically. The two anomaly limits are under Settings > Defaults > Scan and can also be overridden for a Scan operation. Changes apply to subsequent operations.

Prefix¶

The prefix is the first part of the Scan, Remediation, Source Records Audit, and Materialize output names Qualytics writes for this source datastore, so that several sources can share one destination without those outputs colliding. Qualytics normalizes it to lowercase with underscores and a leading underscore: Analytics Bronze becomes _analytics_bronze. Export output names instead use the source datastore's name, independently of this prefix.

Two sources writing to the same destination cannot use the same prefix. Qualytics checks this when you link and when you change the prefix, and refuses the change instead of letting the outputs mix.

Changing the prefix does not rename anything already written. New outputs are created under the new prefix and the old ones stay where they are. Remove the old outputs directly from the destination if you no longer need them.

The prefix is read-only in the link dialog while the current destination remains selected. It can be changed when selecting a different destination or through the API.

Source examples¶

Maximum Source Examples per Anomaly controls how many rows of the source data Qualytics writes into <prefix>_source_records for each anomaly, so you can inspect the records that failed a check. More rows mean more storage on every Scan.

Remediation strategies¶

The remediation strategy decides whether a Scan also writes a snapshot of the anomalous records of each container, in a separate output with one column per source field. Each anomaly contributes the same records it keeps as source examples, so it writes at most Maximum Source Examples per Anomaly rows (10 by default) even when it holds more anomalous records. Raise that limit to capture more rows per anomaly.

Source records audit¶

Separately from remediation, a Scan can write a full snapshot of the records it loaded, one new output per container per Scan. This is a per-Scan choice, not a datastore setting: turn on Persist Source Records for Audit when running or scheduling the Scan, or in a Flow scan action. It requires a linked destination. See Persist Source Records for Audit and Source records audit outputs.

What gets written¶

Every container in a destination has a purpose, shown as a badge. Qualytics uses confirmed writes and recognized output names to identify written data; computed assets and other contents have their own purposes. A recognized purpose does not by itself establish a last-write time.

The seven purposes, who writes each, and what lands in the destination are listed on Enrichment Purposes.

Profile and Sync operations on the source write nothing to the destination. For the physical names and columns, see Enrichment Outputs and Export Outputs.

What happens after each write¶

An output Qualytics writes is usable in Qualytics once two things have happened: the output has been added to the destination as a container, and that container has been profiled, so its fields have types, row counts are recorded, and the data is ready for browsing, computed assets, and other operations.

After every successful write from a Scan (including External Scan), Materialize, or Export, Qualytics does the following:

1. The output is added to the destination. This always happens, whatever the setting. Each write reports the table or file it wrote, described the same way a Sync of the destination would describe it: name, columns, native types, and partition field. Qualytics adds or updates that container and its fields directly from this description, so the output appears in the destination's container list as soon as the write is reported. No Sync of the destination runs. The container also gets its display name and its lineage back to the source.
2. The written containers are profiled, when the setting is on. Qualytics then runs one Profile of the written containers that are new or have changed since their last Profile. This Profile analyzes up to 1,000 records per partition, with AI check generation and correlation testing turned off. It is this Profile that gives the fields their types and records row counts, so field types appear once it finishes. It appears in the destination's activity. Other containers in the destination are not included, even when they have never been profiled: only what this write touched is profiled.

The Profile is controlled per source datastore by the Profile written tables in the enrichment destination setting, on by default. When it is off, that source's outputs are still added, named, and linked to their source, but no Profile runs on its own. Their fields stay untyped (shown as Unknown) and they have no row counts or statistics until you run a Profile on the destination. Checks that depend on field types do not apply to those fields until then.

Several source datastores can write into the same destination. Each source's own setting decides whether the outputs it wrote are profiled. A Sync you start by hand, from a schedule, or from a Flow is not followed by the automatic Profile.

When Qualytics syncs the destination instead¶

Qualytics falls back to a Sync of the destination when it cannot add a write's output from its description. When the source's setting is on, the automatic Profile follows that Sync and covers only the outputs written by the writes that Sync was started for. This happens when:

• The description is missing or cannot be used, for example a write reported by an older version of the platform, or an existing container whose description did not arrive and whose schema may have changed.
• A write ended without any successful output.

Several writes in quick succession are handled together. When a fallback Sync is already in progress, later writes share it; if a write arrives after that Sync started, Qualytics schedules a follow-up Sync to discover any outputs the first one missed.

Partition fields of written outputs¶

No written output is given an incremental identifier automatically.

Global exports¶

Export Check Templates and the Access Report export from a Global Export action write for the whole organization rather than for a source datastore, so no source's setting governs them. Their output is always added to the destination and profiled automatically, the same way as above, including the fallback to a Sync.

Output names and display names¶

In the destination itself, outputs carry their physical names: <prefix>_failed_checks, <prefix>_remediation_<container_id>, <prefix>_mat_<container_name>, and so on.

In Qualytics, each written container also gets a display name built from its purpose and its source, so you do not have to decode prefixes:

• Failed Checks (TPCH) for an output a Scan or an Export writes, naming the output and the source datastore.
• ORDERS / Remediation (TPCH) for an output written for one source container, naming the container, the purpose, and the source datastore.

Qualytics refreshes these names after each write, and after each Profile or Sync of the destination. If you rename the source datastore or source container, the generated names update the next time one of these runs. If someone changes a written container's display name by hand, Qualytics preserves that choice. Clearing the name has the same effect.

Lineage¶

Materialized and remediation outputs are copies of a source container, so Qualytics links each one back to the container it came from, at container level and at field level, with no setup on your part. The links show on the Lineage tab of both containers. See Lineage for how lineage is displayed.

Last write¶

Every written container shows when it was last written and by which operation, in the Origin section of its overview: the source datastore that wrote it, the operation type, and the time of the last write. When an Export or Materialize operation wrote the container, the last write also links to that operation. Scan and Remediation outputs show the time only, because several operations write to them in batches. The time comes from the write itself, not from the source's most recent operation, so a container shows no last write at all rather than a borrowed one when Qualytics cannot confirm having written it (for example, an output written before this tracking existed).

The destination's Overview aggregates the same information per source in its Linked Datastores panel: every source datastore that writes here, with its connector, its last write, and a breakdown by purpose. Clicking a source opens a panel with that source's last write overall and by purpose, its enrichment prefix and settings, and a button to open the source datastore itself.

Sharing one destination¶

Several source datastores can write to the same destination. Each keeps its own settings. Unique enrichment prefixes keep Scan, Remediation, Source Records Audit, and Materialize outputs apart; distinct normalized datastore names keep Export outputs apart. This is the usual arrangement when you want every quality result in one database or bucket, or want to query anomalies across sources with a single SQL statement.

Changing the destination¶

To move a source datastore to a different destination, open the Enrichment Destination dialog, pick another datastore under Use existing, and save. You do not need to unlink first. Outputs already written stay in the old destination; new writes go to the new one, under the same prefix. The API works the same way: linking a different destination replaces the current one.

Unlinking¶

Unlinking removes the connection between a source datastore and its destination. It needs the Admin role, and it is refused while the source has Export or Materialize operations in a flow or on a schedule.

When you unlink:

• The remediation strategy is reset to None.
• Future Scans continue without writing enrichment data. Materialize and datastore Export require a destination and cannot run until another is linked. Written outputs are no longer added and profiled automatically on behalf of this source.
• Everything already written stays in the destination. Qualytics deletes nothing; remove the outputs directly from the destination if you want them gone.

Unlinking is reversible. Relink the same destination with the same prefix and the next Scan continues writing to the same outputs. For the steps, see Unlink an Enrichment Destination.

Deleting a destination¶

Deleting an enrichment datastore uses the ordinary datastore delete dialog and needs the Admin role. It removes the datastore and its tracked records from Qualytics, removes incoming links, and resets the linked sources' remediation strategies to None. Physical data already written remains in your storage. Deletion is blocked while a flow action runs an operation on the datastore or a Global Export action writes to it, or while linked sources have Export or Materialize operations on a schedule or in a flow.

See Also¶