How Enrichment Works

This page is the reference for the mechanics of enrichment: the settings a source datastore carries, what each operation writes into the destination, what Qualytics does on its own after a write, and what happens when you change or remove the link. For the concepts, start with the Introduction; for what the outputs look like once written, see Enrichment Purposes.

Enrichment settings

The settings live on the datastore whose results are being written. Open Settings > Enrichment to choose its destination, prefix, remediation strategy, and automatic Sync/Profile behavior. The two anomaly limits are under Settings > Defaults > Scan and can also be overridden for a Scan operation. Changes apply to subsequent operations.

Setting	Default	Range	What it controls
Prefix	Generated from the datastore name	Up to 60 characters	The start of Scan, Remediation, and Materialize output names. See Prefix.
Maximum Source Examples per Anomaly	Inherited from Platform Defaults; shipped value 10	1 to 1,000,000,000	How many source records are stored for each anomaly. See Source examples.
Maximum Record Anomalies per Check	Inherited from Platform Defaults; shipped value 10	1 to 1,000	How many individual anomalies a check may raise per Scan before they are rolled up into one.
Remediation Strategy	None	None, Append, Overwrite	Whether a Scan also writes a snapshot of the anomalous records. See Remediation strategies.
Sync and profile the enrichment destination	On	On, Off	Whether Qualytics syncs and profiles the destination on its own after this datastore writes to it. See Automatic sync and profile.

Prefix

The prefix is the first part of the Scan, Remediation, and Materialize output names Qualytics writes for this source datastore, so that several sources can share one destination without those outputs colliding. Qualytics normalizes it to lowercase with underscores and a leading underscore: Analytics Bronze becomes _analytics_bronze. Export output names instead use the source datastore's name, independently of this prefix.

Two sources writing to the same destination cannot use the same prefix. Qualytics checks this when you link and when you change the prefix, and refuses the change instead of letting the outputs mix.

Changing the prefix does not rename anything already written. New outputs are created under the new prefix and the old ones stay where they are. Remove the old outputs directly from the destination if you no longer need them.

The prefix is read-only in the link dialog while the current destination remains selected. It can be changed when selecting a different destination or through the API.

Source examples

Maximum Source Examples per Anomaly controls how many rows of the source data Qualytics writes into <prefix>_source_records for each anomaly, so you can inspect the records that failed a check. More rows mean more storage on every Scan.

Value	Use case	Storage impact
`10` (default)	Quick investigation, enough to see the pattern.	Minimal
`100`	Root cause analysis, enough to see how the issue is distributed.	Low
`1,000`	Detailed audit, a comprehensive sample for compliance reviews.	Moderate
`1,000,000,000`	The largest supported limit; stores up to this many examples per anomaly.	High, and it grows with every Scan

Storage impact

Higher values mean more data written to your destination on every Scan. For large containers with many anomalies, a high value produces substantial storage growth. Start with the default and raise it as needed.

Remediation strategies

The remediation strategy decides whether a Scan also writes a snapshot of the anomalous records of each container, in a separate output with one column per source field.

Strategy	Behavior
None	No remediation outputs are written. Source examples are still written to `<prefix>_source_records`. This is the default.
Append	Anomalous records are added to the container's remediation output after every Scan, building a history of everything that ever failed.
Overwrite	The container's remediation output is replaced with the anomalous records of the latest Scan. Only the most recent failures are kept.

Warning

Link a destination before enabling Append or Overwrite. Qualytics rejects those settings when no destination is linked and resets the strategy to None when you unlink or delete the destination.

What gets written

Every container in a destination has a purpose, shown as a badge. Qualytics uses confirmed writes and recognized output names to identify written data; computed assets and other contents have their own purposes. A recognized purpose does not by itself establish a last-write time.

The six purposes, who writes each, and what lands in the destination are listed on Enrichment Purposes.

Profile and Sync operations on the source write nothing to the destination. For the physical names and columns, see Enrichment Outputs and Export Outputs.

Automatic sync and profile

An output Qualytics writes is not usable in Qualytics until two things have happened: a Sync of the destination, so the output becomes a container, and a Profile of that container, so its fields have types and the data is ready for browsing, computed assets, and other operations.

When automatic Sync/Profile is enabled and a Scan, Materialize, or Export reports a write into a destination, Qualytics runs a Sync of the destination and then a Profile of the written containers that are new or have changed since their last Profile. This automatic Profile analyzes up to 1,000 records per partition, with AI check generation and correlation testing turned off. Both operations appear in the destination's activity.

This behavior is controlled per source datastore by the Sync and profile the enrichment destination setting, on by default. Turn it off to manage discovery yourself. Run Sync on the destination to discover the written outputs, then Profile the containers you want to use; a manually started Sync does not automatically start this Profile.

Two details worth knowing:

Several writes can share a Sync already in progress. If another write arrives after that Sync started, Qualytics schedules a follow-up Sync to discover any outputs the first one missed.
Export Check Templates writes for the whole organization rather than for a source datastore, so it does not trigger this sequence. Run Sync and then Profile on the selected enrichment datastore after exporting templates.

Output names and display names

In the destination itself, outputs carry their physical names: <prefix>_failed_checks, <prefix>_remediation_<container_id>, <prefix>_mat_<container_name>, and so on.

In Qualytics, each written container also gets a display name built from its purpose and its source, so you do not have to decode prefixes:

Failed Checks (TPCH) for an output a Scan or an Export writes, naming the output and the source datastore.
ORDERS / Remediation (TPCH) for an output written for one source container, naming the container, the purpose, and the source datastore.

Qualytics refreshes these names after profiling the destination. If you rename the source datastore or source container, a subsequent Profile of the destination updates the generated names. If someone changes a written container's display name by hand, Qualytics preserves that choice. Clearing the name has the same effect.

Lineage

Materialized and remediation outputs are copies of a source container, so Qualytics links each one back to the container it came from, at container level and at field level, with no setup on your part. The links show on the Lineage tab of both containers. See Lineage for how lineage is displayed.

Last write

Every written container shows when it was last written and by which operation, in the Origin section of its overview: the source datastore that wrote it, the operation type, and the time of the last write with a link to that operation. The time comes from the write itself, not from the source's most recent operation, so a container shows no last write at all rather than a borrowed one when Qualytics cannot confirm having written it (for example, an output written before this tracking existed).

The destination's Overview aggregates the same information per source in its Linked Datastores panel: every source datastore that writes here, with its connector, its last write, and a breakdown by purpose. Clicking a source opens a panel with that source's last write overall and by purpose, its enrichment prefix and settings, and a button to open the source datastore itself.

Several source datastores can write to the same destination. Each keeps its own settings. Unique enrichment prefixes keep Scan, Remediation, and Materialize outputs apart; distinct normalized datastore names keep Export outputs apart. This is the usual arrangement when you want every quality result in one database or bucket, or want to query anomalies across sources with a single SQL statement.

Changing the destination

To move a source datastore to a different destination, open the Enrichment Destination dialog, pick another datastore under Use existing, and save. You do not need to unlink first. Outputs already written stay in the old destination; new writes go to the new one, under the same prefix. The API works the same way: linking a different destination replaces the current one.

Unlinking

Unlinking removes the connection between a source datastore and its destination. It needs the Admin role, and it is refused while the source has Export or Materialize operations in a flow or on a schedule.

When you unlink:

The remediation strategy is reset to None.
Future Scans continue without writing enrichment data. Materialize and datastore Export require a destination and cannot run until another is linked. Automatic Sync/Profile no longer runs on behalf of this source.
Everything already written stays in the destination. Qualytics deletes nothing; remove the outputs directly from the destination if you want them gone.

Unlinking is reversible. Relink the same destination with the same prefix and the next Scan continues writing to the same outputs. For the steps, see Unlink an Enrichment Destination.

Deleting a destination

Deleting an enrichment datastore uses the ordinary datastore delete dialog and needs the Admin role. It removes the datastore and its tracked records from Qualytics, removes incoming links, and resets the linked sources' remediation strategies to None. Physical data already written remains in your storage. Deletion is blocked while the datastore is used by a flow, or while linked sources have Export or Materialize operations on a schedule or in a flow.