Source Records¶

Reviewing anomaly source records

Source records are the rows behind an anomaly: the actual data that failed its checks, shown as a table on the anomaly with the failing cells highlighted. They are the evidence you read to decide whether an anomaly is a real problem, a false alarm, or an exception the business accepts.

Where Source Records Come From¶

A Scan writes the source records of each anomaly to the source datastore's enrichment datastore, and the anomaly reads them back from there. A datastore without an enrichment datastore produces anomalies with no source records.

How many rows are kept depends on the anomaly type:

The limit is set on the Scan before it runs. Raising it later does not add rows to anomalies that already exist: a new Scan captures more rows on the new anomalies it creates. See Maximum Source Examples per Anomaly.

Reading the Table¶

Each row of the table is one source record, and each column is a field of the table or file. The header stays visible while you scroll.

• Failing cells are highlighted. The cells of the fields that failed a check carry an orange outline and a tinted background, so the problem stands out from the rest of the row. Their column headers carry a badge.
• Masked fields are marked. A field that is masked shows its values as ***MASKED*** and a masked icon in its header. See Masked Fields.

Two controls above the table change what you see:

The defaults depend on the container:

Anomalous First is turned off while Only Anomalous is on, because every visible field is then anomalous.

The sort you pick becomes your default the next time you open source records, kept separately for computed containers and for the others, so a choice made on a plain table does not replace the query order of a computed container. The preference is saved in your browser. You can also drag a column header to move that column.

How Fresh the Records Are¶

Source records are kept in your browser for up to 8 hours after they are first loaded, so reopening an anomaly is fast. The tooltip of the Refresh button shows when they were last loaded, and the button loads them again from the enrichment datastore. See Refresh Source Records.

Downloading the Records¶

The Download button exports the source records as a .csv file named source_records_<anomaly id>.csv, of up to 250 MB. The file holds the rows the Scan kept, not every failing row in the data. Its columns follow the order on screen, including the Sort By choice and any column you dragged. Columns hidden with Fields are still included, after the visible ones. On a very wide table, where the full column order is too long to send with the download, the columns past that point keep their original order. See Download Source Records.

When the Enrichment Datastore Is No Longer Linked¶

If the enrichment datastore is unlinked or deleted, the Source Records section shows a message saying the datastore no longer has an enrichment datastore linked, with a retry control. The Source Records tab of a Scan's results shows the same message, and rows loaded earlier are cleared from the screen.

The records are not lost. They stay in the enrichment datastore, and linking the same enrichment datastore again, with the same prefix, brings them back. See Link an Enrichment Destination.

Masked Fields¶

A masked field shows ***MASKED*** in place of its values in the source records. Masking protects sensitive data from people who can see the anomaly but should not see the values.

• Revealing is per anomaly. A user with the Editor team permission can reveal the masked values of an anomaly. Revealing shows the masked values of every source record of that anomaly at once, not one row at a time. See Reveal Masked Values.
• Every reveal is audited. Each reveal is recorded in the Masking Audit Log with the user, the time, the IP address, and the fields revealed. Users with the Admin role can read it. See View the Masking Audit Log.
• Messages stay masked. Check failure messages and anomaly descriptions show <masked> in place of a masked value, permanently. The original value is not stored on the anomaly, so revealing the source records is the only way to see it.

Comparison Source Records¶

An anomaly from a Data Diff check configured with row identifiers lists the mismatched rows side by side rather than as plain records. It is the same table the Data Reconciliation recipe uses when you resolve the mismatches, without the controls for choosing a side.

Two columns are pinned to the left edge. Found in shows a marker for each side, filled where the row exists and outlined where it does not, and Status gives its change type:

The change type is read from the checked asset's point of view, so a row the reference is missing counts as added rather than removed.

Row identifier columns are marked with a key. On a changed row, only the fields the two sides disagree on are split into a left and a right half, with the differing parts emphasized. Click one to open the full comparison. An added or removed row exists on one side only, so each of its cells shows that single value instead of a split.

An added or removed row is highlighted across the whole row, while a changed row highlights only the cells in dispute.

For the same table in the context of correcting the mismatches, see Scan and Reconcile.

Permissions¶

See Anomaly Permissions.

See Also¶