Enrichment Tables

When anomalies are detected, the platform writes metadata into four primary enrichment tables:

• <enrichment_prefix>_check_metrics
• <enrichment_prefix>_failed_checks
• <enrichment_prefix>_source_records
• <enrichment_prefix>_scan_operations

_CHECK_METRICS_Table

Captures and logs detailed metrics for every data quality check performed within the Qualytics Platform, providing insights into asserted and anomalous records across datasets.

Columns

Name	Data Type	Description
OPERATION_ID	NUMBER	Unique Identifier for the check metric.
CONTAINER_ID	NUMBER	Identifier for the container associated with the check metric.
SOURCE_DATASTORE	STRING	Datastore where the source data resides.
SOURCE_CONTAINER	STRING	Name of the source data container.
SOURCE_PARTITION	STRING	Partition of the source data.
ASSERTION_RESULT	STRING	Result of the check assertion: one of passed, failed, or unasserted.
ASSERTION_DETAILS	STRING	Text description explaining any warnings, errors, or notes from the check.
QUALITY_CHECK_ID	NUMBER	Unique identifier for the quality check performed.
ASSERTED_RECORDS_COUNT	NUMBER	Count of records expected or asserted in the source.
ANOMALOUS_RECORDS_COUNT	NUMBER	Count of records identified as anomalous.
_QUALYTICS_SOURCE_PARTITION	STRING	Partition information specific to Qualytics metrics.

_FAILED_CHECKS Table

Acts as an associative entity that consolidates information on failed checks, associating anomalies with their respective quality checks.

Columns

Name	Data Type	Description
QUALITY_CHECK_ID	NUMBER	Unique identifier for the quality check.
ANOMALY_UUID	STRING	UUID for the anomaly detected.
QUALITY_CHECK_MESSAGE	STRING	Message describing the quality check outcome.
SUGGESTED_REMEDIATION_FIELD	STRING	Field suggesting remediation.
SUGGESTED_REMEDIATION_VALUE	STRING	Suggested value for remediation.
SUGGESTED_REMEDIATION_SCORE	FLOAT	Score indicating confidence in remediation.
QUALITY_CHECK_RULE_TYPE	STRING	Type of rule applied for quality check.
QUALITY_CHECK_TAGS	STRING	Tags associated with the quality check.
QUALITY_CHECK_PARAMETERS	STRING	Parameters used for the quality check.
QUALITY_CHECK_DESCRIPTION	STRING	Description of the quality check.
OPERATION_ID	NUMBER	Identifier for the operation detecting anomaly.
DETECTED_TIME	TIMESTAMP	Timestamp when the anomaly was detected.
SOURCE_CONTAINER	STRING	Name of the source data container.
SOURCE_PARTITION	STRING	Partition of the source data.
SOURCE_DATASTORE	STRING	Datastore where the source data resides.
FINGERPRINT	INTEGER	Unique identifier created when Reactivate Recurring Anomalies is enabled.

Info

This table is not characterized by unique ANOMALY_UUID or QUALITY_CHECK_ID values alone. Instead, the combination of ANOMALY_UUID and QUALITY_CHECK_ID serves as a composite key, uniquely identifying each record in the table.

_SOURCE_RECORDS Table

Stores source records in JSON format, primarily to enable the preview source record feature in the Qualytics App.

Columns

Name	Data Type	Description
SOURCE_CONTAINER	STRING	Name of the source data container.
SOURCE_PARTITION	STRING	Partition of the source data.
ANOMALY_UUID	STRING	UUID for the associated anomaly.
CONTEXT	STRING	Contextual information for the anomaly.
RECORD	STRING	JSON representation of the source record.

_SCAN_OPERATIONS Table

Captures and stores the results of every scan operation conducted on the Qualytics Platform.

Columns

Name	Data Type	Description
OPERATION_ID	NUMBER	Unique identifier for the scan operation.
DATASTORE_ID	NUMBER	Identifier for the source datastore associated with the operation.
CONTAINER_ID	NUMBER	Identifier for the container associated with the operation.
CONTAINER_SCAN_ID	NUMBER	Identifier for the container scan associated with the operation.
PARTITION_NAME	STRING	Name of the source partition on which the scan operation is performed.
INCREMENTAL	BOOLEAN	Boolean flag indicating whether the scan operation is incremental.
RECORDS_PROCESSED	NUMBER	Total number of records processed during the scan operation.
ENRICHMENT_SOURCE_RECORD_LIMIT	NUMBER	Maximum number of records written to the enrichment for each anomaly detected.
MAX_RECORDS_ANALYZED	NUMBER	Maximum number of records analyzed in the scan operation.
ANOMALY_COUNT	NUMBER	Total number of anomalies identified in the scan operation.
START_TIME	TIMESTAMP	Timestamp marking the start of the scan operation.
END_TIME	TIMESTAMP	Timestamp marking the end of the scan operation.
RESULT	STRING	Textual representation of the scan operation's status.
MESSAGE	STRING	Detailed message regarding the process of the scan operation.