Enrichment API

This API links a datastore to an enrichment destination, changes its settings, creates a datastore for use as a destination, and reads what has been written into one.

Tip

For complete API documentation, including request/response schemas, visit the API docs.

All endpoints use the base URL of your Qualytics deployment (e.g., https://your-instance.qualytics.io/api) with a Bearer token.

Permissions

Linking, changing the destination, and editing the settings need the Member role and the Editor team permission on the source datastore. Creating a destination needs the Manager role. Unlinking needs the Admin role, with no team check. Reading settings needs Member and Reporter on the datastore in question; reading source records needs Viewer on the destination.

Link an Enrichment Destination

Link a source datastore to a destination. Calling it while another destination is linked replaces that destination; no unlink is needed.

Endpoint: PATCH /api/datastores/{datastore_id}/enrichment/{enrichment_id}

Permission: Member user role + Editor team permission on the source datastore

Parameter	Type	Description
`datastore_id`	`integer`	The source datastore ID.
`enrichment_id`	`integer`	The ID of the enrichment datastore to use as the destination. It must have been created with `enrichment_only: true`.
`enrichment_prefix`	`string`	Optional query parameter. A new prefix to save with the link. Omit it to keep the datastore's current prefix.

Request Details

No JSON body is required. This endpoint sets the link, and the prefix when enrichment_prefix is given; the other settings (limits, strategy, automatic sync) are changed with PUT /api/datastores/{datastore_id} below. Qualytics refuses the link if the prefix is already used by another source writing to the same destination.

Example request and response

Request (no body required):

curl -X PATCH "https://your-instance.qualytics.io/api/datastores/42/enrichment/100" \
  -H "Authorization: Bearer YOUR_TOKEN"

Response (abbreviated; the full response includes all datastore fields):

{
  "id": 42,
  "name": "Healthcare Database",
  "store_type": "jdbc",
  "enrichment_only": false,
  "enrichment_prefix": "_healthcare_database",
  "enrichment_source_record_limit": 10,
  "enrichment_remediation_strategy": "none",
  "enrichment_auto_sync": true,
  "high_count_rollup_threshold": 10,
  "enrich_datastore": {
    "id": 100,
    "name": "Healthcare Enrichment",
    "store_type": "jdbc",
    "type": "postgresql",
    "enrichment_only": true
  }
}

Idempotent

Calling this endpoint again with the same enrichment_id re-confirms the link. Calling it with a different ID moves the datastore to that destination; outputs already written stay where they were.

For the UI equivalent, see Link an Enrichment Destination.

Unlink an Enrichment Destination

Remove the link from a source datastore. No request body is required.

Endpoint: DELETE /api/datastores/{datastore_id}/enrichment

Permission: Admin user role (no team permission check)

Parameter	Type	Description
`datastore_id`	`integer`	The source datastore ID.

Idempotent

Calling this endpoint when nothing is linked returns 204 No Content without error.

Example request and response

Request (no body required):

curl -X DELETE "https://your-instance.qualytics.io/api/datastores/42/enrichment" \
  -H "Authorization: Bearer YOUR_TOKEN"

Response: 204 No Content

The link is removed and the remediation strategy is reset to none.

Warning

Unlinking is refused while the source datastore has Export or Materialize operations in flows or on a schedule.

For the UI equivalent, see Unlink an Enrichment Destination.

Update Enrichment Settings

Change the enrichment settings on a source datastore. They control how, and what, that datastore writes to its destination.

Endpoint: PUT /api/datastores/{datastore_id}

Permission: Member user role + Editor team permission on the source datastore

Parameter	Type	Description
`datastore_id`	`integer`	The source datastore ID.

Include the current datastore configuration

This endpoint requires name, connection_id, enrichment_only, and enrichment_prefix, plus the connector's location fields (database and schema for the PostgreSQL examples below, or root_path for file-based datastores). Read the datastore first and preserve its current values. Also include the current source-record limit, anomaly threshold, and remediation strategy: omitting them resets them to 10, 10, and none. A body containing only the setting you want to change is not valid.

Request Body (enrichment fields):

Field	Type	Default	Range	Description
`enrichment_prefix`	`string`	Required on update	Max 60 chars	Start of Scan, Remediation, and Materialize output names. Exports use the normalized datastore name instead. Normalized to lowercase with underscores and a leading underscore. Must be unique among the sources writing to the same destination.
`enrichment_source_record_limit`	`integer`	`10`	1 to 1,000,000,000	Source records stored per anomaly as examples.
`enrichment_remediation_strategy`	`string`	`"none"`	`none`, `append`, `overwrite`	Whether a Scan also writes a snapshot of the anomalous records, and whether it accumulates.
`high_count_rollup_threshold`	`integer`	`10`	1 to 1,000	Individual anomalies a check may raise per Scan before they are rolled up into one. Values above 1,000 are silently capped.
`enrichment_auto_sync`	`boolean`	Unchanged when omitted on update		Whether Qualytics syncs and profiles the destination after this datastore writes to it. New datastores start with this enabled.

Remediation Strategy Constraint

Link a destination before setting enrichment_remediation_strategy to append or overwrite. Otherwise, the update is rejected with 409 Conflict.

The examples below use a PostgreSQL source datastore with connection ID 7, database healthcare, and schema public. Replace these values with the source datastore's current configuration.

Change remediation strategy to Append

Request:

curl -X PUT "https://your-instance.qualytics.io/api/datastores/42" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Healthcare Database",
    "connection_id": 7,
    "database": "healthcare",
    "schema": "public",
    "enrichment_only": false,
    "enrichment_prefix": "_healthcare_database",
    "enrichment_source_record_limit": 10,
    "high_count_rollup_threshold": 10,
    "enrichment_remediation_strategy": "append"
  }'

Response (abbreviated):

{
  "id": 42,
  "name": "Healthcare Database",
  "enrichment_prefix": "_healthcare_database",
  "enrichment_source_record_limit": 10,
  "enrichment_remediation_strategy": "append",
  "enrichment_auto_sync": true,
  "high_count_rollup_threshold": 10,
  "enrich_datastore": {
    "id": 100,
    "name": "Healthcare Enrichment"
  }
}

Increase source record limit and change prefix

Request:

curl -X PUT "https://your-instance.qualytics.io/api/datastores/42" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Healthcare Database",
    "connection_id": 7,
    "database": "healthcare",
    "schema": "public",
    "enrichment_only": false,
    "enrichment_prefix": "_healthcare_prod",
    "enrichment_source_record_limit": 100,
    "high_count_rollup_threshold": 10,
    "enrichment_remediation_strategy": "append"
  }'

Response (abbreviated):

{
  "id": 42,
  "name": "Healthcare Database",
  "enrichment_prefix": "_healthcare_prod",
  "enrichment_source_record_limit": 100,
  "enrichment_remediation_strategy": "append",
  "enrichment_auto_sync": true,
  "high_count_rollup_threshold": 10,
  "enrich_datastore": {
    "id": 100,
    "name": "Healthcare Enrichment"
  }
}

Turn off the automatic sync and profile

Request:

curl -X PUT "https://your-instance.qualytics.io/api/datastores/42" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Healthcare Database",
    "connection_id": 7,
    "database": "healthcare",
    "schema": "public",
    "enrichment_only": false,
    "enrichment_prefix": "_healthcare_prod",
    "enrichment_source_record_limit": 100,
    "high_count_rollup_threshold": 10,
    "enrichment_remediation_strategy": "append",
    "enrichment_auto_sync": false
  }'

Response (abbreviated):

{
  "id": 42,
  "name": "Healthcare Database",
  "enrichment_prefix": "_healthcare_prod",
  "enrichment_source_record_limit": 100,
  "enrichment_remediation_strategy": "append",
  "enrichment_auto_sync": false,
  "high_count_rollup_threshold": 10,
  "enrich_datastore": {
    "id": 100,
    "name": "Healthcare Enrichment"
  }
}

For the UI equivalent, see Link an Enrichment Destination.

Get Enrichment Info

Read a source datastore's enrichment settings and its destination.

Endpoint: GET /api/datastores/{datastore_id}

Permission: Member user role + Reporter team permission on the source datastore

Parameter	Type	Description
`datastore_id`	`integer`	The source datastore ID.

General Datastore Endpoint

This is the general datastore GET endpoint. It returns all datastore fields; the example shows only the enrichment-related ones.

Example request and response

Request:

curl -X GET "https://your-instance.qualytics.io/api/datastores/42" \
  -H "Authorization: Bearer YOUR_TOKEN"

Response (enrichment fields only):

{
  "id": 42,
  "name": "Healthcare Database",
  "enrichment_only": false,
  "enrichment_prefix": "_healthcare_prod",
  "enrichment_source_record_limit": 100,
  "enrichment_remediation_strategy": "none",
  "enrichment_auto_sync": true,
  "high_count_rollup_threshold": 10,
  "enrich_datastore": {
    "id": 100,
    "name": "Healthcare Enrichment",
    "store_type": "jdbc",
    "type": "postgresql",
    "enrichment_only": true
  }
}

If no destination is linked, enrich_datastore is null.

Create a Datastore to Use as a Destination

A destination is an ordinary datastore with enrichment_only set to true. Create it with the datastore creation endpoint on any connector that Qualytics can write to; the connection and location fields are the connector's own.

Endpoint: POST /api/datastores

Permission: Manager user role

Example request (PostgreSQL)

curl -X POST "https://your-instance.qualytics.io/api/datastores" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Healthcare Enrichment",
    "connection_id": 7,
    "database": "quality",
    "schema": "qualytics",
    "enrichment_only": true,
    "teams": ["Data Platform"]
  }'

Use a dedicated location for enrichment output. teams contains team names, and a Manager needs the Editor team permission on at least one team assigned to the new datastore. Creating the destination does not link a source to it; use the link endpoint afterward.

For connector-specific payloads, see the connector's API page from Supported Connectors. For the full datastore creation endpoint, see the Datastore API.

Browse the Written Outputs

The outputs Qualytics writes into a destination become containers once the destination has been synced. This happens automatically after supported writes when the source's enrichment_auto_sync is enabled; otherwise, run Sync yourself. List the containers with the endpoint below to see their purpose, attributed source, and any confirmed last write.

Endpoint: GET /api/containers?datastore={enrichment_id}

Permission: Member user role + Reporter team permission on the destination

Field in each container	Description
`purpose`	`scan_metadata`, `remediation`, `materialized`, `exports`, `computed`, or `other`.
`purpose_dataset`	For Scan and Export outputs, which one: `failed_checks`, `source_records`, `scan_operations`, `check_metrics`, `export_anomalies`, `export_checks`, `export_field_profiles`, or `export_check_templates`.
`written_by`	The source datastore that wrote the output, or `null` when Qualytics cannot attribute it.
`last_write`	When Qualytics last wrote to the output, or `null` when it cannot confirm having written it.
`last_write_operation_id`	The operation that produced that write.

Filter and sort by purpose with purpose=<value> and sort_purpose=asc|desc, and sort by recency with sort_last_write=asc|desc.

Deprecated listing endpoint

GET /api/datastores/{id}/listing still works but is deprecated. Use GET /api/containers?datastore={id}, which returns the same containers with their purpose and last write.

Read Source Records

Read the rows of a written output through Qualytics rather than connecting to the destination directly. path is the physical output name; filter is an optional SQL predicate.

Endpoint: GET /api/datastores/{enrichment_id}/source-records?path={table_name}

Endpoint with a filter: GET /api/datastores/{enrichment_id}/source-records?path={table_name}&filter={predicate}

Permission: Member user role + Viewer team permission on the destination

The RECORD column is always masked on this route

When path points at a _source_records output, every row comes back with RECORD set to ***MASKED***, regardless of field masking, and there is no parameter to unmask it. To read the actual values, use the anomaly endpoints GET /api/anomalies/{id}/source-record and GET /api/anomalies/{id}/source-record/download with include_masked=true (Editor team permission, audited). See Source records. The other outputs are returned as written.

Source records of one anomaly

Request:

curl -X GET "https://your-instance.qualytics.io/api/datastores/100/source-records?path=_healthcare_prod_source_records&filter=anomaly_uuid='f11d4e7c-e757-4bf1-8cd6-d156d5bc4fa5'" \
  -H "Authorization: Bearer YOUR_TOKEN"

Response:

{
  "source_records": "[{\"source_container\":\"table_name\",\"source_partition\":\"partition_name\",\"anomaly_uuid\":\"f11d4e7c-e757-4bf1-8cd6-d156d5bc4fa5\",\"context\":null,\"record\":\"***MASKED***\"}]"
}

Failed checks of one anomaly

Request:

curl -X GET "https://your-instance.qualytics.io/api/datastores/100/source-records?path=_healthcare_prod_failed_checks&filter=anomaly_uuid='1a937875-6bce-4bfe-8701-075ba66be364'" \
  -H "Authorization: Bearer YOUR_TOKEN"

Response:

{
  "source_records": "[{\"quality_check_id\":155481,\"anomaly_uuid\":\"1a937875-6bce-4bfe-8701-075ba66be364\",\"quality_check_message\":\"{\\\"SNPSHT_TIMESTAMP\\\":\\\"2023-09-03 10:26:15.0\\\"}\",\"suggested_remediation_field\":null,\"suggested_remediation_value\":null,\"suggested_remediation_score\":null,\"quality_check_rule_type\":\"greaterThanField\",\"quality_check_tags\":\"Time-Sensitive\",\"quality_check_parameters\":\"{\\\"field_name\\\":\\\"SNPSHT_DT\\\",\\\"inclusive\\\":false}\",\"quality_check_description\":\"Must have a value greater than the value of SNPSHT_DT\",\"operation_id\":28162,\"detected_time\":\"2024-03-29T15:08:07.585Z\",\"source_container\":\"ACTION_TEST_CLIENT_V3\",\"source_partition\":\"ACTION_TEST_CLIENT_V3\",\"source_datastore\":\"DB2 Dataset\"}]"
}

Scan operations of one operation

Request:

curl -X GET "https://your-instance.qualytics.io/api/datastores/100/source-records?path=_healthcare_prod_scan_operations&filter=operation_id=22871" \
  -H "Authorization: Bearer YOUR_TOKEN"

Response:

{
  "source_records": "[{\"operation_id\":22871,\"datastore_id\":850,\"container_id\":7239,\"partition_name\":\"ACTION_TEST_CLIENT_V3\",\"incremental\":true,\"records_processed\":0,\"enrichment_source_record_limit\":10,\"max_records_analyzed\":-1,\"anomaly_count\":0,\"start_time\":\"2023-12-04T20:35:54.194Z\",\"end_time\":\"2023-12-04T20:35:54.692Z\",\"result\":\"success\",\"message\":null}]"
}

The same endpoint reads remediation outputs (<prefix>_remediation_<container_id>) and Export outputs (_<datastore_name>_export_anomalies, _<datastore_name>_export_checks, _<datastore_name>_export_field_profiles). Choose filters from the columns available in the output you are querying. For the columns of every output, see Enrichment Outputs and Export Outputs.

Error Responses

Status Code	Description
`401 Unauthorized`	Missing or invalid API token.
`403 Forbidden`	User does not have the required role or team permission.
`404 Not Found`	The source datastore or the destination does not exist.
`409 Conflict`	The target is not an enrichment datastore, its connector cannot host one, the prefix is already used on that destination, remediation is enabled without a linked destination, or the unlink is blocked by Export or Materialize operations.
`422 Unprocessable Entity`	Invalid request body (e.g., a remediation strategy other than `none`, `append`, `overwrite`).

Error response examples

403 Forbidden:

{ "detail": "You must have editor permission in at least one of these teams: ['Data Platform']" }

404 Not Found:

{ "detail": "Datastore id: 999 not found" }

409 Conflict (the target is not an enrichment datastore):

{ "detail": "The datastore id: 100 does not support enrichment" }

409 Conflict (creating a destination on a connector Qualytics cannot write to):

{ "detail": "Datastore type 'athena' doesn't support enrichment" }

409 Conflict (another source already writes to that destination with the same prefix):

{ "detail": "Datastore 'Retail Warehouse' already writes enrichment data into datastore id: 100 with the prefix '_healthcare_prod'" }

409 Conflict (unlinking with Export or Materialize operations in flows or schedules):

{ "detail": "The enrichment datastore cannot be unlinked because it's used by export or materialize operations within flows or scheduled datastore operations" }

Permission Summary

Operation	Minimum Permission
Link or change the destination	Member user role + Editor team permission on the source
Unlink	Admin user role
Update enrichment settings	Member user role + Editor team permission on the source
Get enrichment info	Member user role + Reporter team permission on the source
Create a destination	Manager user role
Browse the written outputs	Member user role + Reporter team permission on the destination
Read source records	Member user role + Viewer team permission on the destination

Prefix Normalization

enrichment_prefix is normalized to lowercase with underscores and a leading underscore. For example, Analytics Bronze becomes _analytics_bronze. Maximum length is 60 characters.