Incremental Scanning for Computed Tables¶

For very large Computed Tables, reading every row on every scan can be expensive. An incremental setting lets Qualytics scan only the rows that changed since the last scan, using either a "last-modified" timestamp field or a monotonically increasing "batch value" field on the Computed Table.

Enabling Incremental Scanning¶

Configure two fields on the Computed Table (typically via PUT /containers/{id}):

• incremental_field_name: the name of a field produced by the Computed Table's SELECT.
• incremental_identifier_type: one of "last-modified" or "batch-value".

The Two Modes¶

• last-modified: the field is a timestamp. Each scan records the highest timestamp seen, and the next scan reads only rows with a newer value.
• batch-value: the field is a monotonically increasing numeric value (integer or decimal), such as a batch number or version counter. Each scan records the highest value seen, and the next scan reads only rows with a higher value.

Restrictions¶

The incremental field must satisfy three conditions:

• It must exist in the Computed Table. If the name does not match a real field, the update returns 404 Not Found.
• It cannot be a computed field. Fields defined by an expression on top of other fields (computed output fields) are not allowed as the incremental identifier. The update returns 409 Conflict with the message "is a computed field and cannot be used as incremental field".
• It cannot be a masked field. A field flagged for masking cannot be used as the incremental identifier.

Behavior When the Incremental Field Goes Missing¶

If you edit the Computed Table's query so the incremental field is no longer produced (for example, you drop the timestamp column from your SELECT list), Qualytics automatically clears both incremental_field_name and incremental_identifier_type on the next profile run. Scans then read the full Computed Table again until you point the setting at a different field.