Skip to content

Incremental Profiling for Computed Tables

For very large Computed Tables, running a full profile on every operation can be expensive. Incremental profiling lets Qualytics scan only the rows that changed since the last profile, using either a "last-modified" timestamp field or a monotonically increasing "batch value" field on the Computed Table.

Enabling Incremental Profiling

Configure two fields on the Computed Table (typically via PUT /containers/{id}):

  • incremental_field_name: the name of a field produced by the Computed Table's SELECT.
  • incremental_identifier_type: one of "last-modified" or "batch-value".

The Two Modes

  • last-modified: the field is a timestamp. Qualytics tracks the highest timestamp seen and reads only rows with a newer value on the next run.
  • batch-value: the field is a monotonically increasing numeric value (integer or decimal), such as a batch number or version counter. Qualytics tracks the highest value and reads only rows with a higher value on the next run.

Restrictions

The incremental field must satisfy three conditions:

  • It must exist in the Computed Table. If the name does not match a real field, the update returns 404 Not Found.
  • It cannot be a computed field. Fields defined by an expression on top of other fields (computed output fields) are not allowed as the incremental identifier. The update returns 409 Conflict with the message "is a computed field and cannot be used as incremental field".
  • It cannot be a masked field. A field flagged for masking cannot be used as the incremental identifier.

Behavior When the Incremental Field Goes Missing

If you edit the Computed Table's query so the incremental field is no longer produced (for example, you drop the timestamp column from your SELECT list), Qualytics automatically clears both incremental_field_name and incremental_identifier_type on the next profile run. The container returns to full-profile behavior until you re-enable incremental profiling by pointing the setting at a different field.