Max Partition Size Check FAQ

Answers to common questions about what Max Partition Size counts, how partitions are defined, and how the anomaly is reported, grouped by topic.

Behavior

What exactly is a partition?

One unit of data the Scan loads: a file for a file container, or the slice the read strategy selected for a table. The check runs once per loaded partition.

Does the threshold apply per partition or per Scan?

Per partition. A Scan that loads ten partitions evaluates the threshold ten times, and each failing partition produces its own anomaly.

Why is there no field to select?

The rule counts records rather than reading values, so it never touches a column. That is also why it takes no filter clause and no coverage threshold.

Why can I not set a filter clause?

A row filter would change the very count the rule measures. The form omits the field and the API ignores a filter value.

Does the count include rows the Scan skipped?

No. The check counts what the Scan actually loaded from that partition, which on an Incremental scan is only the new slice.

Anomaly Reporting

What kind of anomaly does the check produce?

A Shape Anomaly, one per failing partition. There is no Record Anomaly, because the rule evaluates a count rather than individual rows.

Why does the anomaly show no source records?

There is no offending row to show. The anomaly reports that a partition is the wrong size, and the message carries the partition name and both counts.

Can I customize the anomaly message?

No. Custom Anomaly Description only replaces the message on Record Anomalies, and Max Partition Size emits a Shape Anomaly only.

Configuration

How should I pick the threshold?

Size it from what the container actually loads over a few weeks, with enough headroom for legitimate variation. See Best Practices.

Can I change the threshold later?

Yes. In the UI, open the check, change the property, and click Update; see Edit a Check for the full steps. Through the API, a PUT to /api/quality-checks/{id} updates the properties object. The rule type, the target container, and the associated Check Template stay immutable.

What permission do I need to create or edit a Max Partition Size check?

The Drafter team permission on the datastore covers Draft work (creating a check as Draft or editing it while it stays Draft). Anything that puts the check into evaluation, such as creating or editing an Active check, archiving, or deleting, requires the Author team permission (or above). Viewing only requires Reporter. See Permissions for the full matrix.

Should I use Max Partition Size or Volumetric?

Use Max Partition Size when you have a fixed limit a partition must respect. Use Volumetric when the question is whether the container changed size compared to how it usually behaves, which needs no threshold to maintain.