Min Partition Size Check Examples

The situation: Order lines land in one daily partition of roughly 300,000 records. A partition far below that means the upstream export was cut short, and every downstream total for that day will be understated. A floor of 100,000 catches a truncated load without firing on a genuinely quiet day.

Check configuration

Payload

{
    "description": "Every daily partition must load at least 100,000 records.",
    "rule": "minPartitionSize",
    "fields": null,
    "container_id": 145,
    "properties": {
        "value": 100000
    },
    "tags": ["load-monitoring", "ingestion"],
    "additional_metadata": {"jira": "DATA-10305"},
    "template_id": null,
    "status": "Active",
    "owner_id": 7,
    "default_anomaly_assignee_id": 12
}

Sample Data (the partitions this Scan loaded)

What gets flagged

The 11 March partition holds 42,760 records against a normal 300,000: the export was interrupted partway through. The two earlier partitions pass. Only the failing partition produces an anomaly, and it names the partition and both counts.

Flowchart

graph TD
    A["Scan loads a partition"] --> B["Count the records loaded"]
    B --> C{"Is the count at or above 100,000?"}
    C -->|Yes| D["Partition passes"]
    C -->|No| E["Shape Anomaly naming<br/>the partition and both counts"]
    D --> F["Next partition"]
    E --> F

Equivalent SQL

-- The partitions the check would flag.
SELECT dt AS partition_name,
       count(*) AS records
FROM order_lines
GROUP BY dt
HAVING count(*) < 100000;

The situation: A partner drops one CSV per store into a folder every night. Every store trades daily, so a file with almost nothing in it means the export failed for that store rather than that the store was closed. A floor of 500 rows separates a failed export from a slow night.

Check configuration

Payload

{
    "description": "Every store file must contain at least 500 transactions.",
    "rule": "minPartitionSize",
    "fields": null,
    "container_id": 512,
    "properties": {
        "value": 500
    },
    "tags": ["partner-feed", "load-monitoring"],
    "additional_metadata": {"jira": "DATA-10318"},
    "template_id": null,
    "status": "Active",
    "owner_id": 7,
    "default_anomaly_assignee_id": 47
}

Sample Data (each file in the folder is one partition)

What gets flagged

store_0144.csv arrived with 12 rows, so the partner's export for that store failed after writing the header and a handful of records. Because each file is its own partition, the two healthy files pass and only the short one is reported.

Flowchart

graph TD
    A["Scan loads a partition"] --> B["Count the records loaded"]
    B --> C{"Is the count at or above 500?"}
    C -->|Yes| D["Partition passes"]
    C -->|No| E["Shape Anomaly naming<br/>the partition and both counts"]
    D --> F["Next partition"]
    E --> F

Equivalent SQL

-- The files the check would flag, if the folder were queried as one table
-- with a column identifying the source file.
SELECT source_file AS partition_name,
       count(*) AS records
FROM store_transactions
GROUP BY source_file
HAVING count(*) < 500;

The situation: A product catalog is rebuilt from scratch on every load: the table is replaced rather than appended to. A rebuild that produces almost nothing silently empties the catalog and every join against it starts returning nothing. A floor of 5,000 records makes an empty rebuild visible before it reaches the reports.

Check configuration

Payload

{
    "description": "The product catalog must never be rebuilt with fewer than 5,000 products.",
    "rule": "minPartitionSize",
    "fields": null,
    "container_id": 733,
    "properties": {
        "value": 5000
    },
    "tags": ["reference-data", "load-monitoring"],
    "additional_metadata": {"jira": "DATA-10339"},
    "template_id": null,
    "status": "Active",
    "owner_id": 7,
    "default_anomaly_assignee_id": 41
}

Sample Data (a Full scan loads the whole table as one partition)

What gets flagged

The rebuild produced an empty table: the upstream source was unavailable and the job wrote nothing rather than failing. Without this check the emptiness would only surface when downstream joins started returning no rows. The Scan loaded the whole table as one partition, so a single anomaly is raised for it.

Flowchart

graph TD
    A["Scan loads a partition"] --> B["Count the records loaded"]
    B --> C{"Is the count at or above 5,000?"}
    C -->|Yes| D["Partition passes"]
    C -->|No| E["Shape Anomaly naming<br/>the partition and both counts"]
    D --> F["Next partition"]
    E --> F

Equivalent SQL

-- The check would flag the rebuild when this returns a row.
SELECT count(*) AS records
FROM product_catalog
HAVING count(*) < 5000;