Max Partition Size Check Examples

The situation: Order lines land in one daily partition of roughly 300,000 records. A partition well above that means a replay ran twice or an upstream filter was dropped, and the downstream aggregation will double-count. A ceiling of 500,000 leaves room for a busy day while still catching a duplicate load.

Check configuration

Payload

{
    "description": "No daily partition may load more than 500,000 records.",
    "rule": "maxPartitionSize",
    "fields": null,
    "container_id": 145,
    "properties": {
        "value": 500000
    },
    "tags": ["load-monitoring", "ingestion"],
    "additional_metadata": {"jira": "DATA-10300"},
    "template_id": null,
    "status": "Active",
    "owner_id": 7,
    "default_anomaly_assignee_id": 12
}

Sample Data (the partitions this Scan loaded)

What gets flagged

The 11 March partition holds 622,880 records, more than double a normal day and above the 500,000 ceiling. A replay reprocessed the day without clearing the previous load. The two earlier partitions pass. Only the failing partition produces an anomaly.

Flowchart

graph TD
    A["Scan loads a partition"] --> B["Count the records loaded"]
    B --> C{"Is the count at or below 500,000?"}
    C -->|Yes| D["Partition passes"]
    C -->|No| E["Shape Anomaly naming<br/>the partition and both counts"]
    D --> F["Next partition"]
    E --> F

Equivalent SQL

-- The partitions the check would flag.
SELECT dt AS partition_name,
       count(*) AS records
FROM order_lines
GROUP BY dt
HAVING count(*) > 500000;

The situation: A partner drops one CSV per store into a folder every night, each holding that store's transactions. Files run to a few thousand rows. A file an order of magnitude larger means the partner exported the whole chain into one store's file, which would attribute every transaction to the wrong store.

Check configuration

Payload

{
    "description": "No store file may contain more than 20,000 transactions.",
    "rule": "maxPartitionSize",
    "fields": null,
    "container_id": 512,
    "properties": {
        "value": 20000
    },
    "tags": ["partner-feed", "load-monitoring"],
    "additional_metadata": {"jira": "DATA-10312"},
    "template_id": null,
    "status": "Active",
    "owner_id": 7,
    "default_anomaly_assignee_id": 47
}

Sample Data (each file in the folder is one partition)

What gets flagged

store_0144.csv holds 248,300 rows against a normal few thousand: the partner exported the full chain into a single store's file. Because each file is its own partition, the two healthy files pass and only the oversized one is reported.

Flowchart

graph TD
    A["Scan loads a partition"] --> B["Count the records loaded"]
    B --> C{"Is the count at or below 20,000?"}
    C -->|Yes| D["Partition passes"]
    C -->|No| E["Shape Anomaly naming<br/>the partition and both counts"]
    D --> F["Next partition"]
    E --> F

Equivalent SQL

-- The files the check would flag, if the folder were queried as one table
-- with a column identifying the source file.
SELECT source_file AS partition_name,
       count(*) AS records
FROM store_transactions
GROUP BY source_file
HAVING count(*) > 20000;

The situation: A tenant-partitioned table should stay balanced: no single tenant's partition may dominate, because the processing job allocates the same resources to each one. A ceiling of 1,000,000 records flags the tenant whose growth is about to make that job time out.

Check configuration

Payload

{
    "description": "No tenant partition may exceed 1,000,000 events.",
    "rule": "maxPartitionSize",
    "fields": null,
    "container_id": 733,
    "properties": {
        "value": 1000000
    },
    "tags": ["skew", "capacity"],
    "additional_metadata": {"jira": "DATA-10331"},
    "template_id": null,
    "status": "Active",
    "owner_id": 7,
    "default_anomaly_assignee_id": 41
}

Sample Data (the partitions this Scan loaded)

What gets flagged

The initech partition has grown past 1.8 million events while the others sit near 400,000. The check reports it so the partition can be split or given more resources before the job starts failing. Skew is a shape problem, not a value problem, which is why no source records accompany the anomaly.

Flowchart

graph TD
    A["Scan loads a partition"] --> B["Count the records loaded"]
    B --> C{"Is the count at or below 1,000,000?"}
    C -->|Yes| D["Partition passes"]
    C -->|No| E["Shape Anomaly naming<br/>the partition and both counts"]
    D --> F["Next partition"]
    E --> F

Equivalent SQL

-- The partitions the check would flag.
SELECT tenant AS partition_name,
       count(*) AS records
FROM events_by_tenant
GROUP BY tenant
HAVING count(*) > 1000000;