Complex Data Types¶

Qualytics provides comprehensive support for complex data types (Arrays, Structs, and nested combinations) across the entire data quality pipeline: schema discovery, profiling, quality checks (scanning), anomaly reporting, and automatic constraint inference.

What Are Complex Data Types?¶

Most data fields are simple (scalar) values like a single number, string, date, or boolean. Complex data types contain structured or repeated data within a single field.

Think of it this way:

• A simple field is like a single box holding one item, such as a name, a price, or a date.
• A complex field is like a box that contains other boxes inside it, such as a list of tags, a customer record with multiple sub-fields, or an order with multiple line items.

Here are the three main types:

Why Does This Matter?¶

Modern data formats (JSON, Parquet, Avro) commonly use complex types. Without proper support, quality checks could only validate flat, scalar columns, leaving structured and nested data completely unmonitored.

In simple terms: If your data has lists or nested records inside a single column, you need complex type support to validate it.

With complex type support, you can answer questions like:

• "Are all email addresses in the list valid?"
• "Does the tags list have at least 2 items?"
• "Is the user's country one of US, CA, or GB?"
• "Is every item's price between 0 and 200?"

Why Should You Use Complex Data Type Checks?¶

1. Validate Data You Couldn't Before¶

Without complex type support, arrays and structs were invisible to quality checks. Now you can monitor every level of your nested data.

2. Catch Hidden Issues Early¶

A table might look fine at first glance (no missing rows, no schema errors), but inside an array, there could be null elements, unexpected values, or duplicates. Complex type checks catch these before they reach dashboards or models.

3. No New Rules to Learn¶

Qualytics reuses the same quality check rules you already know (notNull, between, expectedValues, etc.) and makes them work intelligently with arrays and structs. No special syntax or new rule types required.

4. Automatic Discovery¶

When you profile your data, Qualytics automatically discovers all nested fields and makes them individually checkable. You don't need to configure anything. Just profile and start creating checks.

Real-Life Example: E-Commerce Order Validation¶

The Situation¶

An e-commerce company stores order data in Parquet files on Amazon S3. Each order record contains a line_items field, which is an Array of Structs where each item has a sku, quantity, and price.

{
  "order_id": "ORD-9921",
  "customer": { "name": "Alice Smith", "country": "US" },
  "line_items": [
    { "sku": "SKU-1001", "quantity": 2, "price": 29.99 },
    { "sku": "SKU-1002", "quantity": -1, "price": 49.99 }
  ]
}

The Problem¶

The data pipeline ran without errors: no missing rows, no schema mismatches. However, some financial reports were showing incorrect revenue totals. The issue was hidden inside the nested line_items array: some items had negative quantities, which should never happen.

The Solution¶

After profiling the data in Qualytics, the team discovered the nested fields line_items.sku, line_items.quantity, and line_items.price. They set up the following quality checks:

The Outcome¶

The next scan immediately flagged 23 orders with negative quantities and 4 orders with unrecognized SKU formats. The root cause was traced to an upstream system migration that introduced bad default values. The checks now run automatically on every scan, preventing bad data from reaching financial reports.

Supported Complex Types¶

What the Columns Mean¶

Profiling: Complex types require additional profiling logic beyond what scalar fields need:

• Array-level statistics are computed: min/max/mean array length and total element count. These metrics don't exist for scalar fields. The std dev of length is not computed.
• Element-level statistics treat each array element as a value and compute distinct count, min/max (numeric arrays), and min/max string length (string arrays). A separate element histogram shows the frequency distribution of individual values inside arrays.
• Projection of Array[Struct] fields into individual sub-fields (e.g., items.sku, items.price) happens during profiling. This is the step that creates the dot-path fields you use for quality checks.
• Map keys held by at least 5% of the non-null maps are promoted to fields of their own (e.g., attrs.src), each with a full standard profile. The map itself keeps its entry counts, distinct keys, key frequencies, and a profile of all its values.

Scanning: The scan engine handles complex types differently from scalar fields:

• Arrays are validated element-wise. For example, notNull on items.sku checks every element in every row's array, not just one value per row.
• Dot-paths are navigated to reach nested struct sub-fields within the source data.

Constraint Authoring: Qualytics AI must understand which check types and thresholds are valid for arrays vs scalars:

• minLength/maxLength on an Array field checks array length (number of elements), not string length.
• expectedValues on an Array[String] infers from element values, not the whole array as a blob.
• "Limited" (Array of Arrays) means the engine cannot generate meaningful constraints for doubly-nested structures.

How Fields Are Discovered and Named¶

All fields, including both top-level parents and complex type sub-fields, only appear in the UI after you run a Profile operation. The profile operation discovers all fields, expands complex types (Array[Struct], nested Structs) into dot-path sub-fields, and computes statistical profiles.

In simple terms: If you have a user struct with a name sub-field that has a first value, Qualytics names it user.name.first, and you can create quality checks on it just like any other field.

Naming Convention¶

Top-level fields:      id, event_time, tags
Struct sub-fields:     metadata.version, metadata.active
Nested Struct fields:  user.profile.name
Array[Struct] fields:  items.sku, items.price.amount

Examples¶

Given this data structure:

{
  "id": 1,
  "tags": ["billing", "prod"],
  "user": {
    "name": { "first": "Alice", "last": "Smith" },
    "emails": ["alice@example.com"]
  },
  "items": [
    { "sku": "SKU-1001", "price": { "amount": 29.99, "currency": "USD" } }
  ],
  "metadata": { "source": "api" }
}

After running a profile operation, Qualytics discovers all fields, both top-level fields and their sub-fields:

How Array[Struct] Fields Are Projected¶

When Qualytics encounters an Array of Structs column, it automatically projects (extracts) each struct member into its own independent field. This is the key mechanism that makes nested data quality-checkable.

Example: A column line_items: Array[Struct{sku: String, qty: Int, price: Decimal}] becomes:

line_items             (Array[Struct], parent field, declared)
├── line_items.sku       (Array[String], projected sub-field)
├── line_items.qty       (Array[Integer], projected sub-field)
└── line_items.price     (Array[Decimal], projected sub-field)

Each projected sub-field becomes its own array column and receives full array-level + element-level profiling. Quality checks can target each sub-field independently. For example, you can check that every price element is positive, or that every sku matches a pattern.

For deeply nested structures like Array[Array[Struct{a: String, b: Int}]], Qualytics handles the nesting transparently so you see the projected sub-fields just like any other field. The original schema type is preserved in metadata, so the UI and API always show the true declared type.

Struct Fields in the UI Sidebar¶

Struct fields appear as expandable nodes in the field sidebar tree. Clicking a struct parent reveals its child fields. A dedicated "Fields" tab appears on the parent field's detail page, listing all child sub-fields. This hierarchy makes it easy to navigate complex schemas.

Field Metadata¶

After profiling, Qualytics provides metadata that helps you understand how each field was discovered and how its type was determined.

Column Origin¶

Tells you how the field got into your field list:

Type Source¶

Tells you how the data type was determined:

Element Type Source¶

Same concept as Type Source, but specifically for array element types. Only populated when the field is an Array type.

• Tracks whether the element type (e.g., String in Array[String]) came from the schema or was inferred from values.
• Type inference within arrays: Qualytics performs type inference on array elements. For example, an Array[String] where all elements are numeric strings (e.g., ["100", "200", "350"]) will have its element type inferred as Integral. This enables numeric checks like between and notNegative on arrays that are technically typed as strings but contain numeric data.

Profiling Behavior¶

When you run a Profile operation on data containing complex types, Qualytics generates rich statistical profiles, just as it does for scalar fields, but with additional metrics specific to arrays and their elements.

In simple terms: Profiling tells you "what does my data look like?" For arrays, you get two layers of answers: what the arrays look like (sizes, completeness) and what the values inside the arrays look like (ranges, patterns, distribution).

General Profiling Behavior¶

• Array fields are fully profiled with both array-level and element-level statistics.
• Struct sub-fields are discovered and appear as individual fields with dot-notation paths. Each sub-field is profiled independently.
• Nested combinations are fully supported. Arrays of Structs, Structs within Structs, etc. are all flattened and profiled.
• Map fields are profiled with their entry counts, distinct keys, most frequent keys, and a profile of all their values. Frequent keys become fields of their own and are profiled like scalar fields. See Map Field Profiling.

Array-Level Statistics¶

For every Array field, Qualytics computes statistics about the arrays themselves (how many elements each row's array contains):

Element-Level Statistics¶

For Array fields, Qualytics also profiles the individual elements inside the arrays. These statistics describe the values contained within the arrays:

Example: For a scores field of type Array[Long] containing values like [85, 92, 78], [60, 95], [100, 45, 88]:

• Element data type: Integral
• Element approx distinct count: ~8
• Element min: 45
• Element max: 100

Value Distribution and Histograms¶

For scalar (non-array) fields, Qualytics produces a single histogram showing the distribution of values across rows. For Array fields, Qualytics produces two separate distributions, each answering a different question.

Array Value Distribution (Standard Histogram)¶

What it answers: "What does the overall column look like across rows?"

For array fields, the standard histogram shows the distribution of the array as a column. This is the same histogram type used for scalar fields. Depending on the field type, this may show the distribution of array sizes or the top-level column values.

Example: tags field (Array[String])

Suppose the tags field across 5 rows has these arrays:

• Row 1: ["billing", "prod"] (2 elements)
• Row 2: ["billing", "staging", "alerts"] (3 elements)
• Row 3: ["prod", "alerts"] (2 elements)
• Row 4: ["billing"] (1 element)
• Row 5: ["prod", "staging", "alerts", "export"] (4 elements)

The standard histogram would represent the distribution at the column level. The accompanying array-level statistics would show:

• Min Length: 1 (smallest array)
• Max Length: 4 (largest array)
• Mean Length: 2.4 (average array size)

Element Value Distribution (Element Histogram)¶

What it answers: "What individual values appear inside the arrays, and how often?"

The element histogram is unique to array fields. It flattens all arrays across all rows and counts the frequency of each individual element value. This gives you a complete picture of the value landscape inside your arrays.

Example: same tags field as above

After flattening all arrays into individual elements:

• "billing" appears in rows 1, 2, 4 → count = 3
• "prod" appears in rows 1, 3, 5 → count = 3
• "alerts" appears in rows 2, 3, 5 → count = 3
• "staging" appears in rows 2, 5 → count = 2
• "export" appears in row 5 → count = 1

The element histogram would show:

Side-by-Side Comparison¶

Scalar vs. Array Field Profiles¶

To make the difference concrete, here is a comparison of what profile data you get for a scalar field vs. an array field.

Scalar Field Profile (e.g., age of type Long)¶

Array Field Profile (e.g., tags of type Array[String])¶

Array Field Profile for Numeric Types (e.g., scores of type Array[Long])¶

Struct Field Profiling¶

For Struct fields, Qualytics provides:

Example: For a user struct containing {name: {first: String, last: String}, age: Int}:

• The user field profile shows sub-field paths: ["user.name.first", "user.name.last", "user.age"]
• user.name.first gets its own profile with completeness, distinct count, histogram, etc.
• user.age gets its own profile with min, max, mean, histogram, etc.

Map Field Profiling¶

A Map column holds key-value pairs whose keys vary from row to row, so Qualytics profiles it in three layers: the map as a whole, its keys, and its values.

The field's type reads Map followed by the type of its values, for example Map<Fractional>, or plain Map when the values are structs, arrays, or maps.

Map keys become fields. A key held by at least 5% of the non-null maps is promoted to a field of its own, named like a struct sub-field: attrs.src for the key src of the map attrs. A key that contains a dot is written in backticks, as in attrs.`a.b`. The field carries the column origin map_key, is labeled Map Key with a tooltip naming its map, and gets a full standard profile: type inference, min and max, histogram, and the rest. Its value is null in the rows whose map does not hold the key.

• Only maps with string keys promote their keys. A map with integer or date keys keeps its key frequencies, entry counts, and values profile, and one of its keys can still be read with a computed field.
• At most 20 keys are promoted per map, and 200 per container. The empty key is never promoted, and of two keys that differ only in case, the more frequent one is.
• Nested maps promote keys up to three levels deep (attrs.geo.city), and a map inside a struct works the same way (s.attrs.k). A struct value gets its own sub-fields, and a map value gets its own key frequencies.
• An array of maps promotes keys one level deep. items.sku becomes an array field holding each element map's value, with the array as its parent, and the array field shows the entries per map as Min Entries, Max Entries, Avg Entries, and Total Entries.
• Once a key is promoted during a profile, it stays promoted for the rest of that profile, even if its share drops under 5% in later partitions.
• A key that falls under 5% is marked Missing at the next profile, like any field that disappears, so its profiles and history stay. Missing map keys are left out of the default field listing and come back active when the key is promoted again.

A map holds values of one type, so what the profile shows depends on that type.

Example with text values: For an attrs map holding {src: "web", campaign: "spring"} in most rows:

• The attrs field profile shows its entries, distinct keys, key frequencies, and the values profile pooled across both keys.
• attrs.src and attrs.campaign each get their own profile with completeness, distinct count, histogram, and so on.

Example with nested maps: For an attrs map holding {geo: {city: "Lisbon", country: "PT"}} in most rows:

• The attrs field profile shows its entries, distinct keys, and key frequencies. There is no values profile, because its values are maps rather than simple values.
• attrs.geo gets its own key frequencies, and attrs.geo.city and attrs.geo.country each get their own profile.

Profiling Summary by Complex Type¶

Nested Array Profiling¶

For nested arrays that arise from struct projections (e.g., orders.items.price producing Array[Array[Int]]), Qualytics automatically flattens the nesting so that element-level statistics cover all values regardless of nesting depth.

For top-level Array[Array[T]] columns in the source schema, element statistics operate on the inner arrays as units rather than individual scalar values.

Array-level metrics like minLength / maxLength always measure the outermost array dimension, meaning how many top-level elements each row's array contains.

Quality Check Rules for Arrays¶

One of the most important things to understand: there are no dedicated "array" rule types. You use the same quality check rules you already know (like notNull, between, expectedValues) and Qualytics makes them work intelligently with arrays.

In simple terms: If you know how to create a check on a regular field, you already know how to create a check on an array field.

How It Works¶

When you create a quality check targeting an array field, Qualytics needs to know one thing: are you checking the array itself or each element inside the array?

Think of it like inspecting a bag of apples:

• Array-level check: "Does this bag have at least 3 apples?" (checking the container)
• Element-level check: "Is every apple in this bag fresh?" (checking each item)

For most rules, this is automatically determined. But for a few "dual-mode" rules, you control the behavior by setting a property called is_element_context.

The is_element_context Property¶

The is_element_context property is a boolean toggle available on quality checks. It appears in the check's properties section and controls how a rule behaves when targeting an array field.

Dual-Mode Rules¶

These rules have two distinct behaviors depending on the flag. Qualytics does not auto-set the flag for these rules, so you must explicitly choose which mode you want.

Example: Two Different Checks on the Same tags Array Field¶

Both the container-level and element-level variants can exist simultaneously on the same field. They are treated as separate checks.

Auto-Set Rules¶

These rules automatically set is_element_context: true when you target an Array field. You do not need to set the flag yourself. Just create the check as you normally would and point it at the array field.

Rules That Do Not Support Arrays¶

These rules only work on scalar fields or operate at the dataset/shape level. They do not support is_element_context:

Quick Decision Guide¶

Use this flowchart to decide how to set up your quality check on an array field:

graph TD
    A["Is the target field an Array type?"] --> |No| B["Use the rule normally<br/>(scalar behavior)"]
    A --> |Yes| C["Is the rule in the<br/>Dual-Mode list?"]
    C --> |Yes| D["Do you want to check<br/>EACH ELEMENT?"]
    D --> |Yes| E["Set is_element_context: true"]
    D --> |No| F["Leave is_element_context unset<br/>(container-level check)"]
    C --> |No| G["Is the rule in the<br/>Auto-Set list?"]
    G --> |Yes| H["Just create the check normally<br/>(element-wise is automatic)"]
    G --> |No| I["Rule doesn't support arrays<br/>(use satisfiesExpression instead)"]

Complete Rule Support Table¶

How Element-Level Validation Works¶

When an element-level check runs against an array, the check passes only if the condition holds for every element in the array. For nested structures (arrays within arrays), validation applies at every nesting level.

Profiling vs. Check Assertions¶

Profiling computes aggregated statistics across all values, while check assertions validate each row individually. A row with 99 passing elements and 1 failing element counts as 1 failing row because checks are all-or-nothing per row by design.

Completeness: Three Independent Levels¶

For array fields, "completeness" is not a single number. Think of it like checking a carton of eggs: the carton can exist (column completeness), the carton can have eggs in it (non-emptiness), and each egg can be intact (element completeness). These are three independent questions:

A row can have a non-null array that is empty. A non-empty array can contain null elements. These checks are fully independent and should be set up separately based on your data quality requirements.

Quality Checks on Maps¶

A Map column and its promoted keys take checks in different ways.

On the map itself, rules that treat the whole value apply, such as notNull. These checks do not read the keys or the values individually.

On a promoted key, the key is evaluated like a scalar field, and a row whose map does not hold the key reads as null. These rules apply: between, greaterThan, lessThan, minValue, maxValue, positive, notNegative, equalTo, matchesPattern, minLength, maxLength, isType, expectedValues, afterDateTime, beforeDateTime, notFuture, containsEmail, containsUrl, containsCreditCard, containsSocialSecurityNumber, and notNull, plus requiredValues, distinctCount, unique, and satisfiesExpression when you author them yourself. A key of an array of maps, such as items.sku, is an array field, so only the rules that accept arrays apply to it.

Any other rule is refused on a map key, in the check form and through the API, with a message that names the rule. Entity Resolution, cross-container rules such as existsIn and dataDiff, and field-to-field comparisons such as equalToField are refused. A map key also cannot be a referenced field: not the comparison field of a rule, not a row identifier, not the distinction field of Entity Resolution, and not the field an anomaly description is built from.

A key that cannot be read when the scan runs, for example a key reached through nested arrays or a key of a map whose keys are not strings, is reported as not asserted. The container's scan still succeeds.

How Array Context Is Determined¶

A field is considered to be in "array context" if:

• The field itself is an Array type, OR
• Any parent in the dot-path is an Array type

This determines whether element-wise validation applies.

Examples¶

tags                  → ARRAY CONTEXT (tags is Array[String])
items.sku             → ARRAY CONTEXT (items is Array[Struct], so sku is inside an array)
metadata.version      → NOT array context (metadata is Struct, version is a plain scalar)
items.features        → ARRAY CONTEXT (items is Array[Struct], features is Array[String], nested arrays)

Automatic Constraint Inference¶

After profiling your data, Qualytics can automatically suggest quality checks for complex type fields. These suggestions are based on patterns detected in your data.

AI Managed Checks for Complex Types¶

AI Generation Behavior¶

Qualytics AI automatically skips check types that are not meaningful for complex types:

• unique checks are not suggested for complex type columns
• expectedValues is not suggested for Array[Struct] or Array[Array] fields
• On map keys, relations, existsIn, unique, distinctCount, requiredValues, and the other multi-row or multi-field rules are not suggested

Filtering Quality Checks by Element Context¶

You can filter your quality checks to show only container-level or element-level checks:

• Element-level checks only: Filter with is_element_context = true to see all checks that validate individual array elements.
• Container-level checks only: Filter with is_element_context = false to see all checks that validate the column/array as a whole.

This is useful when reviewing your check coverage on array fields, where you may have both types of checks active.

Check Overlap Detection¶

Qualytics enforces a uniqueness constraint to prevent duplicate quality checks. You can have at most:

• One check at 100% coverage per combination of rule type + fields + filter + is_element_context
• One check at less than 100% coverage per the same combination

What This Means in Practice¶

CONFLICT (will be rejected):
  minLength on tags, value=2, coverage=100%   ← First check, created OK
  minLength on tags, value=3, coverage=100%   ← REJECTED, same rule + field + coverage band

OK (different coverage bands):
  minLength on tags, value=2, coverage=100%   ← 100% coverage band
  minLength on tags, value=3, coverage=50%    ← Less-than-100% coverage band, no conflict

Anomaly Messages¶

An array field uses the same anomaly message as a scalar field.

Record-Level Messages (Individual Row Violations)¶

These messages appear when a specific row fails a check:

Shape-Level Messages (Dataset-Wide Aggregates)¶

These messages summarize violations across the entire dataset:

Unique, Not Empty, Min Length, and Max Length evaluate an array field as a whole, so they report at the shape level only and have no per-row message. See Unique and Not Empty.

Configuring Quality Checks: Practical Examples¶

Now that you understand the concepts, let's put them into practice. This section provides ready-to-use configuration examples for every type of quality check on complex data types. Use these as templates when setting up your own checks.

Array Size Checks¶

Minimum Array Length (minLength)¶

Ensure an array has at least a specified number of elements. Leave is_element_context unset: with the flag on, the rule checks the length of each element instead.

{
  "fields": ["tags"],
  "rule": "minLength",
  "description": "Tags array must have at least 2 elements",
  "properties": {
    "value": 2
  }
}

{
  "fields": ["user.profile.attributes.roles"],
  "rule": "minLength",
  "description": "User roles array must have at least 1 element",
  "properties": {
    "value": 1
  }
}

Maximum Array Length (maxLength)¶

Ensure an array has at most a specified number of elements.

{
  "fields": ["tags"],
  "rule": "maxLength",
  "description": "Tags array must have at most 5 elements",
  "properties": {
    "value": 5
  }
}

Array Element Uniqueness and Completeness¶

No Duplicate Elements (unique with is_element_context)¶

Ensure that no array contains duplicate values.

{
  "fields": ["tags"],
  "rule": "unique",
  "description": "Tags array must not contain duplicate values",
  "properties": {
    "is_element_context": true
  }
}

No Null Elements (notNull with is_element_context)¶

Ensure that no array contains null values.

{
  "fields": ["tags"],
  "rule": "notNull",
  "description": "Tags array must not contain null elements",
  "properties": {
    "is_element_context": true
  }
}

Array Must Not Be Empty (notEmpty)¶

Ensure that every array has at least one element.

{
  "fields": ["tags"],
  "rule": "notEmpty",
  "description": "Tags array must not be empty (must have at least one element)"
}

Array Element Value Validation¶

These rules automatically validate every element in the array. No special flags needed.

Expected Values (expectedValues)¶

Ensure every element in the array is one of the allowed values.

{
  "fields": ["tags"],
  "rule": "expectedValues",
  "description": "All tag values must be from the allowed set",
  "properties": {
    "list": ["alerts", "beta", "billing", "export", "ingest", "onboarding", "prod", "retention", "staging", "transform"]
  }
}

{
  "fields": ["items.sku"],
  "rule": "expectedValues",
  "description": "All item SKUs must be from the known set",
  "properties": {
    "list": ["SKU-1001", "SKU-1002", "SKU-1003"]
  }
}

Pattern Matching (matchesPattern)¶

Ensure every element in the array matches a regex pattern.

{
  "fields": ["tags"],
  "rule": "matchesPattern",
  "description": "All tag values must be lowercase letters only",
  "properties": {
    "pattern": "^[a-z]+$"
  }
}

{
  "fields": ["items.sku"],
  "rule": "matchesPattern",
  "description": "All item SKUs must match the SKU-NNNN pattern",
  "properties": {
    "pattern": "^SKU-\\d{4}$"
  }
}

Numeric Range (between)¶

Ensure every numeric element in the array is within a range.

{
  "fields": ["scores"],
  "rule": "between",
  "description": "All score values must be between 0 and 100",
  "properties": {
    "min": 0,
    "max": 100
  }
}

{
  "fields": ["items.price.amount"],
  "rule": "between",
  "description": "All item prices must be between 0 and 200",
  "properties": {
    "min": 0,
    "max": 200
  }
}

Non-Negative Values (notNegative)¶

Ensure every numeric element in the array is zero or positive.

{
  "fields": ["scores"],
  "rule": "notNegative",
  "description": "All score values must be non-negative"
}

{
  "fields": ["items.qty"],
  "rule": "notNegative",
  "description": "All item quantities must be non-negative"
}

String Length on Array Elements (minLength / maxLength)¶

When applied to a string array with is_element_context: true, these check the character length of each string element.

{
  "fields": ["tags"],
  "rule": "minLength",
  "description": "All tag values must have at least 2 characters",
  "properties": {
    "value": 2,
    "is_element_context": true
  }
}

{
  "fields": ["tags"],
  "rule": "maxLength",
  "description": "All tag values must have at most 15 characters",
  "properties": {
    "value": 15,
    "is_element_context": true
  }
}

Email Pattern Detection (containsEmail)¶

Ensure every string element in the array contains a valid email address.

{
  "fields": ["user.emails"],
  "rule": "containsEmail",
  "description": "All email values in the array must contain valid email addresses"
}

Credit Card Pattern Detection (containsCreditCard)¶

{
  "fields": ["credit_cards_array"],
  "rule": "containsCreditCard",
  "description": "All values in credit cards array must contain valid credit card numbers"
}

URL Pattern Detection (containsUrl)¶

{
  "fields": ["urls_array"],
  "rule": "containsUrl",
  "description": "All values in URLs array must contain valid URLs"
}

SSN Pattern Detection (containsSocialSecurityNumber)¶

{
  "fields": ["ssns_array"],
  "rule": "containsSocialSecurityNumber",
  "description": "All values in SSNs array must contain valid SSN patterns"
}

Struct Sub-Field Checks (Dot-Notation)¶

These use standard scalar rules targeting nested struct fields via their dot-notation paths.

{
  "fields": ["user.name.first"],
  "rule": "notNull",
  "description": "User first name must not be null"
}

{
  "fields": ["geo.country"],
  "rule": "expectedValues",
  "description": "Country must be one of the allowed values",
  "properties": {
    "list": ["CA", "DE", "GB", "IN", "JP", "US"]
  }
}

{
  "fields": ["metrics.cost.amount"],
  "rule": "between",
  "description": "Cost amount must be between 0.001 and 0.025",
  "properties": {
    "min": 0.001,
    "max": 0.025
  }
}

{
  "fields": ["session.attrs.utm.source"],
  "rule": "expectedValues",
  "description": "UTM source must be one of the known values",
  "properties": {
    "list": ["ads", "newsletter", "organic", "partner"]
  }
}

Expression Checks on Complex Data¶

Use satisfiesExpression to write custom validation expressions that can reference array sizes and struct fields.

{
  "fields": [],
  "rule": "satisfiesExpression",
  "description": "Tags must have at least 2 elements and items at least 1",
  "properties": {
    "expression": "size(tags) >= 2 AND size(items) >= 1"
  }
}

{
  "fields": [],
  "rule": "satisfiesExpression",
  "description": "Latency must be under 1000ms",
  "properties": {
    "expression": "metrics.latency_ms < 1000"
  }
}

Multi-Field Checks¶

At Least One Field Not Null (anyNotNull)¶

{
  "fields": ["nullable_field_a", "nullable_field_b", "nullable_field_c"],
  "rule": "anyNotNull",
  "description": "At least one of the nullable fields must have a value"
}

Shape-Level Checks (Dataset-Wide Validation)¶

These checks validate properties of the entire dataset, not individual records.

{
  "fields": ["geo.country"],
  "rule": "distinctCount",
  "description": "There must be exactly 6 distinct countries",
  "properties": {
    "value": 6,
    "comparison": "eq"
  }
}

{
  "fields": [],
  "rule": "fieldCount",
  "description": "Dataset must have exactly 36 fields",
  "properties": {
    "value": 36,
    "comparison": "eq"
  }
}

{
  "fields": ["id"],
  "rule": "sum",
  "description": "Sum of all IDs must equal 5050",
  "properties": {
    "value": 5050
  }
}

Quality Check Rule Reference¶

Complete reference of all quality check rules and their complex type support.

Configuration¶

No special configuration is needed to enable complex type support. It is automatic:

• If a field is an Array type, array-specific checks and element-wise rules apply automatically.
• If a field is a Struct type, sub-fields are discovered during profiling and are individually checkable using dot-notation paths.
• For dual-mode rules (minLength, maxLength, notNull, unique, notEmpty), set is_element_context: true in the check properties to get array-element behavior.
• For all other array-compatible rules, element-wise validation is automatic when the target field is an array.

UI Experience¶

Visual Indicators¶

Tooltips show the full composite type notation (e.g., Array[Struct{sku: String, qty: Int}]).

When a quality check has is_element_context active, the check form shows a visual indicator, an array icon with a tooltip explaining: "Check runs on each element individually."

Field Navigation¶

• Complex type fields appear as expandable nodes in the sidebar tree
• Each sub-field has its own detail page with profile metrics, checks, and quality score
• A map's promoted keys appear under the map in the sidebar tree, the same way struct sub-fields do
• A "Fields" tab appears on any parent field that has children, listing all child sub-fields
• Deleting a parent field cascades to delete all of its child sub-fields

Field Display¶

• "Declared Type" means the field's type came from the source schema.
• "Detected Type" means the field's type wasn't declared by the source, so Qualytics picked the data type that best matches the field's actual values during profiling.
• For array fields, the label reflects the element type source specifically. An array may show "Detected Type" if its element type was detected, even if the array column itself was declared in the schema.
• "Map Key Type" labels a field that is a promoted key of a map. Its tooltip names the map it comes from. A map field's own type reads Map<Fractional> or Map<String> after the type of its values, or plain Map.
• In the check, template, and bulk-create forms, a map key is disabled under a rule that refuses it, with a message naming the rule, such as "Map keys can't be used with Entity Resolution".

Restrictions¶

• Complex type fields and projected sub-fields cannot be renamed because names come from the source schema structure
• Struct parent fields show completeness, distinct values, and a histogram of the whole struct value. Numeric metrics such as min, max, and mean live on the sub-fields

API Surface¶

Complex types work through existing API endpoints with additional response fields and query parameters. No new endpoints were needed.

Limitations¶