Materialize Operation

Materialize Operation captures snapshots of selected containers from a datastore and writes them to its enrichment destination for seamless data loading. Users can run it instantly or schedule it at set intervals, ensuring structured data is readily available for analysis and integration.

Show me how

The app can walk you through this. Click Show me how in the Materialize dialog's header, or press H while it is open, and the Run a Materialize walkthrough highlights each step while you fill in the real form. See The How-tos Scope for every way to start one.

Materialize Naming Conventions

To keep materialized data organized and compatible across different enrichment destinations, the system applies specific naming conventions. These conventions ensure that the resulting container names remain valid, readable, and free of conflicts.

Default Naming Convention

Used when the container name is safe to use as-is.

<enrichment_prefix>_mat_<container_name>

This naming format is applied when:

The container name length is 60 characters or less.
The container name does not contain special characters that may cause invalid table or file names.

Example:

If the enrichment prefix is _sales and the container name is orders_2024:

_sales_mat_orders_2024

Fallback Naming Convention

If the container name contains characters that may cause issues in downstream systems, the system switches to a safer naming structure by using the container ID instead.

<enrichment_prefix>_materialize_<container_id>

This fallback is used when:

The container name exceeds 60 characters.
The container name includes restricted or special characters. (e.g., symbols, glob patterns when moving from DFS to JDBC).

Example:

If the enrichment prefix is _sales and the container ID is 1023456:

_sales_materialize_1023456

Note

The fallback naming ensures successful loading into the enrichment destination by preventing invalid or non-compliant table names.

Let’s get started 🚀

Step 1: Select the datastore whose containers you want to materialize. It must have an enrichment destination linked.

select

For demonstration purposes, we have selected the “COVID-19 Data” Snowflake datastore.

datastore

Step 2: Open the Tables or Files tab, open the actions menu of one of its containers, and click Materialize. The same dialog also opens from the tree footer: under Writes enrichment to, open the menu next to the destination and click Materialize, then choose the containers to materialize inside the dialog.

Step 3: After clicking Materialize, a modal window appears, allowing you to configure the data export settings for the Materialize Operation.

window

Step 4: Choose whether to materialize all containers, specific containers, or containers with selected tags, then click Next.

record

Run Now

Click Run Now to instantly materialize selected containers.

run

After clicking Run Now, a confirmation message appears stating "Operation triggered". Go to the Activity tab to see the progress of the materialize operation: its counter reads 0/N Tables while queued and counts up as each table is written successfully. A table that fails to write is not counted and fails the run, so a failed or aborted run can stop short of N/N; a successful run always ends there.

operation-triggered

Schedule

Timezone-aware scheduling

Schedules can run in any IANA timezone (for example, America/New_York, Europe/Paris, Asia/Tokyo), and Daylight Saving Time transitions are handled automatically. UTC is the default for new and existing schedules. The configured timezone is shown on the schedule card as an abbreviation, such as Schedule (UTC) by default or Schedule (EST) after selecting another timezone.

Deactivating a schedule keeps its cron expression

When you deactivate a schedule, its cron expression is kept. Reactivating it later resumes the same schedule without setting it up again.

If a schedule was deactivated before May 7, 2026 and doesn't run after you reactivate it, re-enter its cron expression once to restore it. Schedules deactivated on or after that date keep working normally.

Step 1: Click Schedule to configure scheduling options for the Materialize Operation.

schedule

Step 2: Choose the Timezone for this schedule. UTC is selected by default. To run in a different timezone, type to search by city, region, or abbreviation and pick an IANA timezone from the list. The selected timezone applies to every tab below, and the banner above the tabs shows the current time in that timezone.

Step 3: Configure the scheduling preferences for the Materialize Operation. Time inputs apply to the timezone selected in Step 2.

Hourly: Runs every set number of hours at a specified minute. (e.g., Every 1 hour at 00 minutes).
Daily: Runs once per day at a specific time. (e.g., Every day at 00:00 in the selected timezone).
Weekly: Runs on selected weekdays at a set time. (e.g., Every Sunday and Friday at 00:00 in the selected timezone).
Monthly: Runs on a specific day of the month at a set time. (e.g., 1st day of every month at 00:00 in the selected timezone).
Advanced: Use Cron expressions for custom scheduling (e.g., 0 12 * * 1-5 runs at 12 PM, Monday to Friday). The Custom Cron Schedule field label shows the abbreviation for the currently selected timezone (for example, Custom Cron Schedule (UTC) or Custom Cron Schedule (EST)).

time

Step 4: Define the Schedule Name to identify the scheduled Materialize Operation when it runs.

name

Step 5: Click Schedule to finalize and schedule the Materialize Operation.

schedule2

After clicking Schedule, a confirmation message appears stating "Operation scheduled". The schedule is listed on the Schedule tab of the Activity page. Each time it runs, the run appears on the Runs tab with the same progress counter.

operation-scheduled

Daylight Saving Time

When you pick a timezone that observes DST (such as America/New_York or Europe/London), the schedule automatically shifts with each transition. A job set to run at 9:00 AM in America/New_York runs at 9:00 AM local time year-round, regardless of whether the zone is in EST or EDT at the time. No reconfiguration is required.

Field Masking and Materialize

If your datastore contains masked fields, masking is applied to the source data before it is written to the enrichment destination during a Materialize operation.

Source record values for masked fields are obfuscated in every container snapshot written to the enrichment destination. This applies to all containers included in the materialize run.

To obtain revealed data in materialized snapshots, enable the Reveal Masked Values toggle when triggering the Materialize operation, or pass include_masked=true via the API.

Note

Masking is applied by Qualytics before writing each container to the enrichment destination. The destination receives masked values by default. Enabling Reveal Masked Values instead writes the original values, so access to the destination must be controlled accordingly.

Review Materialized Data

After a confirmed write, Qualytics syncs and profiles the destination when Sync and profile the enrichment destination is enabled on the source, as it is by default. If it is disabled, run Sync and then Profile on the destination before reviewing the new containers.

Step 1: Open the enrichment destination. In the datastore tree, the Enrichment filter chip narrows the list to enrichment datastores.

Step 2: Open the Tables or Files tab, depending on whether it is a JDBC or a DFS datastore, and filter by Purpose set to Materialize. Each materialized container carries the Materialize badge and a display name built from the source container and datastore, such as ORDERS / Materialized (COVID-19 Data), with the physical name beside it. Its Lineage tab links it back to the source container.

Step 3: Click a materialized container to review the snapshot.

The materialized data is displayed in a table format, showing key details about the selected containers. It typically includes columns indicating data structure, completeness, and uniqueness. You can use this data for analysis, validation, and integration.

preview2