Transformation Types¶

Every Computed Field is built around a transformation type. Four are available today: Cast, Cleaned Entity Name, Convert Formatted Numeric, and Custom Expression. Pick the type that matches the shape of the value you want to derive.

Cast¶

The Cast transformation converts field values to a specified target data type using Spark SQL casting rules. Use it when a field is ingested as a string but needs to behave as a numeric, date, timestamp, or boolean value for profiling, checks, or downstream computations.

When to Use Cast¶

Reach for Cast when:

• A numeric field is ingested as a string (for example, "1000" instead of 1000).
• Dates or timestamps need explicit typing.
• Boolean logic is required on string values.
• Quality checks or aggregations require strict data types.

Options for Cast¶

Target Type Examples¶

Format Examples (Date / Timestamp Casting)¶

The Format field is only required when casting string values into date or timestamp.

Common examples:

Cleaned Entity Name¶

The Cleaned Entity Name transformation removes common business signifiers from entity names, producing a cleaner and more uniform version of the source value.

Options for Cleaned Entity Name¶

Example for Cleaned Entity Name¶

Convert Formatted Numeric¶

The Convert Formatted Numeric transformation converts formatted numeric values into a plain numeric format, stripping out characters like commas, currency symbols, or parentheses that are not numerically significant.

Options for Convert Formatted Numeric¶

Example for Convert Formatted Numeric¶

Custom Expression¶

The Custom Expression transformation creates a new field based on a Spark SQL expression. Use it for logic that is not covered by the other transformation types: combining fields, applying conditional logic, or using window and aggregate functions.

Options for Custom Expression¶

Using Custom Expression¶

You can combine multiple fields, apply conditional logic, or use any valid Spark SQL function to derive your new computed field.

Example: to create a field that adds two existing fields in each row, use the expression field1 + field2. Aggregate functions such as SUM take a single argument, so SUM(field1, field2) is rejected.

Advanced Example: Detecting Overlapping Leases¶

Consider a log of leases where each row captures a single lease's details:

Lease 1 overlaps with Lease 3 for the same Asset (42). This is hard to detect without transforming the data. A Computed Field named Next_Lease_Start using the following custom expression produces the value of the next lease's start date for each asset:

LEAD(Lease_Start, 1) OVER (PARTITION BY AssetID ORDER BY Lease_Start)

The table now includes the derived field:

With Next_Lease_Start in place, you can author a quality check stating that Lease_End must always be less than Next_Lease_Start to catch overlaps. Qualytics AI will automatically generate that check for you at AI Effort = High.

More Examples for Custom Expression¶