Transformation Types

Every Computed Field is built around a transformation type. Four are available today: Cast, Cleaned Entity Name, Convert Formatted Numeric, and Custom Expression. Pick the type that matches the shape of the value you want to derive.

Cast

The Cast transformation converts field values to a specified target data type using Spark SQL casting rules. Use it when a field is ingested as a string but needs to behave as a numeric, date, timestamp, or boolean value for profiling, checks, or downstream computations.

When to Use Cast

Reach for Cast when:

• A numeric field is ingested as a string (for example, "1000" instead of 1000).
• Dates or timestamps need explicit typing.
• Boolean logic is required on string values.
• Quality checks or aggregations require strict data types.

Options for Cast

Target Type Examples

Format Examples (Date / Timestamp Casting)

The Format field is only required when casting string values into date or timestamp.

Common examples:

Cleaned Entity Name

The Cleaned Entity Name transformation removes common business signifiers from entity names, producing a cleaner and more uniform version of the source value.

Options for Cleaned Entity Name

Example for Cleaned Entity Name

Convert Formatted Numeric

The Convert Formatted Numeric transformation converts formatted numeric values into a plain numeric format, stripping out characters like commas, currency symbols, or parentheses that are not numerically significant.

Options for Convert Formatted Numeric

Example for Convert Formatted Numeric

Custom Expression

The Custom Expression transformation creates a new field based on a Spark SQL expression. Use it for logic that is not covered by the other transformation types: combining fields, applying conditional logic, or using window and aggregate functions.

Options for Custom Expression

Using Custom Expression

You can combine multiple fields, apply conditional logic, or use any valid Spark SQL function to derive your new computed field.

Example: to create a field that sums two existing fields, use the expression SUM(field1, field2).

Advanced Example: Detecting Overlapping Leases

Consider a log of leases where each row captures a single lease's details:

Lease 1 overlaps with Lease 3 for the same Asset (42). This is hard to detect without transforming the data. A Computed Field named Next_Lease_Start using the following custom expression produces the value of the next lease's start date for each asset:

LEAD(Lease_Start, 1) OVER (PARTITION BY AssetID ORDER BY Lease_Start)

The table now includes the derived field:

With Next_Lease_Start in place, you can author a quality check stating that Lease_End must always be less than Next_Lease_Start to catch overlaps. Qualytics AI will automatically generate that check for you at AI Effort = High.

More Examples for Custom Expression