OpenLineage Integration¶

OpenLineage is the open standard for lineage events, used by orchestrators, processing engines, and metadata repositories alike. Qualytics accepts OpenLineage events directly, so any tool that emits them, or a short script that translates another system's exports, can send lineage and dataset metadata to Qualytics without a custom integration.

Every relationship an event describes is kept with a record of who asserted it: the job it describes, the producer that sent the event, the Qualytics user who submitted it, the run, and when. Those records decide when a connection is retired, and you can read them on the graph and through the API.

This page explains how events become connections. For the endpoint, request format, and a Python example, see Send OpenLineage Events in the Lineage API.

Events Qualytics Reads¶

OpenLineage defines three kinds of event. Qualytics reads all three:

Qualytics validates the event envelope, including an RFC 3339 eventTime with a timezone offset or Z and URIs for producer and schemaURL, and the facets it uses. Unknown facets and extra properties are retained in the stored event without being interpreted or applied to an asset, even if an unknown facet has an unexpected shape. Stored raw events and unknown facets are not exposed through the REST API. Qualytics validates the content it uses rather than every field in the published schema, and never fetches a caller-supplied schema URL. Events that can be decoded and stored are retained, including rejected events. A body containing NaN, Infinity, a number that overflows to infinity, a \u0000 character, or an unpaired Unicode surrogate is refused with 400 Bad Request. Malformed gzip compression and JSON nested more than 100 levels also receive 400 Bad Request. A validation error identifies the problem’s location; other events in the request are processed independently.

An event whose eventTime is more than five minutes in the future is rejected. Qualytics orders a job's events by their time, so one producer with a clock running ahead would otherwise outrank every later event of that job.

Job Identity¶

A job is identified by its job.namespace and job.name, as the OpenLineage spec defines it. The producer URI the event carries is recorded, and the most recent one is shown, but it is not part of the job's identity. A producer URI usually includes the integration's version, so upgrading an integration keeps reconciling the same job rather than starting a new one beside it.

How a Dataset Is Matched to a Container¶

OpenLineage names a dataset with a namespace, which identifies the system it lives in, and a name, which identifies the dataset inside that system. For example, a PostgreSQL table has the namespace postgres://warehouse.example.com:5432 and the name analytics.public.orders.

Qualytics matches a dataset to a container in two steps:

1. The namespace selects the connections. Its scheme picks the datastore type and its host picks the connections on that host. The port, when both the namespace and the connection state one, has to agree, except for Hive.
2. The name selects the container. It is compared with the database, schema, and container name of every datastore on those connections.

Names are matched without regard to case. When one datastore holds two tables whose names differ only in case, such as Orders and orders, only the table spelled exactly as the event spells it matches. A name part in identifier quotes ("a.b", `a.b`, or [a.b]) is read as one part, dots included.

Hosts have to match exactly, except where producers shorten them:

• Snowflake producers name the account (snowflake://myorg-myaccount) rather than the full host, so a connection whose host begins with that account matches.
• Redshift producers name cluster.region, which matches a connection host with the same cluster name in that region.
• BigQuery namespaces carry no host, so every BigQuery connection is considered and the project in the name decides.
• Hive ignores the port. A producer often names the metastore's port while the connection holds HiveServer2's.

File and Object Store Datasets¶

A namespace such as s3://my-bucket or abfss://container@account.dfs.core.windows.net names an object store. It matches the DFS connections on the same bucket or authority, treating aliases such as s3a:// and s3:// as one store. The dataset name, a path, then selects the container:

• A container that is a directory, such as a Delta table, matches the datastore's root path followed by the container's relative path and file name.
• A container that is a single file matches the datastore's root path followed by the container's name, extension included.

Paths are compared with their case as written: Sales/orders and sales/orders are different datasets.

Alternate Names¶

When a producer knows a dataset by more than one name, for example a file path and the warehouse table built over it, it can list the others in the dataset's symlinks facet. Qualytics tries the dataset's own name first and then each alternate name, and uses the first that matches a container. Aliases are supplied on the event; there is no persistent mapping configuration for logical source-system names.

Several Matches¶

A dataset name can match a container in more than one datastore, when several connections point at the same host. These are copies of the same physical table. An Admin updates every match. For Members and Managers, only matches in datastores where the caller has Editor permission receive connections or metadata. For example, Editor on one match and Viewer on another updates only the Editor match. There is no configurable canonical datastore selection: all writable matches are used, and a relationship is expanded across matching source and target datastores. If there are known matches but none are writable, the event is denied. See Lineage Permissions.

Datasets That Are Not in Qualytics¶

A dataset that matches no container is kept as an asset outside Qualytics, named namespace/name, so the relationship still shows on the graph. Namespace and name remain separate parts of its identity: slashes inside either part cannot merge two different datasets. Graph node IDs are opaque identifiers; use the returned ID when referring to a node in API requests. See Reading the Graph for how these appear. A relationship with neither side in Qualytics is not drawn yet. Its event is stored, and when a Sync or a direct registration catalogs either dataset, Qualytics replays the event as the user who sent it, within the datastores they could edit when they sent it. A newer COMPLETE run or job definition of the same job replaces what an older event asked to replay within its authorized datastores. An event from a user with narrower permissions preserves replay for the other datastores.

When a later Sync or direct registration brings the table into Qualytics, the jobs that named it can be applied again: the connection moves onto the new container, and any field-level lineage that could not be stored against an outside asset is added. This replay uses the original submitter’s current permissions, limited to the datastores they could edit when they sent the event. Losing Editor permission or restricting the request to a team prevents replay from changing other datastores. A connection stays attached to the outside asset until an authorized replay can replace it.

Send the event again to include a datastore added after a Member or Manager first submitted it, or after their permissions expand.

Matching datastores can register their tables at different times. Registering the first match does not discard pending lineage for the others, even when a newer event replaced the relationship only in the first datastore. Pending relationships remain eligible for authorized replay as the other matching tables arrive.

An unfinished run can report different relationships involving the same datasets. For example, A → B followed by B → A retains both relationships for replay. Different column lineage and metadata omitted from a later report are retained too; only identical job, input and output assertions are treated as repeated reports.

This automatic replay follows lineage relationships. A DatasetEvent that supplied metadata before its dataset was registered has no relationship to trigger replay; resend that metadata after registration.

Field-level Lineage¶

An output dataset's columnLineage facet describes which input fields each output field is derived from. Each pair becomes a field-level connection, including pairs that span two datastores, as long as both datasets and both fields are in Qualytics. A field pair with an outside side is carried by the table-level connection between the two datasets instead.

The transformations the facet lists, such as DIRECT/IDENTITY or INDIRECT/FILTER, are kept with the connection. A field-level connection that has no description gets one written from them. One you have written is left alone.

Sending an Event Again¶

Sending the same event again changes nothing: connections and their records are updated in place, never duplicated, and the request never fails because a connection already exists. An integration can resend everything it knows on every run without tracking what Qualytics already has.

Who Asserted a Connection¶

Each connection keeps a separate record for each job that asserts it, with:

• The job, as namespace/name.
• The producer, the tool or integration that sent the job's latest event.
• The kind of event: a run or a job definition.
• When the job last asserted the connection, taken from the event's time.
• Submitted by, the Qualytics user whose API token sent the event. The producer URI is whatever the client reports, so the submitter is the attribution you can rely on.
• For a run, the run ID and its state. For field-level lineage, the transformations. When the job sent them, its SQL or source code, kept on the table-level connection.

The job's identity, run, datasets, transformations, submitter, and SQL or source code can quote values from your data, so they are shown only to users with the Viewer team permission on the datastore of every container the connection touches, and to Admins. Anyone else who can see the connection sees who asserted it and when: the producer, the kind of event, and the times. See Lineage Permissions.

On the graph, hover the badge on a connection to see who asserted it. Through the API, the same records are in the evidence of the edge response.

Descriptions derived from OpenLineage transformations follow the same Viewer requirement on every response, including the graph and field connections.

An event can attempt at most 10,000 dataset and field relationships, including all writable datastore matches. A single limit covers table-level and field-level relationships together. Larger input/output combinations or column-lineage expansions are rejected with HTTP 422. Use separate job identities for independent parts of a larger job.

Retirement¶

A connection from OpenLineage is retired, not deleted, when nothing asserts it any more.

• A job's latest authoritative event is the full truth for that job. A COMPLETE RunEvent or a JobEvent lists everything the job reads and writes. A connection it no longer lists loses that job's record only when the caller can edit both endpoints. Another job's record stays independent.
• A connection retires when its last record does. A connection two jobs assert stays as long as either one still lists it.
• A run's events count together. A run's COMPLETE event covers everything any of that run's events listed, including START and RUNNING. A producer that lists datasets only at the start of a run does not retire them at the end. Dataset facets omitted from a later listing are retained; a supplied dataset facet replaces the previous facet with the same name, and _deleted: true withdraws it.
• A failed run takes back what only it asserted. A FAIL or ABORT event retires the connections that run asserted and no completed run or job definition has settled. A connection an earlier COMPLETE already listed stays, and so does one that another run of the same job, still in progress, also asserted. This holds when deliveries arrive out of order. A FAIL or ABORT received before an earlier START or RUNNING prevents that delayed event from asserting lineage within the terminal event’s authorized datastore scope.
• Late events are ignored. An event older than the job's latest authoritative event asserts no lineage within that newer event's authorized datastore scope. It can still apply to a matching datastore that the newer event could not edit.
• Dropped and renamed datasets. A lifecycleStateChange facet of DROP retires the dataset's connections, and RENAME retires the connections of the name it had before. Evidence asserted after the lifecycle event is preserved.

A retired connection leaves the graph but stays queryable, with the time it was retired and the event that retired it, so the history of a relationship can be audited. If a job asserts it again, the same connection returns.

Deleting a connection from OpenLineage in Qualytics also retires it rather than removing it, recording the user who deleted it. A later event that asserts it again brings it back.

Run States¶

A JobEvent is authoritative for its job in the same way as a COMPLETE RunEvent. Deliver a run's START and RUNNING events before its COMPLETE: an earlier event received after that COMPLETE does not retroactively add to it.

Retirement only ever affects connections from OpenLineage. Manual, data catalog, source collection, and Qualytics managed connections are never touched, and a data catalog sync or a source collection never retires a connection from OpenLineage.

Dataset Metadata¶

Events can also describe the datasets themselves. Qualytics applies these facets to containers and fields that are already in Qualytics. OpenLineage never adds a container: that is what a Sync does.

Which Value Is Shown¶

A description can be set by more than one source: a user, a Sync, a data catalog, and OpenLineage. Qualytics keeps each source's value and shows one by a single rule:

1. A user's edit always wins. A description you type is never replaced by a source. Clear it to hand the description back to the sources.
2. Otherwise, the value that changed most recently wins. A source sending the same value again does not count as a change, so a Sync re-reading an unchanged column comment does not replace a newer description from OpenLineage. If the comment in the source does change, the Sync's new value is the most recent and is shown.

An existing container description is preserved as a user's. An existing field description counts as Sync's only while it matches the last source comment Sync recorded; otherwise it is preserved as a user's edit.

Older metadata events cannot replace a newer report, even when that newer report repeated an unchanged value.

This precedence is fixed; there is no configurable authority policy by attribute or source. Declaring a producer authoritative does not make its description override a user edit or a newer changed catalog description.

Values are kept by source category. All OpenLineage producers share one current value for each asset and attribute, so two independent producers do not retain separate competing descriptions or owner lists. The producer URI identifies who supplied the current OpenLineage value. Description alternatives from the user, Sync and each catalog category remain available through the source display and API.

Owner names supplied by OpenLineage are descriptive metadata. They do not assign a Qualytics user as the asset's owner or grant any role or team permission. The Owners line shows the effective list; there is no ownership conflict editor or comparison display.

Withdrawing Metadata¶

A facet sent as {"_deleted": true}, the spec's way to delete a facet, withdraws what OpenLineage asserted with it:

• A deleted documentation facet withdraws OpenLineage's description. The next most recent source's description is shown instead.
• A deleted ownership facet withdraws the owners OpenLineage sent.
• A deleted tags facet removes the tags OpenLineage added. Tags you added stay.

A deleted columnLineage, lifecycleStateChange, or symlinks facet is treated as if it were absent, so the rest of the event still applies.

Tags follow the same idea, one tag at a time. A tag the producer starts sending is added. A tag it stops sending is removed only if OpenLineage added it. A tag you removed stays removed while the producer keeps sending the same set, and a tag you added is never removed by OpenLineage.

A change an event makes appears in the container's timeline as co-authored by the producer that sent it, with the event's number, beside the user whose API token sent the event. Container timelines and metadata-source responses omit the job and run identity; read those through the connection's evidence, which checks Viewer permission on every endpoint.

On a container or a field, a mark beside the Description title provides the recorded non-user source history and identifies the effective source when one exists. If every source has withdrawn its description, a neutral history button still opens the withdrawn values, and none is marked Shown. Hover or keyboard-focus the mark to see each recorded source category's description, when it last changed, and which one is shown. Owners an OpenLineage producer sent appear as an Owners line on the container.

Each source's value, with when it last changed and which one is shown, is listed by the container's metadata assertions in the API.

See Also¶