Add an Azure Data Lake Storage Source Datastore
A source datastore is a storage location Qualytics connects to so it can profile, scan, and monitor data. Adding Azure Data Lake Storage as a source lets Qualytics read files directly from your container and run quality operations on the data they contain.
Before you start, review the Azure Data Lake Storage Permissions and the available Authentication methods.
Tip
Azure Data Lake Storage can also hold an enrichment datastore, where Qualytics writes anomalies and metadata. Link one during creation via Link Enrichment on Datastore Creation, or afterwards via Link Enrichment Datastore.
Field reference
The Add Datastore page shows the sections below when Azure Data Lake Storage is selected. When reusing an existing connection, the Connection Properties, Authentication, and Secrets Management sections come already filled in and read-only: Qualytics has already validated those credentials, so you fill in only the Datastores Extraction and the Datastore Properties. To change a saved connection's credentials, edit the connection through the Manage Connections page; edits there apply to every datastore that reuses the connection.
Connection Properties
These fields define where the container lives. They belong to the connection: when reusing an existing connection, they come already filled in and read-only.
| FIELD | REQUIRED | TYPE | DESCRIPTION |
|---|---|---|---|
| Connection Name | Text | A label for the saved connection (e.g., acme_adls_lake), so other datastores can reuse it later. |
|
| URI | Text | The container-level URI, in the form abfss://<container>@<account_name>.dfs.core.windows.net. Leave out any folder path here and use Root Path below to scope to a subfolder. |
Authentication
Choose how Qualytics authenticates to Azure. Setting Type changes the credential fields shown below it, so pick the tab that matches your choice. These fields also belong to the connection: already filled in and read-only when reusing one. See Authentication for how each method works.
| FIELD | REQUIRED | TYPE | DESCRIPTION |
|---|---|---|---|
| Type | Option | Set to Shared Key, which is the default (SHARED_KEY in the API). |
|
| Account Name | Text | The storage account the container belongs to. Sent as the connection's access_key in the API. |
|
| Access Key | Text | One of the storage account's two access keys. Sent as the connection's secret_key in the API. |
| FIELD | REQUIRED | TYPE | DESCRIPTION |
|---|---|---|---|
| Type | Option | Set to Service Principal to authenticate through Microsoft Entra ID (SERVICE_PRINCIPAL in the API). |
|
| Client ID | Text | The application (client) ID of the Microsoft Entra ID app registration. Sent as the connection's access_key in the API. |
|
| Client Secret | Text | The client secret generated for that app registration. Sent as the connection's secret_key in the API. |
|
| Tenant ID | Text | The Microsoft Entra ID tenant the app registration belongs to. |
Prerequisites
The app registration needs a role on the storage account that allows reading the container, such as Storage Blob Data Reader. An enrichment datastore needs write access as well.
Secrets Management
This group is optional: use it only if you want Qualytics to pull credentials from a secrets manager instead of typing them into the form. Turn on HashiCorp Vault to show the fields below. Despite the label, any secrets manager that exposes a compatible REST API works, not only HashiCorp Vault; see Secrets Management. It also belongs to the connection: read-only when reusing an existing connection.
| FIELD | REQUIRED | TYPE | DESCRIPTION |
|---|---|---|---|
| Login URL | Text | The Vault endpoint Qualytics uses to authenticate (e.g., https://vault.example.com/v1/auth/approle/login). |
|
| Credentials Payload | Text | A JSON body containing the credentials Vault expects (e.g., {"role_id":"...","secret_id":"..."}). |
|
| Token JSONPath | Text | The JSONPath that extracts the client token from Vault's response. Defaults to $.auth.client_token. |
|
| Secret URL | Text | The Vault path where the secret is stored (e.g., https://vault.example.com/v1/secret/data/adls). |
|
| Token Header Name | Text | The HTTP header name used to send the token. Defaults to X-Vault-Token. |
|
| Data JSONPath | Text | The JSONPath that extracts the secret payload from Vault's response. Defaults to $.data. |
Note
Once the secrets manager is configured, reference any secret from a Connection Properties or Authentication field with ${key}, for example ${secret_key}. Qualytics resolves it each time the connection is opened, so a changed value takes effect on the next connection.
Datastores Extraction
Pick the folder inside the container Qualytics should read from. You fill this in on both flows.
| FIELD | REQUIRED | TYPE | DESCRIPTION |
|---|---|---|---|
| Root Path | Option | The folder inside the container where the data lives (e.g., /raw/orders/). Defaults to /, which reads from the container root. Click the refresh icon to load the folders the credentials can see. Each datastore sets its own root path, so reusing a connection does not tie you to the previous one. |
Datastore Properties
Common fields for every source datastore, shown below the Datastores Extraction section. You fill these in on both flows.
| FIELD | REQUIRED | TYPE | DESCRIPTION |
|---|---|---|---|
| Name Template | Text | Defines the naming pattern for the source datastore being created. Left empty, the datastore is named from the connection name and the root path. | |
| Group | Option | Organizes your datastores under a shared group in the navigation tree. Select an existing group or create a new one with the Add New Group toggle. | |
| Teams | Option | Select one or more teams to associate with this source datastore. | |
| Initiate Sync | Checkbox | Automatically sync the datastore to detect containers and fields after creation. |
Allowlisting
A banner at the bottom of the form shows the IP address the connection is established from. Allowlist it in your storage account's network rules so the connection can be opened.
Steps
There are two ways to set up the connection: reuse a connection you already saved (Existing Connection) or create a new one from scratch (New Connection). The tabs below walk through each option; pick the one you want to follow. Each field is described in the Field reference above.
Step 1: Navigate to the Source Datastores page.
Step 2: Click the Add Source Datastore button at the top-right corner.
Step 3: The Add Datastore page opens.
Step 4: Select New Connection next to the Search field.
Step 5: Select Azure Datalake Storage from the connector grid. Use the search field to filter connectors by name.
Step 6: Fill in the Connection Properties: the Connection Name and the URI.
Step 7: Fill in the Authentication fields for the Type you choose.
Step 8: Optionally, expand Secrets Management to retrieve credentials from a secrets manager.
Step 9: Fill in the Datastores Extraction field (Root Path) and the Datastore Properties.
Step 10: Click Test connection. A success message confirms that the connection has been verified.
Info
The Finish and Next buttons stay disabled until the connection test passes on the current values. If the test fails, see Troubleshooting.
Step 11: Click Finish to create the datastore.
Tip
To link an enrichment datastore so Qualytics can store anomalies and metadata from the first operation, click Next instead of Finish. See Link Enrichment on Datastore Creation.
Step 12: A success dialog confirms that your datastore has been added. Click Go to your datastore to open its page.
Step 1: Navigate to the Source Datastores page.
Step 2: Click the Add Source Datastore button at the top-right corner.
Step 3: The Add Datastore page opens.
Step 4: Select Existing Connection next to the Search field.
Step 5: Select the saved Azure Data Lake Storage connection from the grid. Use the search field to filter connections by name. The Connection Properties, Authentication, and Secrets Management sections come already filled in and read-only.
Start a new connection from this one
To use the selected connection as a starting point for a brand-new connection instead, click the Duplicate as a new connection button on the selected connection. The form switches to New Connection mode with the connection's settings already filled in for you to adjust.
Step 6: Fill in the Datastores Extraction field (Root Path) and the Datastore Properties. These are the only fields left to fill in.
Step 7: Click Test connection. A success message confirms that the connection has been verified.
Info
The Finish and Next buttons stay disabled until the connection test passes on the current values. If the test fails, see Troubleshooting.
Step 8: Click Finish to create the datastore.
Tip
To link an enrichment datastore so Qualytics can store anomalies and metadata from the first operation, click Next instead of Finish. See Link Enrichment on Datastore Creation.
Step 9: A success dialog confirms that your datastore has been added. Click Go to your datastore to open its page.