Azure Data Lake Storage API

This page documents Azure Data Lake Storage Gen2-specific payload fields for the datastore API. For generic operations (get, list, update, delete, link enrichment), see Datastore API.

Complete API Reference

For the full interactive API documentation with all request/response schemas, visit the API docs.

Azure Data Lake Storage Gen2 supports two authentication types (Shared Key by default, or Service Principal) and can be created either as a source datastore (read-only) or as an enrichment datastore (read-write). All flows use the same endpoint:

Endpoint: POST /api/datastores

Permission: Manager

UI ↔ API field mapping

The API names a few fields differently from the UI. The mapping below avoids the most common copy-paste mistakes when switching from the form to the JSON payload.

UI form field	API field	Notes	Example
Connection > Connection Name	`connection.name`	Saved-connection label. Optional.	`acme_adls_lake`
(set by connector choice)	`connection.type`	Connector identifier. Always `"abfs"` for Azure Data Lake Storage Gen2.	`abfs`
Connection > URI	`connection.uri`	Container-level ABFS URI.	`abfss://raw@acmelake.dfs.core.windows.net`
Datastores Extraction > Root Path	`root_path`	Subfolder inside the container.	`/raw/orders/`
Connection > Type	`connection.parameters.authentication_type`	`SHARED_KEY` (Shared Key) or `SERVICE_PRINCIPAL`. Defaults to `SHARED_KEY` when omitted.	`SHARED_KEY`
Connection > Account Name (Shared Key) / Client ID (Service Principal)	`connection.access_key`	UI label changes with `authentication_type`. The API field name stays `access_key`.	`acmelake` or `00000000-0000-0000-0000-000000000000`
Connection > Access Key (Shared Key) / Client Secret (Service Principal)	`connection.secret_key`	UI label changes with `authentication_type`. The API field name stays `secret_key`.	`<storage-account-access-key>` or `<client-secret>`
Connection > Tenant ID	`connection.parameters.tenant_id`	Required only when `authentication_type` is `SERVICE_PRINCIPAL`.	`11111111-1111-1111-1111-111111111111`
Datastore > Initiate Sync	`trigger_catalog`	When `true`, syncs metadata immediately after creation.	`true`

Connection Fields

Property	Type	Required	Default	Description
`name`	string	Yes		Datastore label shown in the UI.
`teams`	array of strings	Yes		Teams that can access the datastore.
`root_path`	string	Yes	`/`	Subfolder inside the container where the data lives (source) or where Qualytics writes enrichment files (enrichment).
`enrichment_only`	boolean	Yes		`false` for a source datastore. `true` for an enrichment datastore.
`connection.name`	string	Yes		Saved-connection label.
`connection.type`	string	Yes		Connector identifier. Always `abfs` for Azure Data Lake Storage Gen2.
`connection.uri`	string	Yes		Container-level ABFS URI in the format `abfss://<container>@<account>.dfs.core.windows.net`.
`connection.access_key`	string	Yes		Account Name when `authentication_type` is `SHARED_KEY`. Client ID when `authentication_type` is `SERVICE_PRINCIPAL`.
`connection.secret_key`	string	Yes		Access Key when `authentication_type` is `SHARED_KEY`. Client Secret when `authentication_type` is `SERVICE_PRINCIPAL`.
`connection.parameters.authentication_type`	string	No	`SHARED_KEY`	`SHARED_KEY` (Shared Key) or `SERVICE_PRINCIPAL`.
`connection.parameters.tenant_id`	string	If `SERVICE_PRINCIPAL`		Directory (Tenant) ID of the Microsoft Entra ID app registration.

About trigger_catalog

The trigger_catalog: true flag in the examples below tells Qualytics to immediately sync metadata after creating the datastore (equivalent to ticking the Initiate Sync toggle in the UI). Set it to false if you want to defer the first sync.

Create a Source Datastore

A source datastore uses a read-only identity. Set enrichment_only: false to mark the datastore as a source.

Shared Key (SHARED_KEY)Service Principal (SERVICE_PRINCIPAL)Existing Connection

The default authentication type. Use when you can provide the storage account name and its access key.

Example request and response

Request:

curl -X POST "https://your-instance.qualytics.io/api/datastores" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Orders Lake",
    "teams": ["Engineering"],
    "root_path": "/",
    "enrichment_only": false,
    "trigger_catalog": true,
    "connection": {
      "name": "acme_adls_lake",
      "type": "abfs",
      "uri": "abfss://raw@acmelake.dfs.core.windows.net",
      "access_key": "acmelake",
      "secret_key": "<storage-account-access-key>",
      "parameters": {
        "authentication_type": "SHARED_KEY"
      }
    }
  }'

Response (200 OK):

{
  "id": 42,
  "name": "Orders Lake",
  "type": "abfs",
  "store_type": "dfs",
  "root_path": "/",
  "enrichment_only": false,
  "connected": true
}

Authenticate via a Microsoft Entra ID app registration. Set parameters.authentication_type to "SERVICE_PRINCIPAL" and provide the Client ID, Client Secret, and Tenant ID.

Example request and response

Request:

curl -X POST "https://your-instance.qualytics.io/api/datastores" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Orders Lake",
    "teams": ["Engineering"],
    "root_path": "/",
    "enrichment_only": false,
    "trigger_catalog": true,
    "connection": {
      "name": "acme_adls_lake",
      "type": "abfs",
      "uri": "abfss://raw@acmelake.dfs.core.windows.net",
      "access_key": "00000000-0000-0000-0000-000000000000",
      "secret_key": "<client-secret>",
      "parameters": {
        "authentication_type": "SERVICE_PRINCIPAL",
        "tenant_id": "11111111-1111-1111-1111-111111111111"
      }
    }
  }'

Response (200 OK):

{
  "id": 42,
  "name": "Orders Lake",
  "type": "abfs",
  "store_type": "dfs",
  "root_path": "/",
  "enrichment_only": false,
  "connected": true
}

Reuse a previously created Azure Data Lake Storage connection by passing its connection_id. The authentication type and credentials are reused from the existing connection. You only specify datastore-level fields like name, teams, and root_path.

Example request and response

Request:

curl -X POST "https://your-instance.qualytics.io/api/datastores" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Finance Lake",
    "teams": ["Finance Team"],
    "root_path": "/finance/",
    "enrichment_only": false,
    "trigger_catalog": true,
    "connection_id": 7
  }'

Response (200 OK):

{
  "id": 43,
  "name": "Finance Lake",
  "type": "abfs",
  "store_type": "dfs",
  "root_path": "/finance/",
  "enrichment_only": false,
  "connected": true
}

Create an Enrichment Datastore

An enrichment datastore uses a read-write identity. Set enrichment_only: true to mark the datastore as an enrichment store.

Shared Key (SHARED_KEY)Service Principal (SERVICE_PRINCIPAL)Existing Connection

Provide the storage account name and its access key. The account must have write permissions on the container where Qualytics writes enrichment files.

Example request and response

Request:

curl -X POST "https://your-instance.qualytics.io/api/datastores" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Lake Enrichment",
    "teams": ["Engineering"],
    "root_path": "/qualytics-enrichment/",
    "enrichment_only": true,
    "trigger_catalog": true,
    "connection": {
      "name": "acme_adls_enrichment",
      "type": "abfs",
      "uri": "abfss://raw@acmelake.dfs.core.windows.net",
      "access_key": "acmelake",
      "secret_key": "<storage-account-access-key>",
      "parameters": {
        "authentication_type": "SHARED_KEY"
      }
    }
  }'

Response (200 OK):

{
  "id": 44,
  "name": "Lake Enrichment",
  "type": "abfs",
  "store_type": "dfs",
  "root_path": "/qualytics-enrichment/",
  "enrichment_only": true,
  "connected": true
}

Use a Microsoft Entra ID app registration whose role assignment grants read and write access on the container (for example, Storage Blob Data Contributor). Set parameters.authentication_type to "SERVICE_PRINCIPAL" and provide the Client ID, Client Secret, and Tenant ID.

Example request and response

Request:

curl -X POST "https://your-instance.qualytics.io/api/datastores" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Lake Enrichment",
    "teams": ["Engineering"],
    "root_path": "/qualytics-enrichment/",
    "enrichment_only": true,
    "trigger_catalog": true,
    "connection": {
      "name": "acme_adls_enrichment",
      "type": "abfs",
      "uri": "abfss://raw@acmelake.dfs.core.windows.net",
      "access_key": "00000000-0000-0000-0000-000000000000",
      "secret_key": "<client-secret>",
      "parameters": {
        "authentication_type": "SERVICE_PRINCIPAL",
        "tenant_id": "11111111-1111-1111-1111-111111111111"
      }
    }
  }'

Response (200 OK):

{
  "id": 44,
  "name": "Lake Enrichment",
  "type": "abfs",
  "store_type": "dfs",
  "root_path": "/qualytics-enrichment/",
  "enrichment_only": true,
  "connected": true
}

Reuse a previously created Azure Data Lake Storage connection by passing its connection_id. The authentication type and credentials are reused from the existing connection. You only specify datastore-level fields like name, teams, and root_path.

Example request and response

Request:

curl -X POST "https://your-instance.qualytics.io/api/datastores" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Finance Enrichment",
    "teams": ["Finance Team"],
    "root_path": "/qualytics-enrichment/",
    "enrichment_only": true,
    "trigger_catalog": true,
    "connection_id": 9
  }'

Response (200 OK):

{
  "id": 45,
  "name": "Finance Enrichment",
  "type": "abfs",
  "store_type": "dfs",
  "root_path": "/qualytics-enrichment/",
  "enrichment_only": true,
  "connected": true
}

Other Operations

For get, list, update, delete, test connection, and link-enrichment endpoints, see the Datastore API reference.