Amazon S3 API

This page documents S3-specific payload fields for the datastore API. For generic operations (get, list, update, delete, link enrichment), see Datastore API.

Complete API Reference

For the full interactive API documentation with all request/response schemas, visit the API docs.

S3 supports two authentication types (Access Key by default, or IAM Role) and can be created either as a source datastore (read-only) or as an enrichment datastore (read-write). You can create a datastore with a new inline connection or reuse an existing connection. All flows use the same endpoint:

Endpoint: POST /api/datastores

Permission: Manager

UI ↔ API field mapping

The API names a few fields differently from the UI. The mapping below avoids the most common copy-paste mistakes when switching from the form to the JSON payload.

UI form field	API field	Notes	Example
Connection > Connection Name	`connection.name`	Saved-connection label. Optional.	`acme_s3_lake`
(set by connector choice)	`connection.type`	Connector identifier. Always `"s3"` for Amazon S3.	`s3`
Connection > URI	`connection.uri`	Bucket-level S3 URI.	`s3://my-bucket`
Datastores Extraction > Root Path	`root_path`	Subfolder inside the bucket.	`/raw/orders/`
Connection > Type	`connection.parameters.authentication_type`	`SHARED_KEY` (Access Key) or `IAM_ROLE`. Defaults to `SHARED_KEY` when omitted.	`SHARED_KEY`
Connection > Access Key	`connection.access_key`	AWS Access Key ID. Required only when `authentication_type` is `SHARED_KEY`.	`AKIA…`
Connection > Secret Key	`connection.secret_key`	AWS Secret Access Key. Required only when `authentication_type` is `SHARED_KEY`.	`wJalrXUt…`
Connection > Role ARN	`connection.parameters.role_arn`	Target role for STS AssumeRole. Required when `authentication_type` is `IAM_ROLE`.	`arn:aws:iam::123456789012:role/MyRole`
Connection > External ID	`connection.parameters.external_id`	Optional external ID enforced by the role's trust policy.	`my-external-id`
Datastore > Initiate Sync	`trigger_catalog`	When `true`, syncs metadata immediately after creation.	`true`

When authentication_type is IAM_ROLE, omit access_key and secret_key from the payload (both fields are nullable). Qualytics obtains short-lived credentials via STS instead of using static keys.

Connection Fields

Property	Type	Required	Default	Description
`uri`	string	Yes		Bucket-level S3 URI in the format `s3://<bucket-name>`.
`root_path`	string	Yes	`/`	Subfolder inside the bucket where the data lives (source) or where Qualytics writes enrichment files (enrichment).
`parameters.authentication_type`	string	No	`SHARED_KEY`	`SHARED_KEY` (Access Key) or `IAM_ROLE`.
`parameters.role_arn`	string	If `IAM_ROLE`		IAM role ARN to assume via AWS STS.
`parameters.external_id`	string	No		External ID for STS AssumeRole. Include only when your trust policy requires one.
`access_key`	string	If `SHARED_KEY`		AWS Access Key ID.
`secret_key`	string	If `SHARED_KEY`		AWS Secret Access Key.

About trigger_catalog

The trigger_catalog: true flag in the examples below tells Qualytics to immediately sync metadata after creating the datastore (equivalent to ticking the Initiate Sync toggle in the UI). Set it to false if you want to defer the first sync.

Create a Source Datastore

A source datastore uses a read-only IAM identity. Set enrichment_only: false to mark the datastore as a source.

Access KeyIAM RoleExisting Connection

The default authentication type. Use when you can provide static AWS credentials with read permissions on the bucket.

Example request and response

Request:

curl -X POST "https://your-instance.qualytics.io/api/datastores" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Orders Lake",
    "teams": ["Data Platform"],
    "root_path": "/raw/orders/",
    "enrichment_only": false,
    "trigger_catalog": true,
    "connection": {
      "name": "acme_s3_lake",
      "type": "s3",
      "uri": "s3://acme-data-lake",
      "access_key": "AKIAIOSFODNN7EXAMPLE",
      "secret_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLE",
      "parameters": {
        "authentication_type": "SHARED_KEY"
      }
    }
  }'

Response (200 OK):

{
  "id": 42,
  "name": "Orders Lake",
  "type": "s3",
  "store_type": "dfs",
  "root_path": "/raw/orders/",
  "enrichment_only": false,
  "connected": true
}

Authenticate via AWS STS by assuming an IAM role. Qualytics uses short-lived credentials that refresh automatically before they expire. Set parameters.authentication_type to "IAM_ROLE" and provide the role ARN. Omit access_key and secret_key. They are not used in this mode. The external_id field is optional and only required if your trust policy enforces it.

Example request and response

Request:

curl -X POST "https://your-instance.qualytics.io/api/datastores" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Orders Lake",
    "teams": ["Data Platform"],
    "root_path": "/raw/orders/",
    "enrichment_only": false,
    "trigger_catalog": true,
    "connection": {
      "name": "acme_s3_lake",
      "type": "s3",
      "uri": "s3://acme-data-lake",
      "parameters": {
        "authentication_type": "IAM_ROLE",
        "role_arn": "arn:aws:iam::123456789012:role/QualyticsS3Role",
        "external_id": "<your-external-id>"
      }
    }
  }'

Response (200 OK):

{
  "id": 42,
  "name": "Orders Lake",
  "type": "s3",
  "store_type": "dfs",
  "root_path": "/raw/orders/",
  "enrichment_only": false,
  "connected": true
}

Note

See Authentication and IAM Role Authentication for the trust policy setup and per-deployment base role details.

Reuse a previously created S3 connection by passing its connection_id. The authentication type and credentials are reused from the existing connection. You only specify datastore-level fields like name, teams, and root_path.

Example request and response

Request:

curl -X POST "https://your-instance.qualytics.io/api/datastores" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Finance Lake",
    "teams": ["Finance Team"],
    "root_path": "/finance/",
    "enrichment_only": false,
    "trigger_catalog": true,
    "connection_id": 7
  }'

Response (200 OK):

{
  "id": 43,
  "name": "Finance Lake",
  "type": "s3",
  "store_type": "dfs",
  "root_path": "/finance/",
  "enrichment_only": false,
  "connected": true
}

Create an Enrichment Datastore

An enrichment datastore uses a read-write IAM identity. Set enrichment_only: true to mark the datastore as an enrichment store.

Access KeyIAM RoleExisting Connection

Example request and response

Request:

curl -X POST "https://your-instance.qualytics.io/api/datastores" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Lake Enrichment",
    "teams": ["Data Platform"],
    "root_path": "/qualytics-enrichment/",
    "enrichment_only": true,
    "connection": {
      "name": "acme_s3_enrichment",
      "type": "s3",
      "uri": "s3://acme-data-lake-enrichment",
      "access_key": "AKIAIOSFODNN7EXAMPLE",
      "secret_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLE",
      "parameters": {
        "authentication_type": "SHARED_KEY"
      }
    }
  }'

Response (200 OK):

{
  "id": 44,
  "name": "Lake Enrichment",
  "type": "s3",
  "store_type": "dfs",
  "root_path": "/qualytics-enrichment/",
  "enrichment_only": true,
  "connected": true
}

Set parameters.authentication_type to "IAM_ROLE" and provide a role ARN whose attached policy grants the read and write S3 permissions from Enrichment Permissions.

Example request and response

Request:

curl -X POST "https://your-instance.qualytics.io/api/datastores" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Lake Enrichment",
    "teams": ["Data Platform"],
    "root_path": "/qualytics-enrichment/",
    "enrichment_only": true,
    "connection": {
      "name": "acme_s3_enrichment",
      "type": "s3",
      "uri": "s3://acme-data-lake-enrichment",
      "parameters": {
        "authentication_type": "IAM_ROLE",
        "role_arn": "arn:aws:iam::123456789012:role/QualyticsS3EnrichmentRole",
        "external_id": "<your-external-id>"
      }
    }
  }'

Response (200 OK):

{
  "id": 44,
  "name": "Lake Enrichment",
  "type": "s3",
  "store_type": "dfs",
  "root_path": "/qualytics-enrichment/",
  "enrichment_only": true,
  "connected": true
}

Enrichment IAM Role permissions

The assumed role needs read and write permissions on the bucket (s3:PutObject, s3:DeleteObject, s3:AbortMultipartUpload, and so on). See Enrichment Permissions for the full IAM policy.

Reuse a previously created S3 connection that was configured for read-write access. Pass connection_id and set enrichment_only: true.

Example request and response

Request:

curl -X POST "https://your-instance.qualytics.io/api/datastores" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Lake Enrichment",
    "teams": ["Data Platform"],
    "root_path": "/qualytics-enrichment/",
    "enrichment_only": true,
    "connection_id": 9
  }'

Response (200 OK):

{
  "id": 45,
  "name": "Lake Enrichment",
  "type": "s3",
  "store_type": "dfs",
  "root_path": "/qualytics-enrichment/",
  "enrichment_only": true,
  "connected": true
}

Other Operations

For get, list, update, delete, test connection, and link-enrichment endpoints, see the Datastore API reference.