# Amazon S3 Amazon S3 API

Source: https://userguide.qualytics.io/source-datastore/datastore/dfs/amazon-s3/how-tos/api/

This page documents S3-specific payload fields for the datastore API. For generic operations (get, list, update, delete, link enrichment), see [Datastore API](https://userguide.qualytics.io/source-datastore/datastore/api/).

> **Tip: Complete API Reference**
>
> For the full interactive API documentation with all request/response schemas, visit the [API docs](https://userguide.qualytics.io/api/docs).

S3 supports two authentication types (**Access Key** by default, or **IAM Role**) and can be created as a source datastore (read-only) or an enrichment datastore (read-write). You can create a datastore with a new inline connection or reuse an existing connection. All flows use the same endpoint:

**Endpoint**: `POST /api/datastores`

**Permission**: Manager, plus the **Editor** team permission on at least one of the assigned teams (the **Public** team when `teams` is left out). Admins are exempt.

---

## UI ↔ API field mapping

The API names a few fields differently from the UI. The mapping below avoids the most common copy-paste mistakes when switching from the form to the JSON payload.

| UI form field | API field | Notes | Example |
| --- | --- | --- | --- |
| **Connection > Connection Name** | `connection.name` | Saved-connection label. Optional. | `acme_s3_lake` |
| (set by connector choice) | `connection.type` | Connector identifier. Always `"s3"` for Amazon S3. | `s3` |
| **Connection > URI** | `connection.uri` | Bucket-level S3 URI. | `s3://my-bucket` |
| **Location > Root Path** | `root_path` | Subfolder inside the bucket. | `/raw/orders/` |
| **Connection > Type** | `connection.parameters.authentication_type` | `SHARED_KEY` (Access Key) or `IAM_ROLE`. Defaults to `SHARED_KEY` when omitted. | `SHARED_KEY` |
| **Connection > Access Key** | `connection.access_key` | AWS Access Key ID. Required only when `authentication_type` is `SHARED_KEY`. | `AKIA…` |
| **Connection > Secret Key** | `connection.secret_key` | AWS Secret Access Key. Required only when `authentication_type` is `SHARED_KEY`. | `wJalrXUt…` |
| **Connection > Role ARN** | `connection.parameters.role_arn` | Target role for STS AssumeRole. Required when `authentication_type` is `IAM_ROLE`. | `arn:aws:iam::123456789012:role/MyRole` |
| **Connection > External ID** | `connection.parameters.external_id` | Optional external ID enforced by the role's trust policy. | `my-external-id` |
| **Datastore > Initiate Sync** | `trigger_sync` | When `true`, syncs metadata immediately after creation. | `true` |

When `authentication_type` is `IAM_ROLE`, omit `access_key` and `secret_key` from the payload (both fields are nullable). Qualytics obtains short-lived credentials via STS instead of using static keys.

## Connection Fields

| Property | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `uri` | string | Yes |  | Bucket-level S3 URI in the format `s3://<bucket-name>`. |
| `root_path` | string | Yes | `/` | Subfolder inside the bucket where the data lives (source) or where Qualytics writes its outputs (enrichment). |
| `parameters.authentication_type` | string | No | `SHARED_KEY` | `SHARED_KEY` (Access Key) or `IAM_ROLE`. |
| `parameters.role_arn` | string | If `IAM_ROLE` |  | IAM role ARN to assume via AWS STS. |
| `parameters.external_id` | string | No |  | External ID for STS AssumeRole. Include only when your trust policy requires one. |
| `access_key` | string | If `SHARED_KEY` |  | AWS Access Key ID. |
| `secret_key` | string | If `SHARED_KEY` |  | AWS Secret Access Key. |

> **Note: About trigger_sync**
>
> The `trigger_sync: true` flag in the examples below tells Qualytics to immediately sync metadata after creating the datastore (equivalent to ticking the **Initiate Sync** toggle in the UI). Set it to `false` if you want to defer the first sync.

---

## Create a Source Datastore

A source datastore uses a read-only IAM identity. Set `enrichment_only: false` to mark the datastore as a source.

**Tab: Access Key**

The default authentication type. Use when you can provide static AWS credentials with read permissions on the bucket.

> **Example: Example request and response**
>
> **Request**:
>
> ```bash
> curl -X POST "https://your-instance.qualytics.io/api/datastores" \
>   -H "Authorization: Bearer YOUR_TOKEN" \
>   -H "Content-Type: application/json" \
>   -d '{
>     "name": "Orders Lake",
>     "teams": ["Data Platform"],
>     "root_path": "/raw/orders/",
>     "enrichment_only": false,
>     "trigger_sync": true,
>     "connection": {
>       "name": "acme_s3_lake",
>       "type": "s3",
>       "uri": "s3://acme-data-lake",
>       "access_key": "AKIAIOSFODNN7EXAMPLE",
>       "secret_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLE",
>       "parameters": {
>         "authentication_type": "SHARED_KEY"
>       }
>     }
>   }'
> ```
>
> **Response** (200 OK):
>
> ```json
> {
>   "id": 42,
>   "name": "Orders Lake",
>   "type": "s3",
>   "store_type": "dfs",
>   "root_path": "/raw/orders/",
>   "enrichment_only": false,
>   "connected": true
> }
> ```

**Tab: IAM Role**

Authenticate via AWS STS by assuming an IAM role. Qualytics uses short-lived credentials that refresh automatically before they expire. Set `parameters.authentication_type` to `"IAM_ROLE"` and provide the role ARN. Omit `access_key` and `secret_key`. They are not used in this mode. The `external_id` field is optional and only required if your trust policy enforces it.

> **Example: Example request and response**
>
> **Request**:
>
> ```bash
> curl -X POST "https://your-instance.qualytics.io/api/datastores" \
>   -H "Authorization: Bearer YOUR_TOKEN" \
>   -H "Content-Type: application/json" \
>   -d '{
>     "name": "Orders Lake",
>     "teams": ["Data Platform"],
>     "root_path": "/raw/orders/",
>     "enrichment_only": false,
>     "trigger_sync": true,
>     "connection": {
>       "name": "acme_s3_lake",
>       "type": "s3",
>       "uri": "s3://acme-data-lake",
>       "parameters": {
>         "authentication_type": "IAM_ROLE",
>         "role_arn": "arn:aws:iam::123456789012:role/QualyticsS3Role",
>         "external_id": "<your-external-id>"
>       }
>     }
>   }'
> ```
>
> **Response** (200 OK):
>
> ```json
> {
>   "id": 42,
>   "name": "Orders Lake",
>   "type": "s3",
>   "store_type": "dfs",
>   "root_path": "/raw/orders/",
>   "enrichment_only": false,
>   "connected": true
> }
> ```

> **Note**
>
> See [Authentication](https://userguide.qualytics.io/source-datastore/datastore/dfs/amazon-s3/authentication/) and [IAM Role Authentication](https://userguide.qualytics.io/deployments/iam-role-authentication/) for the trust policy setup and per-deployment base role details.

**Tab: Existing Connection**

Reuse a previously created S3 connection by passing its `connection_id`. The authentication type and credentials are reused from the existing connection. You only specify datastore-level fields like `name`, `teams`, and `root_path`.

> **Example: Example request and response**
>
> **Request**:
>
> ```bash
> curl -X POST "https://your-instance.qualytics.io/api/datastores" \
>   -H "Authorization: Bearer YOUR_TOKEN" \
>   -H "Content-Type: application/json" \
>   -d '{
>     "name": "Finance Lake",
>     "teams": ["Finance Team"],
>     "root_path": "/finance/",
>     "enrichment_only": false,
>     "trigger_sync": true,
>     "connection_id": 7
>   }'
> ```
>
> **Response** (200 OK):
>
> ```json
> {
>   "id": 43,
>   "name": "Finance Lake",
>   "type": "s3",
>   "store_type": "dfs",
>   "root_path": "/finance/",
>   "enrichment_only": false,
>   "connected": true
> }
> ```

---

## Create an Enrichment Datastore

An enrichment datastore uses a read-write IAM identity. Set `enrichment_only: true` to create the datastore for enrichment.

**Tab: Access Key**

> **Example: Example request and response**
>
> **Request**:
>
> ```bash
> curl -X POST "https://your-instance.qualytics.io/api/datastores" \
>   -H "Authorization: Bearer YOUR_TOKEN" \
>   -H "Content-Type: application/json" \
>   -d '{
>     "name": "Lake Enrichment",
>     "teams": ["Data Platform"],
>     "root_path": "/qualytics-enrichment/",
>     "enrichment_only": true,
>     "connection": {
>       "name": "acme_s3_enrichment",
>       "type": "s3",
>       "uri": "s3://acme-data-lake-enrichment",
>       "access_key": "AKIAIOSFODNN7EXAMPLE",
>       "secret_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLE",
>       "parameters": {
>         "authentication_type": "SHARED_KEY"
>       }
>     }
>   }'
> ```
>
> **Response** (200 OK):
>
> ```json
> {
>   "id": 44,
>   "name": "Lake Enrichment",
>   "type": "s3",
>   "store_type": "dfs",
>   "root_path": "/qualytics-enrichment/",
>   "enrichment_only": true,
>   "connected": true
> }
> ```

**Tab: IAM Role**

Set `parameters.authentication_type` to `"IAM_ROLE"` and provide a role ARN whose attached policy grants the read **and** write S3 permissions from [Enrichment Permissions](https://userguide.qualytics.io/source-datastore/datastore/dfs/amazon-s3/permissions/#enrichment-datastore-permissions-read-write).

> **Example: Example request and response**
>
> **Request**:
>
> ```bash
> curl -X POST "https://your-instance.qualytics.io/api/datastores" \
>   -H "Authorization: Bearer YOUR_TOKEN" \
>   -H "Content-Type: application/json" \
>   -d '{
>     "name": "Lake Enrichment",
>     "teams": ["Data Platform"],
>     "root_path": "/qualytics-enrichment/",
>     "enrichment_only": true,
>     "connection": {
>       "name": "acme_s3_enrichment",
>       "type": "s3",
>       "uri": "s3://acme-data-lake-enrichment",
>       "parameters": {
>         "authentication_type": "IAM_ROLE",
>         "role_arn": "arn:aws:iam::123456789012:role/QualyticsS3EnrichmentRole",
>         "external_id": "<your-external-id>"
>       }
>     }
>   }'
> ```
>
> **Response** (200 OK):
>
> ```json
> {
>   "id": 44,
>   "name": "Lake Enrichment",
>   "type": "s3",
>   "store_type": "dfs",
>   "root_path": "/qualytics-enrichment/",
>   "enrichment_only": true,
>   "connected": true
> }
> ```

> **Warning: Enrichment IAM Role permissions**
>
> The assumed role needs read **and** write permissions on the bucket (`s3:PutObject`, `s3:DeleteObject`, `s3:AbortMultipartUpload`, and so on). See [Enrichment Permissions](https://userguide.qualytics.io/source-datastore/datastore/dfs/amazon-s3/permissions/#enrichment-datastore-permissions-read-write) for the full IAM policy.

**Tab: Existing Connection**

Reuse a previously created S3 connection that was configured for read-write access. Pass `connection_id` and set `enrichment_only: true`.

> **Example: Example request and response**
>
> **Request**:
>
> ```bash
> curl -X POST "https://your-instance.qualytics.io/api/datastores" \
>   -H "Authorization: Bearer YOUR_TOKEN" \
>   -H "Content-Type: application/json" \
>   -d '{
>     "name": "Lake Enrichment",
>     "teams": ["Data Platform"],
>     "root_path": "/qualytics-enrichment/",
>     "enrichment_only": true,
>     "connection_id": 9
>   }'
> ```
>
> **Response** (200 OK):
>
> ```json
> {
>   "id": 45,
>   "name": "Lake Enrichment",
>   "type": "s3",
>   "store_type": "dfs",
>   "root_path": "/qualytics-enrichment/",
>   "enrichment_only": true,
>   "connected": true
> }
> ```

---

## Other Operations

For get, list, update, delete, test connection, and link-enrichment endpoints, see the [Datastore API](https://userguide.qualytics.io/source-datastore/datastore/api/) reference.

*Last updated: November 3, 2026.*
