Add an Amazon S3 Source Datastore
A source datastore is a storage location Qualytics connects to so it can profile, scan, and monitor data. Adding S3 as a source lets Qualytics read files directly from your bucket and run quality operations on the data they contain.
Before you start, review the Amazon S3 Permissions and the available Authentication methods.
Tip
S3 can also hold an enrichment datastore, where Qualytics writes anomalies and metadata. Link one during creation via Link Enrichment on Datastore Creation, or afterwards via Link Enrichment Datastore.
Field reference
The Add Datastore page shows the sections below when Amazon S3 is selected. When reusing an existing connection, the Connection Properties, Authentication, and Secrets Management sections come already filled in and read-only: Qualytics has already validated those credentials, so you fill in only the Datastores Extraction and the Datastore Properties. To change a saved connection's credentials, edit the connection through the Manage Connections page; edits there apply to every datastore that reuses the connection.
Connection Properties
These fields define where the bucket lives. They belong to the connection: when reusing an existing connection, they come already filled in and read-only.
| FIELD | REQUIRED | TYPE | DESCRIPTION |
|---|---|---|---|
| Connection Name | Text | A label for the saved connection (e.g., acme_s3_lake), so other datastores can reuse it later. |
|
| URI | Text | The bucket-level URI, in the form s3://<bucket_name>. Leave out any folder path here and use Root Path below to scope to a subfolder. The region is worked out from the bucket. |
Authentication
Choose how Qualytics authenticates to AWS. Setting Type changes the credential fields shown below it, so pick the tab that matches your choice. These fields also belong to the connection: already filled in and read-only when reusing one. See Authentication for how each method works.
| FIELD | REQUIRED | TYPE | DESCRIPTION |
|---|---|---|---|
| Type | Option | Set to Access Key, which is the default (SHARED_KEY in the API). |
|
| Access Key | Text | The access key ID of an IAM user holding the permissions listed in Permissions. | |
| Secret Key | Text | The secret access key that goes with it. |
| FIELD | REQUIRED | TYPE | DESCRIPTION |
|---|---|---|---|
| Type | Option | Set to IAM Role (IAM_ROLE in the API). Qualytics assumes the role and uses short-lived credentials that refresh on their own. |
|
| Role ARN | Text | The ARN of the IAM role to assume (e.g., arn:aws:iam::123456789012:role/MyRole). |
|
| External ID | Text | Fill this in only when the role's trust policy requires an external ID. |
AWS-only
IAM Role appears in Type only on AWS and local deployments. On Azure and GCP deployments, only Access Key is offered.
Secrets Management
This group is optional: use it only if you want Qualytics to pull credentials from a secrets manager instead of typing them into the form. Turn on HashiCorp Vault to show the fields below. Despite the label, any secrets manager that exposes a compatible REST API works, not only HashiCorp Vault; see Secrets Management. It also belongs to the connection: read-only when reusing an existing connection.
| FIELD | REQUIRED | TYPE | DESCRIPTION |
|---|---|---|---|
| Login URL | Text | The Vault endpoint Qualytics uses to authenticate (e.g., https://vault.example.com/v1/auth/approle/login). |
|
| Credentials Payload | Text | A JSON body containing the credentials Vault expects (e.g., {"role_id":"...","secret_id":"..."}). |
|
| Token JSONPath | Text | The JSONPath that extracts the client token from Vault's response. Defaults to $.auth.client_token. |
|
| Secret URL | Text | The Vault path where the secret is stored (e.g., https://vault.example.com/v1/secret/data/s3). |
|
| Token Header Name | Text | The HTTP header name used to send the token. Defaults to X-Vault-Token. |
|
| Data JSONPath | Text | The JSONPath that extracts the secret payload from Vault's response. Defaults to $.data. |
Note
Once the secrets manager is configured, reference any secret from a Connection Properties or Authentication field with ${key}, for example ${secret_key}. Qualytics resolves it each time the connection is opened, so a changed value takes effect on the next connection.
Datastores Extraction
Pick the folder inside the bucket Qualytics should read from. You fill this in on both flows.
| FIELD | REQUIRED | TYPE | DESCRIPTION |
|---|---|---|---|
| Root Path | Option | The folder inside the bucket where the data lives (e.g., /raw/orders/). Defaults to /, which reads from the bucket root. Click the refresh icon to load the folders the credentials can see. |
Datastore Properties
Common fields for every source datastore, shown below the Datastores Extraction section. You fill these in on both flows.
| FIELD | REQUIRED | TYPE | DESCRIPTION |
|---|---|---|---|
| Name Template | Text | Defines the naming pattern for the source datastore being created. Left empty, the datastore is named from the connection name and the root path. | |
| Group | Option | Organizes your datastores under a shared group in the navigation tree. Select an existing group or create a new one with the Add New Group toggle. | |
| Teams | Option | Select one or more teams to associate with this source datastore. | |
| Initiate Sync | Checkbox | Automatically sync the datastore to detect containers and fields after creation. |
Allowlisting
A banner at the bottom of the form shows the IP address the connection is established from. Allowlist it wherever your bucket access is restricted by address, so the connection can be opened.
Steps
There are two ways to set up the connection: reuse a connection you already saved (Existing Connection) or create a new one from scratch (New Connection). The tabs below walk through each option; pick the one you want to follow. Each field is described in the Field reference above.
Step 1: Navigate to the Source Datastores page.
Step 2: Click the Add Source Datastore button at the top-right corner.
Step 3: The Add Datastore page opens.
Step 4: Select New Connection next to the Search field.
Step 5: Select Amazon S3 from the connector grid. Use the search field to filter connectors by name.
Step 6: Fill in the Connection Properties: the Connection Name and the URI.
Step 7: Fill in the Authentication fields for the Type you choose.
Step 8: Optionally, expand Secrets Management to retrieve credentials from a secrets manager.
Step 9: Fill in the Datastores Extraction field (Root Path) and the Datastore Properties.
Step 10: Click Test connection. A success message confirms that the connection has been verified.
Info
The Finish and Next buttons stay disabled until the connection test passes on the current values. If the test fails, see Troubleshooting.
Step 11: Click Finish to create the datastore.
Tip
To link an enrichment datastore so Qualytics can store anomalies and metadata from the first operation, click Next instead of Finish. See Link Enrichment on Datastore Creation.
Step 12: A success dialog confirms that your datastore has been added. Click Go to your datastore to open its page.
Step 1: Navigate to the Source Datastores page.
Step 2: Click the Add Source Datastore button at the top-right corner.
Step 3: The Add Datastore page opens.
Step 4: Select Existing Connection next to the Search field.
Step 5: Select the saved S3 connection from the grid. Use the search field to filter connections by name. The Connection Properties, Authentication, and Secrets Management sections come already filled in and read-only.
Start a new connection from this one
To use the selected connection as a starting point for a brand-new connection instead, click the Duplicate as a new connection button on the selected connection. The form switches to New Connection mode with the connection's settings already filled in for you to adjust.
Step 6: Fill in the Datastores Extraction field (Root Path) and the Datastore Properties. These are the only fields left to fill in.
Step 7: Click Test connection. A success message confirms that the connection has been verified.
Info
The Finish and Next buttons stay disabled until the connection test passes on the current values. If the test fails, see Troubleshooting.
Step 8: Click Finish to create the datastore.
Tip
To link an enrichment datastore so Qualytics can store anomalies and metadata from the first operation, click Next instead of Finish. See Link Enrichment on Datastore Creation.
Step 9: A success dialog confirms that your datastore has been added. Click Go to your datastore to open its page.