Getting Started with Datastores

What is a Datastore?

In Qualytics, a Source Datastore is where your data lives. It is the connection Qualytics opens to look at the tables or files inside a system so it can profile them, run quality checks, and detect anomalies.

You may also encounter this concept under names like data source or data connection. In Qualytics, we always call it a Source Datastore, and every underlying technology is treated the same way: whether the system is a relational database, a data warehouse, or a cloud object storage bucket, a Source Datastore is the parent that owns a collection of containers (tables, files, or other data groupings).

Everything else in the platform starts from the Source Datastore: once it exists, you can Sync it to discover the containers inside, Profile those containers to understand their contents, and Scan them for anomalies.

Enrichment Datastores

Qualytics also supports a second flavor called an Enrichment Datastore: a separate, user-managed storage location where the platform writes results back (anomaly records, scan results, and other operation metadata) so you can query them alongside your own data or feed them into downstream reporting. Linking one is optional. See Enrichment Datastore Overview for the full story.

How a datastore maps to your storage system

The exact slice of data a datastore represents depends on the connector:

For relational databases (PostgreSQL, Snowflake, and other JDBC engines), a datastore points at one schema inside a specific database. If a database holds three schemas that you want to monitor, you create three datastores. The Add Datastore form asks for both the database and the schema name so Qualytics knows exactly where to look.
For cloud object storage (Amazon S3, Azure Data Lake Storage, Google Cloud Storage), a datastore points at a folder path inside a bucket, so files under the same prefix are grouped together.

In every case the mental model is the same: the datastore is the boundary Qualytics reads from, and the tables or files inside that boundary are the containers you will work with once Sync completes.

Datastore Lifecycle

Every datastore in Qualytics follows a structured lifecycle — from initial creation through ongoing data quality operations.

graph TD
    A["<b>Create</b><br/>Add datastore & test connection"] --> B["<b>Link Enrichment</b><br/>Connect enrichment datastore"]
    B --> C["<b>Sync</b><br/>Discover tables, files & fields"]
    C --> D["<b>Profile</b><br/>Analyze data & infer checks"]
    D --> E["<b>Scan</b><br/>Run checks & detect anomalies"]
    E -->|"Repeat"| C

Stage	Description
Create	Add a new source datastore by selecting a connector, providing connection credentials, and testing the connection.
Link Enrichment	Optionally link an enrichment datastore to persist scan results, anomalies, and remediation data.
Sync	Discover the schema — tables, files, views, and fields — from your source datastore. This is the first operation after creation.
Profile	Analyze field patterns across records, detect data types, compute statistics, and automatically infer quality checks.
Scan	Execute quality checks against the data, measure data quality metrics, and detect anomalies from failed checks.

Tip

The Sync → Profile → Scan cycle is repeatable. As your data evolves, re-running these operations keeps your quality checks and anomaly detection up to date.

Run Sync from your source datastore

Sync is the only operation triggered directly from the source datastore — it discovers tables, files, and fields and is always the first step after creating a datastore. See the Sync Operation page for step-by-step instructions.

Architecture

The Qualytics Datastore framework is organized in four layers:

Layer	Description
Qualytics I/O	Spark-enabled reads and writes of Fields and Records — the common interface for all datastores.
Namespace	How data is organized within the storage engine: Tables in a Schema (JDBC), Files in a Folder (DFS), or Topics in a Broker (Streaming).
Schema	The data definition layer: RDBMS DDL for relational databases, AVRO/Parquet/CSV/JSON for file systems, or AVRO/JSON/Protobuf for streaming platforms.
Storage Engine	The actual platform where data resides (see supported engines below).

Supported Storage Engines

For the full list of supported JDBC and DFS connectors, see the Available Datastore Connectors page.

How It All Connects

graph LR
    IO["<b>Qualytics I/O</b><br/>Spark-enabled reads and writes<br/>of Fields and Records"]

    IO --> NS1["<b>Tables in a Schema</b>"]
    IO --> NS2["<b>Files in a Folder</b>"]

    NS1 --> SC1["<b>RDBMS DDL</b>"]
    NS2 --> SC2["<b>AVRO, Parquet, CSV, JSON</b>"]

    SC1 --> SE1["Oracle DB"]
    SC1 --> SE2["PostgreSQL"]
    SC1 --> SE3["MySQL"]
    SC1 --> SE4["Snowflake"]
    SC1 --> SE5["Redshift"]
    SC1 --> SE6["...and more"]

    SC2 --> SE7["Amazon S3"]
    SC2 --> SE8["Azure Storage"]
    SC2 --> SE9["GCS"]

The key insight is that Qualytics treats all datastores uniformly at the I/O layer. Whether you connect a PostgreSQL database or an S3 bucket, Qualytics reads and writes Fields and Records the same way — enabling consistent profiling, scanning, and anomaly detection across your entire data landscape.

Available Connectors

Qualytics supports 19 JDBC relational database connectors and 3 DFS cloud storage connectors out of the box.

All Connectors

See the complete list of supported connectors with links to their individual setup guides.

View Connectors

Deep Dive

Explore the details of each datastore type — how JDBC and DFS connectors work, their configuration, and supported features.

JDBC

Learn how to connect relational databases using JDBC connectors like PostgreSQL, Snowflake, Oracle, and more.

Understanding JDBC
DFS

Learn how to connect distributed file systems like Amazon S3, Azure Data Lake Storage, and Google Cloud Storage.

Understanding DFS

Connection

Learn how to set up and manage connections to your datastores — create new connections from scratch or reuse existing credentials.

Connections Overview

Set up new connections or reuse existing credentials to connect your datastores.

Connections Overview

Multiple-Schema

Discover and onboard multiple schemas from a single connection at once — including schema discovery, name templates, supported connectors, and API reference.

Introduction

Discover and onboard multiple schemas from a single connection in one step.

Getting Started
How It Works

Understand the multi-schema creation flow, schema discovery, and name templates.

How It Works
Supported Connectors

See which connectors support multi-schema discovery and their catalog/schema mappings.

Supported Connectors
Permissions

Understand the roles and permissions required for multi-schema creation.

Permissions
API

API endpoints for bulk datastore creation, schema discovery, and validation.

API
FAQ

Answers to common questions about multi-schema source datastore creation.

FAQ

Managing

Add, edit, and delete datastores — whether creating from scratch with a new connection or reusing an existing one.

Add Datastore with new connection

Create a new source datastore by setting up a new connection from scratch.

New Connection
Add Datastore with existing connection

Create a new source datastore by reusing credentials from an existing connection.

Existing Connection
Edit Datastore

Modify the settings and connection details of an existing datastore.

Edit Datastore
Delete Datastore

Remove a datastore and its associated configuration from Qualytics.

Delete Datastore