Google Cloud Storage

Adding and configuring a Google Cloud Storage connection within Qualytics empowers the platform to build a symbolic link with your file system to perform operations like data discovery, visualization, reporting, syncing, profiling, scanning, anomaly surveillance, and more.

This documentation provides a step-by-step guide on how to add Google Cloud Storage as both a source and enrichment datastore in Qualytics. It covers the entire process, from initial connection setup to testing and finalizing the configuration.

By following these instructions, enterprises can ensure their Google Cloud Storage environment is properly connected with Qualytics, unlocking the platform's potential to help you proactively manage your full data quality lifecycle.

Let’s get started 🚀

Google Cloud Storage Setup Guide

This guide walks you through what you need from Google Cloud before adding the datastore to Qualytics: the bucket URI and a service account key.

Retrieve the Google Cloud Storage URI

To retrieve the Cloud Storage URI, follow the given steps:

1. Go to the Cloud Storage Console.
2. Navigate to the location of the object (file) that holds the source data.
3. At the top of the Cloud Storage console, locate and note down the path to the object.
4. Create the URI using the following format:

gs://<bucket_name>

The URI stops at the bucket. You choose the folder inside it with Root Path when you add the datastore, so do not append a path here.

Create a Service Account Key

The connector authenticates with a service account key, which you upload as a JSON file when you add the datastore. To create one:

Step 1: In the Google Cloud console, open IAM & Admin, then Service Accounts, and select the project that owns the bucket.

Step 2: Click Create Service Account, give it a name, and create it. You can skip the optional project-level role step, because the role that matters is granted on the bucket, as described below.

Step 3: Open the service account, go to its Keys tab, and click Add Key, then Create new key.

Step 4: Choose JSON and create the key. The file downloads once and cannot be downloaded again, so store it somewhere safe.

Step 5: Grant the service account the role it needs on the target bucket, per the privileges below.

Datastore Google Cloud Storage Privileges

The permissions required depend on whether you are using Google Cloud Storage as a source or an enrichment datastore. Qualytics accesses the bucket with the service account whose key you upload.

Minimum Permissions (Source Datastore)

The service account must have the following permissions:

Additional Permissions for Enrichment Datastore

When using Google Cloud Storage as an enrichment datastore, the following additional permissions are required:

Example IAM Policy

Replace <SERVICE_ACCOUNT_EMAIL> and <BUCKET_NAME> with your actual values.

Source Datastore (Read-Only)

{
  "bindings": [
    {
      "role": "roles/storage.objectViewer",
      "members": [
        "serviceAccount:<SERVICE_ACCOUNT_EMAIL>"
      ]
    }
  ]
}

Enrichment Datastore (Read-Write)

{
  "bindings": [
    {
      "role": "roles/storage.objectAdmin",
      "members": [
        "serviceAccount:<SERVICE_ACCOUNT_EMAIL>"
      ]
    }
  ]
}

Assigning via gcloud CLI

# Source Datastore (Read-Only)
gsutil iam ch \
  serviceAccount:<SERVICE_ACCOUNT_EMAIL>:roles/storage.objectViewer \
  gs://<BUCKET_NAME>

# Enrichment Datastore (Read-Write)
gsutil iam ch \
  serviceAccount:<SERVICE_ACCOUNT_EMAIL>:roles/storage.objectAdmin \
  gs://<BUCKET_NAME>

GCS Roles Summary

Troubleshooting Common Errors

Detailed Troubleshooting Notes

Authentication Errors

The error Invalid credentials indicates that the service account key is incorrect or malformed.

Common causes:

• Malformed service account key: the JSON key file is corrupted, truncated, or belongs to a different project.
• Service account disabled: the service account has been disabled in the Google Cloud Console.

Permission Errors

The error 403 Forbidden or The caller does not have storage.objects.list access means the credentials are valid but lack the required IAM permissions.

Common causes:

• Missing IAM role: the service account does not have Storage Object Viewer (source) or Storage Object Admin (enrichment) assigned on the target bucket.
• Role assigned at wrong level: the role is assigned at the project level but a bucket-level policy overrides it.
• Uniform bucket-level access: if the bucket uses uniform bucket-level access (recommended), ensure IAM policies are set at the bucket level, not through ACLs.
• Source vs. enrichment mismatch: the service account has Storage Object Viewer but the operation requires write access (enrichment).

Connection Errors

The error 404 Not Found: Bucket not found indicates a configuration issue with the bucket name or URI.

Common causes:

• Bucket does not exist: the bucket name was misspelled or the bucket has been deleted.
• Wrong project: the service account belongs to a different Google Cloud project than the bucket.
• Invalid URI format: the URI must follow gs://<bucket_name>. Extra path segments or incorrect formatting will cause failures.

Add a Source Datastore

A source datastore is a storage location Qualytics connects to so it can profile, scan, and monitor data. Adding Google Cloud Storage as a source lets Qualytics read files directly from your bucket and run quality operations on the data they contain.

Before you start, create the key described in Create a Service Account Key and review the required privileges.

Field reference

The Add Datastore page shows the sections below when Google Cloud Storage is selected. When reusing an existing connection, the Connection Properties and Secrets Management sections come already filled in and read-only: Qualytics has already validated those credentials, so you fill in only the Datastores Extraction and the Datastore Properties. To change a saved connection's credentials, edit the connection through the Manage Connections page; edits there apply to every datastore that reuses the connection.

Connection Properties

These fields define where the bucket lives and how Qualytics authenticates to it. They belong to the connection: when reusing an existing connection, they come already filled in and read-only.

Secrets Management

This group is optional: use it only if you want Qualytics to pull credentials from a secrets manager instead of typing them into the form. Turn on HashiCorp Vault to show the fields below. Despite the label, any secrets manager that exposes a compatible REST API works, not only HashiCorp Vault; see Secrets Management. It also belongs to the connection: read-only when reusing an existing connection.

Datastores Extraction

Pick the folder inside the bucket Qualytics should read from. You fill this in on both flows.

Datastore Properties

Common fields for every source datastore, shown below the Datastores Extraction section. You fill these in on both flows.

Steps

There are two ways to set up the connection: reuse a connection you already saved (Existing Connection) or create a new one from scratch (New Connection). The tabs below walk through each option; pick the one you want to follow. Each field is described in the Field reference above.

Add Enrichment Datastore

An enrichment datastore is where Qualytics writes what it finds: anomalies, remediation files, and record enrichment. Google Cloud Storage is supported for this role, so the same bucket can hold both the data you monitor and the results.

Field reference

The Enrichment Datastore step shows the sections below when Google Cloud Storage is selected. When reusing an existing connection, the Connection Properties and Secrets Management sections come already filled in and read-only.

Connection Properties

These fields define where the bucket lives and how Qualytics authenticates to it. They are the same fields as on the source datastore flow, repeated here so this section stands on its own.

Secrets Management

This group is optional: use it only if you want Qualytics to pull credentials from a secrets manager instead of typing them into the form. Turn on HashiCorp Vault to show the fields below. Despite the label, any secrets manager that exposes a compatible REST API works, not only HashiCorp Vault; see Secrets Management. It also belongs to the connection: read-only when reusing an existing connection.

Enrichment Extraction

Where Qualytics writes the enrichment files.

Enrichment Properties

Advanced Options

Collapsed by default. Expand it to change how anomalous source records are replicated.

Steps

A Google Cloud Storage enrichment datastore can be created from two places: as the second step of creating a source datastore, or on its own from the Enrichment Datastores page. Either way you choose between creating a connection from scratch (New Connection) or reusing a saved one (Existing Connection). The tabs below cover both entry points for each option; each field is described in the Field reference above.

API Payload Examples

This section provides detailed examples of API payloads to guide you through the process of creating and managing datastores using Qualytics API. Each example includes endpoint details, sample payloads, and instructions on how to replace placeholder values with actual data relevant to your setup.

Creating a Source Datastore

This section provides sample payloads for creating the Google Cloud Storage datastore. Replace the placeholder values with actual data relevant to your setup.

Endpoint: /api/datastores (post)

        {
        "name": "your_datastore_name",
        "teams": ["Public"],
        "trigger_sync": true,
        "root_path": "/gcs_root_path",
        "enrichment_only": false,
        "connection": {
            "name": "your_connection_name",
            "type": "gcs",
            "uri": "gs://<bucket_name>",
            "secret_key": "gcs_service_account_key"
        }
    }

   {
        "name": "your_datastore_name",
        "teams": ["Public"],
        "trigger_sync": true,
        "root_path": "/gcs_root_path",
        "enrichment_only": false,
        "connection_id": 123
    }

# Step 1: Create a Connection
qualytics connections create \
    --type gcs \
    --name "your_connection_name" \
    --uri "gs://<bucket_name>" \
    --secret-key ${GCS_SERVICE_ACCOUNT_KEY}

# Step 2: Create a Source Datastore
qualytics datastores create \
    --name "your_datastore_name" \
    --connection-name "your_connection_name" \
    --database . \
    --schema /

Creating an Enrichment Datastore

This section provides sample payloads for creating an enrichment datastore. Replace the placeholder values with actual data relevant to your setup.

Endpoint: /api/datastores (post)

    {
        "name": "your_datastore_name",
        "teams": ["Public"],
        "trigger_sync": true,
        "root_path": "/gcs_root_path",
        "enrichment_only": true,
        "connection": {
            "name": "your_connection_name",
            "type": "gcs",
            "uri": "gs://<bucket_name>",
            "secret_key": "gcs_service_account_key"
        }
    }

    {
        "name": "your_datastore_name",
        "teams": ["Public"],
        "trigger_sync": true,
        "root_path": "/gcs_root_path",
        "enrichment_only": true,
        "connection_id": 123
    }

# Step 1: Create a Connection
qualytics connections create \
    --type gcs \
    --name "your_connection_name" \
    --uri "gs://<bucket_name>" \
    --secret-key ${GCS_SERVICE_ACCOUNT_KEY}

# Step 2: Create an Enrichment Datastore
qualytics datastores create \
    --name "your_datastore_name" \
    --connection-name "your_connection_name" \
    --database . \
    --schema /your_enrichment_path \
    --enrichment-only

Link an Enrichment Datastore to a Source Datastore

Use the provided endpoint to link an enrichment datastore to a source datastore:

Endpoint Details: /api/datastores/{datastore-id}/enrichment/{enrichment-id} (patch)