Databricks on AWS Processing Deployment

This guide explains how to run Qualytics data processing as a continuous job in Databricks on AWS while the application remains in your self-hosted Kubernetes environment.

The approved Databricks bundle is release-specific. Use the bundle and image tag provided for your Qualytics release instead of copying configuration from another deployment.

Architecture

flowchart LR
    Users["Users and API clients"] --> App["Self-hosted Qualytics application"]
    IdP["Identity provider"] --> App
    Job["Databricks on AWS processing job"] -->|"Outbound private TLS on port 5671"| Endpoint["Private processing endpoint"]
    Endpoint --> App
    Job --> Data["Customer datastores"]

The Databricks job initiates the connection to a secure endpoint in the self-hosted environment. Customer data processing remains in Databricks.

Prerequisites

Before starting, ensure you have:

• A self-hosted Qualytics deployment with its required deployment identifier
• A Databricks on AWS workspace with permission to create and run jobs
• The Databricks CLI installed
• A version-matched Databricks bundle and Qualytics image tag
• Qualytics image registry credentials
• A private endpoint that Databricks can reach on TCP port 5671
• Customer-managed private load-balancing and DNS for that endpoint
• A TLS certificate for that endpoint that Databricks trusts

Step 1: Prepare the Self-Hosted Application

Work with your Qualytics account manager to prepare a restricted Helm values override for the approved chart version. The override must:

• Select Databricks as the processing environment
• Disable the in-cluster processing job
• Enable the TLS-protected service endpoint for Databricks
• Enable the chart's TLS-protected inbound service

Create the required TLS Secret named rabbitmq-tls in the Qualytics namespace:

kubectl create secret tls rabbitmq-tls \
  --namespace qualytics \
  --cert=fullchain.pem \
  --key=privkey.pem

Apply the override with your existing protected values file:

CHART_VERSION="<version provided by Qualytics>"

helm upgrade --install qualytics qualytics/qualytics \
  --namespace qualytics \
  --create-namespace \
  --version "$CHART_VERSION" \
  -f values.yaml \
  -f databricks.values.yaml \
  --wait \
  --timeout=20m

The chart exposes the secure service through Kubernetes NodePort 30671. It does not create the private load balancer or DNS name needed by Databricks. Provision a private TCP endpoint that listens on port 5671 and forwards to NodePort 30671 on the cluster nodes.

Confirm that the Databricks workspace can reach that private endpoint and that its DNS name matches the TLS certificate. Do not expose the endpoint to the public internet.

Step 2: Collect the Connection Values

Record the following values through your organization's approved secret-sharing process:

The service password, registry token, deployment identifier, and platform license are separate values. Do not reuse one in place of another.

Step 3: Install and Authenticate the Databricks CLI

Follow the Databricks instructions to install or update the CLI.

Authenticate to the target workspace:

databricks auth login --host <workspace-url>

Step 4: Store the Required Secrets

Create the secret scope named in the approved bundle:

databricks secrets create-scope <scope-name>

The bundle defines the required secret keys. Add the service password and registry credentials using those exact key names:

databricks secrets put-secret <scope-name> <service-password-key>
databricks secrets put-secret <scope-name> <registry-username-key>
databricks secrets put-secret <scope-name> <registry-token-key>

The CLI prompts for each value securely. Confirm that the keys exist without displaying their contents:

databricks secrets list-secrets <scope-name>

Step 5: Configure the Approved Bundle

Update only the customer-specific inputs documented with the bundle:

Do not copy internal job settings from another release. They can change with the Qualytics image and Databricks runtime.

Step 6: Deploy and Start the Processing Job

From the approved bundle directory, deploy the target:

databricks bundle deploy --target <target>

Start the continuous processing job using the job key included with the bundle:

databricks bundle run --target <target> <processing-job-key>

Verify the Deployment

1. Confirm that the Databricks job remains running.
2. Review the job output for a successful connection to the self-hosted service endpoint.
3. Start a Qualytics operation and confirm that the Databricks job processes it.
4. Confirm that the Kubernetes processing job is not running at the same time.

Troubleshooting