Platform Status Troubleshooting

This page is the diagnostic entry point for the Platform Status page. Most of it covers Dataplane Diagnostics, the panel an Admin uses to capture a live report of what the Dataplane is doing, because that is where a slow or stalled operation is actually explained.

Start From the Symptom

WHAT IS WRONG	WHERE TO GO
The Database row is unhealthy.	Not a Dataplane problem. The platform cannot save or read data at all. See the Components Card and the FAQ.
The app is locked behind a Database Connection Lost dialog.	The platform lost its database connection while you were signed in. The dialog rechecks the connection every 10 seconds and closes on its own when it returns. See Platform Outage.
The RabbitMQ row is unhealthy.	Not a Dataplane problem. No operation request can reach the engine. See the Components Card and the FAQ.
A component shows a grey dot.	The status request itself did not return, so the state is unknown rather than unhealthy. Refresh the status summary.
The page warns about executor capacity.	The engine cannot get the workers it needs. See Capacity Warnings.
The Dataplane is unhealthy and will not come back.	Capture a report if the engine still answers, then restart it.
An operation is slow, stalled, or behaving oddly.	The rest of this page.

Dataplane Diagnostics

Dataplane Diagnostics is a subtab of the Dataplane card on the Platform Status page. It captures a detailed report of what the Dataplane is doing at that moment and renders it inline, so an Admin can answer "why is this operation slow?" without leaving the platform.

A report covers the operations running right now with their progress, a timeline of the work each one has done, how much memory, storage, and task capacity is in use, and automated findings that flag a slow or unhealthy pattern along with its likely cause and owner.

Permissions

Only users with the Admin role can open the Diagnostics subtab or capture a report. The subtab is not rendered at all for Managers. See the Permissions page for the full matrix.

How Capture Works

Each capture is live. The platform asks the Dataplane to render its current state, receives the finished report, and displays it. The report reflects the engine at the instant of the capture.

BEHAVIOR	WHAT TO EXPECT
First open	Opening the Diagnostics subtab captures a report automatically.
Capture time	A capture can take up to about 30 seconds, because the engine renders it on demand while it is still doing its normal work.
No history	Nothing is stored. There is no archive of past reports, and leaving the Status page discards the one on screen. Use Download snapshot to keep a copy.
Switching subtabs	Moving to Configuration and back keeps the current report loaded, without re-capturing.
During a recapture	The report already on screen stays visible and holds its scroll position until the replacement has finished loading.
Theme	The report follows the platform's light or dark theme and re-skins instantly when you toggle. Deployments running an older engine build return a light-only report.
Engine unreachable	If the Dataplane answers the health check but produces no report, an error notification appears and the previous report stays on screen.

Keeping evidence before a restart

Restarting the Dataplane clears the state a report describes, and reports are not retained. If you are about to restart a wedged engine, capture and download a report first so the state leading up to the restart can still be reviewed afterward.

Panel Controls

The panel's controls are covered step by step in the how-tos: Capture a Diagnostics Report for the Recapture diagnostics button, Capture on an Interval for Auto recapture , and Download a Diagnostics Report for Download snapshot .

When no report has been captured yet, the panel shows No diagnostics loaded with a prompt to recapture.

Info

The report is read-only. Its links are stripped during capture, so nothing in it navigates anywhere. The only interaction it keeps is column sorting: click a table header to sort by that column.

Where to Look in a Report

A report is long, so jump to the section that matches what you are chasing rather than reading it top to bottom.

WHAT YOU ARE SEEING	START HERE	WHY
One operation is running far longer than usual.	Findings, then Active work.	Findings names the cause outright when a detector fired. Active work confirms whether the operation is still advancing at all.
An operation reports no progress.	Executor stability.	Losing a worker makes the engine silently redo its cached and shuffle work, which reads as a stall everywhere else.
Everything feels slow, not one operation in particular.	The Workload tiles, then the Admission governor.	The tiles show whether the engine is saturated. The governor says how much concurrent work it is currently allowing, and why.
One datastore feels slower than the rest.	Source latency.	It ranks sources by measured p95, so a slow source stops being an impression.
A read of a large table never finishes.	Running loads, then Findings.	A partition count of one on a large table is the single-connection case, and it fires its own finding.
You are worried about memory.	The Spill tile in Workload, then Resources by operation.	Spill means the engine admitted more concurrent work than memory held comfortably. Resources says which operation is holding it.
You cannot tell whether work is still moving.	Nothing in one report.	Movement needs two readings. Turn on automatic recapture and watch Rows read and Input advanced.

A report opens with a one-line health verdict, then presents its sections in a fixed order. Sections with nothing to report are left out entirely, so a healthy idle engine produces a much shorter report than one under load. The sections group into the three questions below.

What Is Happening

The present moment: whether the engine considers itself healthy, how loaded it is, which operations are in flight, and how their steps have unfolded.

Health

A single line at the top of the report: the word Health, a colored badge, the dominant reason, and the job success rate for the last hour.

BADGE	WHEN IT APPEARS
Healthy	No finding reaches beyond a single container, and no job failed. A count of localized advisories may still be shown.
Attention	A finding affects a whole operation, a job failed, or executors were lost unexpectedly.
Degraded	A finding is degrading the whole cluster.

Info

The badge is a worst-of verdict, not an average. A single serious finding cannot be diluted by everything else going well. Localized findings never set the badge on their own, because by definition the work still completes.

Workload

A row of tiles covering both live pressure and the trailing hour. Each tile has a headline value and a smaller detail line.

TILE	HEADLINE VALUE	DETAIL LINE
Operations	How many operations are currently running Spark work.	The breakdown by operation type, or none active.
Container admission	Container slots in use against the total.	Slots waiting and available, plus the per-operation ceiling that a single scan, profile, or materialize can hold.
Task Demand	Tasks running right now.	Tasks pending, live slots, and live executors against the maximum.
Persisted storage	Cached data held in memory.	How many cached datasets, and how much of them spilled to disk.
Active RAM	Total memory across the live engine processes.	The split between executors and driver, and the percentage in use.
Local SSD	Capacity of the local volume the engine writes shuffle and spill data to.	Free space, how many volumes, and how many executors report them.
Data processed (last hour)	Bytes read from sources in the last hour.	The matching row count.
Job success rate (last hour)	Percentage of jobs that succeeded.	The counts behind it, any aborted jobs, and the percentage of task attempts that ran clean.
Rate while active	Read throughput measured only over the time the engine was actually working.	The peak rate for comparison.
Peak ingest	The highest read rate observed in the sampling window.	How many executors were live at that peak.
Peak slot use	The highest share of task slots occupied.	The peak count of running tasks.
Spill (last hour)	Data written out of memory to disk.	The memory and disk split. Zero spill reads no spill, admission healthy; any spill is a sign the engine admitted more concurrent work than memory comfortably held.

A context line below the tiles states how many jobs ran in the hour and how much of that hour the engine was actually busy.

Info

Rates are deliberately measured against busy time rather than wall-clock time. The workload is bursty, so dividing by a mostly-idle hour would understate how fast the engine actually moves data.

Active Work

Two tables describing what is in flight.

Profile and scan operations

COLUMN	WHAT IT SHOWS
Operation	The operation ID.
Type	The operation type.
Progress	Containers completed out of the total, with a percentage.
Active	Containers being processed right now.
Pending	Containers not started yet.
Succeeded	Containers finished successfully.
Failed	Containers that failed.
Elapsed	How long the operation has been running.
Since progress	How long since a container last completed.
Status	Running, or No progress when nothing has completed for 10 minutes. Hovering the badge explains why, including whether executors were recently lost.

Running loads

In-flight source reads, updated from live worker reports rather than only at the end of a task.

COLUMN	WHAT IT SHOWS
Operation	The operation type and ID driving the read.
Container	The table, file, or computed asset being read.
Source	The source type.
Partition key	The field the read is partitioned on, or `--` when there is none.
Partitions	Partitions completed out of those planned, plus how many are running.
Rows read	Rows read so far.
Expected	The row count the load expected, when it reported one.
Progress	Rows read against expected, as a percentage.
Rate	The average read rate.
ETA	Estimated time to finish at the current rate.
Input advanced	How long since the row counters last moved. Two minutes or more is highlighted, because a frozen counter is a hang forming before the task ever ends.

Info

A ≈ prefix means the row had retried or duplicate task attempts. Those attempts re-read rows that were already counted, so progress and rate read higher than reality.

Tip

A rate that makes the ETA absurd is the clearest signal of a skewed or single-connection read. Cross-check the container against the Findings table.

Operation Event Timeline

A visual timeline of instrumented steps, one lane per operation, colored by phase. In a captured report the axis is fitted to the period of activity, so a three-minute incident fills the frame and a busy hour shows the hour. Hover a bar for its details.

MARKING	MEANING
Load	Reading data from the source.
Shuffle / prepare	Redistributing data between workers.
Analytics	Running the quality checks and profiling work.
Write	Writing results out.
Other	Anything not classified above.
Hatched bar	Still running at the moment of capture.
Red outline	Failed.
Shaded band	A stretch with no instrumented work.
Red band	An executor was removed at that point.

Time by Phase

A stacked bar per operation showing where its time went, split by the same phases as the timeline. Bar length is relative to the longest operation, and hovering a segment gives its duration and share.

Info

Totals sum concurrent and nested steps, so a bar can add up to more than the operation's elapsed wall-clock time.

What Is Wrong

The diagnosis. Rather than leaving you to correlate five tables by eye, the engine names the symptom, attributes a cause, says who owns the fix, and states how far the damage reaches.

The Findings section is omitted entirely when there is nothing to say. It appears when a finding fired, when executors were added or removed during active work, or when a capacity warning is active, so a routine scale-down can bring the section up without anything being wrong.

Findings

The section that names what is wrong. It can carry up to three things: a red banner for anything degrading the whole cluster, a note or warning about executor churn and capacity, and a ranked table of findings.

Each finding is one row.

COLUMN	WHAT IT SHOWS
Operation	The operation the finding belongs to, or `--` when it cannot be tied to one.
Container	The table, file, or computed asset being processed.
Symptom	What is observably wrong. See Symptoms.
Cause	The attributed reason for the symptom. See Causes and owners.
Owner	Who can fix it: source, cluster, config, or unattributed.
Evidence	The measured numbers behind the finding, such as the observed rate against the source's own median.
Blast radius	How far the damage reaches. See Blast radius.
Since	How long the finding has been active, expressed as an age rather than a timestamp.

Symptoms

SYMPTOM	WHAT IT MEANS
failing	The container's work is accumulating task failures and retries beyond a normal level.
crawling	Throughput is far below what this source normally delivers, measured against its own history rather than a fixed threshold.
stuck	A task is still running but has read nothing new for an extended period.
single-connection	A large table is being read through a single connection because no usable partition field was available, so the read cannot spread across executors.
skewed	One task is taking dramatically longer than the median for the same stage, which points at an uneven partition key.
small-files	A file source is being read as a very large number of very small files, so per-file overhead dominates.
over-partitioned	The work was split into so many tiny tasks that scheduling overhead outweighs the work itself. The mirror image of single-connection.
memory-pressured	Storage is nearly full and data is spilling, or a single partition is oversized. This is a warning before an out-of-memory failure, not after one.
recompute-thrash	Stages are being retried because shuffle output went missing, so the same work is being redone.

Causes and Owners

CAUSE	OWNER	TYPICAL FIX
source unstable	source	The source is dropping connections. Investigate the source system's stability or connection limits.
source throttling	source	The source is rate-limiting the reads. Reduce concurrency against it or raise its quota.
source slow	source	The source is simply returning data slowly, with no failures. Investigate load on the source system.
source file layout	source	The source is laid out as many tiny files. Compact them at the source.
cluster churn	cluster	Executors were lost and their shuffle output had to be recomputed. Look at node stability and decommissioning behavior.
memory pressure	cluster	The engine is running out of headroom. Consider a larger deployment size.
slow executor	cluster	One worker is materially slower than its peers.
no partition field	config	The container has no usable partition field, so it reads through one connection. Set a partition field on the container.
bad partition key	config	The chosen partition key distributes rows unevenly. Choose a different key or adjust the bounds.
partition sizing	config	Partitions are too small and too numerous. Raise the partition size.
unattributed	unattributed	The symptom is real but no rule matched the evidence. The finding is still shown, with no owner asserted, and is worth reporting to the Qualytics support team.

Blast Radius

BLAST RADIUS	WHAT IT MEANS
localized	Only this container is affected. The operation still completes.
operation	Several containers in the same operation are degraded, so the operation as a whole is dragging.
cluster	This container is holding a large share of shared capacity while other work waits, so it is degrading everything else. Findings at this level also appear in the red banner at the top of the section.

Info

A finding only reaches cluster when something else is actually waiting for the capacity it is holding. On a lightly loaded engine, idle capacity next to a slow operation is expected and is not treated as cluster-wide damage.

Executor Notes

Above the findings table, one of three messages may appear about executors that went away during active work in the last 30 minutes.

MESSAGE TONE	WHAT HAPPENED
Warning	Executors were lost unexpectedly, for example crashed or stopped for memory. Their cached data and shuffle output are recomputed before dependent steps resume, which is why operations can report no progress while recovering.
Neutral note	Executors were decommissioned by the platform, for example a node drain, reclaim, or preemption. Their work is recomputed, but this was not an engine decision.
Neutral note	Executors were scaled down by the engine itself. This is normal elastic scaling, and the note says whether any job actually failed as a result.

Executor Stability

Worker lifecycle events from the last hour, newest first.

COLUMN	WHAT IT SHOWS
Time	When the event happened.
Event	added, scaled down (the engine gave capacity back), reclaimed (the platform took it away), or removed (a crash or a memory kill).
Executor	The worker's identifier.
Last observed live	How many workers were live just before the event.
Active jobs	How many jobs were running at that moment.
Reason	The classified reason, with the raw reason on hover.

A Task retries by cause (last hour) table follows, listing each failure class against the number of task attempts the engine retried.

Info

These are retries, not losses. A task attempt that failed and succeeded on the next try cost time, not results. The line above the table states whether any job in the window actually failed to recover.

Admission Governor

A one-paragraph readout of the component that decides how much container work runs concurrently. It samples the engine every 10 seconds and resizes the pool accordingly.

The paragraph gives the current mode with a colored badge, how long it has been in that mode, the reason for the current decision, the admission target against actual capacity, and the cache headroom against resident cached data. When the mode has changed recently, a table follows with Time, From, To, and Reason for each transition.

The Trailing Hour

Roughly the last hour of activity the engine still retains, which is what turns "this feels slow" into a measured comparison.

Warning

Once that hour passes, the evidence is gone. Reports are never stored, so a completed incident survives only in a file you downloaded.

Source Latency

Load-step duration per data source over the last hour, ranked by p95. This is the section that separates "our engine is slow" from "your source is slow".

COLUMN	WHAT IT SHOWS
Source	The data source.
Load steps	How many load steps ran against it, and how many are still active.
p50	Median load-step duration.
p95	95th percentile duration, also drawn as a bar relative to the slowest source.
Max	The slowest single load step.
Total	Combined time spent loading from this source.
Median load rate	The typical read rate, shown only for sources whose loads report row or byte counts.

Info

Durations cover the whole load step: connecting, querying, transferring, and materializing. Active steps extend to now, so a source that is hanging climbs this table while it hangs.

Resources by Operation

Which operation is holding what. A line above the table gives the sampled peaks for the whole engine.

COLUMN	WHAT IT SHOWS
Operation	The operation.
Cached now / peak	Data this operation currently holds in cache, and the most it has held.
Tasks now / peak	Task slots it currently occupies, and the most it has occupied.
Peak execution memory	The largest execution memory footprint observed for one of its jobs.

Info

A dataset shared by two operations counts toward both, so the column can add up to more than the engine actually holds.

Completed Work

Aggregated over the jobs the engine still retains, in two blocks.

Operation and container task outcomes, split into a By operation table and a By container table with the same columns:

COLUMN	WHAT IT SHOWS
Workload	The operation or container.
Source	The data sources involved.
Jobs success / fail	Job counts, with aborted jobs called out separately.
Job p50 / p95 / max	Job duration percentiles.
Tasks success / fail	Task attempt counts.
Task p50 / p95 / max	Task duration percentiles.

Recent DataFrame materializations, the most recent source loads the engine completed:

COLUMN	WHAT IT SHOWS
Started	When the load started.
Operation	The operation ID.
Type	The operation type.
Container	The container being loaded.
Step	The named step within the operation.
Source	The data source.
Source partitions	How many partitions the read used. A red 1 · single-connection badge marks an oversized read that had no usable partition field.
Rows	Rows materialized.
Cached size	Size of the materialized result in memory, which is not the same as the bytes transferred from the source.
Duration	How long the load took.
Throughput	Rate across reading, transforming, and caching.
Outcome	Success or failure.

Limitations

LIMITATION	DETAIL
Admin only	The subtab does not exist for other roles, and the underlying endpoint rejects them.
No retention	Reports are never stored. Closing the page discards the one on screen. Download anything you need to keep.
Capture cost	Each capture makes the engine render a report while it is working. Avoid leaving a short automatic interval running.
Bounded history	The report describes roughly the last hour of retained activity, plus whatever is live at the moment of capture. Older incidents are gone.
Not available when the engine is down	A capture needs the Dataplane to answer. If it has already stopped responding, there is nothing to render. Capture before restarting, not after.