Demo workload with deliberate HTTP errors. Resource charts compare the workload with the highest single-Pod value at each point; each chart identifies its population.
Open and use the charts
- Open a Deployment, StatefulSet or DaemonSet and select Metrics. In Applications, select one of those workloads, then open its Metrics tab. These are the selected workload’s charts, not a sum across the whole app.
- Choose a time window. The selection stays in the URL through reloads, tab changes and returning from a Pod drilldown. Missing or invalid values use one hour.
- If both HTTP observers are available, choose Istio or Beyla beside Requests. Istio is the default when both have data. Radar never adds their observations together; the choice does not force other workloads to use it.
- Read the population label beside each section. Workload history includes retained previous Pods. Current Pods only excludes previous replicas. Gaps and unavailable data are not measured zeros.
- Scroll to Compare current Pods · latest samples to find an outlier, sort by CPU, memory or throttling, and open a Pod for investigation.
Compare the latest samples for current Pods. Template per Pod is a configuration reference, not the measured or historical capacity of every replica.
What needs to be installed?
These are data prerequisites, not guarantees from an installed chart’s name.
Prometheus, Mimir, VictoriaMetrics and Thanos can expose a compatible query API;
collector configuration, labels, retention and histogram format determine which
panels work. See what each chart needs for exact series
and live-tested coverage for tested configurations.
Radar discovers common in-cluster query Services and checks workload identity
automatically. Hosted URLs, authentication and tenant headers still require
configuration. Configure metrics opens Metrics settings in standalone Radar;
embedded views direct users to their operator. Discover Prometheus retries
discovery. Connecting a backend does not install missing instrumentation.
If you switch between clusters locally, first read
local multi-cluster settings: a manually configured
backend and its headers are not saved separately for each cluster.
What the observations mean
Understand coverage and missing charts
Important scope and data-quality notices stay beside the affected charts. About these metrics opens the Metrics sources & coverage dialog for the selected workload and time window. It explains current-Pod matching for each source, historical ownership, chart definitions and optional troubleshooting.
Matching evidence is separate from historical coverage. An absent optional observer does not invalidate another source's charts.
Network/storage remain separately visible and name-based. Template settings
beside current-Pod comparison can differ from actual Pods after injection or
rollout; omitted values do not mean zero requests or unlimited capacity.
Local multi-cluster settings
Manual Prometheus settings are Radar-wide today, not per-cluster profiles. The configured URL, HTTP headers (including credentials and tenant headers) and environment-variable header mappings persist in the local config. Switching Kubernetes context keeps the manually selected backend and headers. With no manual URL or headers, Radar rediscovers a backend in the selected cluster. With a manual backend, select the appropriate URL and headers in Metrics settings when changing clusters. When changing endpoints, explicitly replace or clear saved headers; editing only the URL retains them. Headers require an explicit URL and are never sent to auto-discovered candidates. The new workload charts invalidate identity evidence on connection changes and recheck it. Optional scope assertions are discarded on context, endpoint or credential changes, including automatic failover to a different metrics service, service port or backend path. Reconnecting to the same discovered service through a new local port-forward preserves the assertion. Those safeguards do not implement per-cluster connection profiles or retrofit identity checks onto older name-based charts. See integration settings when switching clusters. In-cluster Radar has no context switcher; provision its backend and credentials for that installation. This guide does not certify every Radar Cloud transport or managed-service authentication path.Which workloads benefit?
The resource panels apply to service processes and background workers alike. HTTP request panels are conditional on observed metrics, not declared container ports, Services, or Ingresses. Radar does not scan application ports to generate RED (request rate, errors, duration) data.
Radar does not universally exclude health checks, metrics scrapes or admin routes.
If the exporter includes them, the workload aggregate can include them. Multiple
ports do not themselves mean duplicate scrapes, but multiple observation jobs or
known HA populations can trigger the ambiguity guard. Incompatible histogram
populations can withhold latency while request rate remains useful. A future
port/route breakdown requires labels the source actually retains; Kubernetes port
declarations cannot recover dimensions already aggregated away by the exporter.
What each chart needs
All new panels require a reachable Prometheus-compatible query API, workload-read and namespace Pod-list access. Historical charts need verified/asserted cluster scope and retained ownership. The labeled current-only fallback needs verified source identity or an operator assertion. KSM means kube-state-metrics; it exports Kubernetes object state, not application request telemetry.
Metrics Server alone does not provide this historical Prometheus data. A
Prometheus installation scraping kubelet/cAdvisor can support resource panels
without Beyla or Istio. KSM can supply inventory for fallback attribution, but does
not create missing cAdvisor counters or HTTP histograms. Retention, scrape gaps,
collector filters and relabeling still determine which panels are available.
What Radar discovers automatically
There are three separate steps; success at one does not imply success at the next:- Find a query endpoint. At cluster startup/reconnect, probe known Service locations and ranked dynamic candidates. These include common Prometheus and VictoriaMetrics installations; dynamic discovery recognizes query services such as Thanos Query. Radar tries direct connectivity and, from a laptop, Kubernetes port-forwarding when needed and permitted. A manual URL overrides discovery. Hosted endpoints, tenants and credentials are not guessed. Configured HTTP headers require an explicit URL so credentials do not reach discovered candidates. One selected backend supplies these panels; Radar does not combine data across every Prometheus it finds.
- Establish scope and membership. On opening Metrics, independently probe KSM
name/UID anchors for cluster scope, even when sources already match by UID.
Compare raw/recorded ownership coverage over the range. Current attribution
runs separately for comparison/fallback. Historical Beyla queries accept custom
jobs, checking observation populations throughout the range; automatic matching
does not require a job name containing
beyla. - Query usable observations. Fetch the time window and check counter, status, histogram and observation-population coverage. Reachability and identity are not guarantees that a requested chart has usable data.
Automatic attribution
Radar checks each source independently when the workload Metrics tab opens. No scope flag is needed when the evidence is sufficient:- Beyla: match application
k8s_pod_nameandk8s_pod_uidto current Kubernetes Pods. - cAdvisor: match Pod name and an exact Pod UID segment in the
idcgroup path (cgroupfs or systemd). CPU, memory and throttling are checked independently. - Otherwise, use
kube_pod_infoname/UID pairs to establish a unique one- or two-label cluster partition, then require those same labels on the source. Recognized labels arecluster,cluster_name,k8s_cluster_name,kubernetes_cluster, andcluster_id. Istio uses this path, not a guessed UID.
replica,
prometheus_replica, __replica__) with multiple values are rejected; backends
using other replica labels must deduplicate upstream. No arbitrary label stripping
is attempted.
Current-Pod evidence probes run asynchronously with an eight-second deadline after shared
Prometheus discovery finishes (discovery has its own 60-second bound), two active workloads
at most, a bounded 128-entry memo and a 30-second per-workload churn guard.
Evidence is bounded to 1,024 identities per metric family, 256 KSM rows and a
4 MiB response. Positive results expire after five minutes and refresh ahead of
expiry; negative results retry after 30 seconds. A purely negative explanation
remains visible while the same identity/connection is rechecked; expired positive
trust is never extended. During Pod churn, unexpired
evidence is retained only for unchanged name/UID pairs while replacement Pods
are checked. Its original expiry is not extended. Connection changes invalidate
all attribution and their old churn timers. Sources covering a subset report partial coverage.
Historical partition proof has a separate connection-wide five-minute positive
cache and one coalesced probe per connection generation. Proof from an authorized
namespace can be reused; charts still query only their own authorized namespace.
At zero Pods, bounded anchors can come from other Pods in that same namespace.
A cold connection with no anchors and no assertion cannot establish scope.
UID joins inside a store cannot prove which cluster Radar is connected to.
The UI points to operator configuration in About these metrics when this prevents history.
Current UID-filtered charts exclude previous incarnations of same-name Pods.
Historical charts describe the logical cluster/namespace/kind/name workload,
including recreation under that name, not workload-UID lifetimes. Network/storage
and other existing surfaces keep their previous query behavior. Template overlays
are current per-Pod settings, not historical capacity.
Optional operator override
If automatic matching cannot establish identity, an operator can explicitly assert the backend’s scope. These assertions take precedence over probing. For a backend dedicated to the connected Kubernetes cluster:--prometheus-cluster-label for additional exact label constraints. The
matchers are combined with AND, not OR. An empty value is rejected because it
would also match series without that label. A tenant header alone does not prove
single-cluster isolation. Use existing --prometheus-header-from-env for credentials.
These assertions are process-local, apply to the startup Kubernetes connection,
and are not saved to the config file. Changing context, endpoint or headers discards
the assertion and returns to automatic matching. Restart to supply a fresh assertion. Existing resource
charts retain their existing query behavior; this is not a retrofit of cluster
isolation onto every metrics API.
Helm installations can provision the assertion with
traffic.prometheusSingleCluster: true or traffic.prometheusClusterLabels,
and optionally traffic.beylaJobSelector. Defaults do not assert trust. Use a
chart and Radar image version containing this feature together; adding these
arguments to an older image is not supported. In-cluster OSS and Radar Cloud
viewers should ask their installation’s operator to update Helm/GitOps settings,
not try to configure a private scope for their own charts.
Desktop supports automatic matching, but its separate flag parser and Settings
do not expose these overrides. If an explicit assertion is required, use the
standalone CLI. These flags have no config.json keys or environment-variable
equivalents. See configuration by run mode.
Normal startup reinitialization and unchanged settings saves preserve the assertion.
Changing the Kubernetes endpoint/auth configuration under the same context name
discards it too. New resource queries require cAdvisor container identity labels;
they do not use the older charts’ container-label-free fallback.
Reconstructing a kubeconfig proxy-url callback on laptop reconnect also discards
the assertion conservatively. The UI reports this loss; restart to reassert it.
Explicit assertions are labeled in the UI and do not bypass the request charts’
per-evaluation checks for multiple known replica/job populations.
Historical scope and query limits
Historical queries join metrics with retained ownership at each timestamp. They do not enumerate today’s Pod names; query size does not grow with replica count. CPU/memory/throttling return workload total (weighted ratio for throttling) and maximum Pod, not stacked. Reporting counts have no denominator of today’s replicas or all owned Pods: neither proves instrumentation completeness. The separate current comparison/fallback remains capped at 100 Pods; it never samples a claimed historical total. Charts have about 360 evaluations, three concurrent chart queries and a 25-second request deadline. Timestamps align to the evaluation step. Raw HTTP route, method and peer labels are not returned. Workload range queries and attribution probes use POST form bodies and a 16,000-byte decoded query limit. Current-only name/UID queries can reach it before 100 Pods, especially Istio’s repeated histogram checks. Historical joins avoid replica-dependent growth but retain backend evaluation cost and deadline/response/query bounds. A failed historical query is an error, not permission to silently change scope. Independent current-Pod comparison and explicitly current-only HTTP observations remain usable when the ownership lookup fails. While current attribution is pending, comparison shows a matching-in-progress state and refreshes every three seconds. When identity-checked CPU/memory charts cannot be served, the existing basic charts remain in a collapsed Basic CPU / memory · identity unverified section (the label lists only the affected charts). They do not establish Pod UID, historical workload membership, or cluster identity; matching names in a shared backend may include another cluster. They never substitute for a verified workload total. Network/storage retain this same older name-matched contract independently. HTTP error and coverage ratios are calculated from matching timestamps in the returned counters; histogram quantiles remain calculated by the metrics backend. Warnings orisPartial: true withhold the response; informational annotations
alone do not. Unknown HA label conventions still require upstream deduplication.
Why the population and query bounds exist
Current Pods are enumerated for identity anchors, comparison and explicit fallback, not historical aggregation. Their 100-Pod cap bounds evidence and rows; it is not a Prometheus limit. Historical aggregation covers the full matched population in the backend. Latency combines buckets before deriving quantiles; averaging Pod p95s would be incorrect. Unknown HA labels require upstream deduplication, and large-backend performance is not proven by the small lab.Historical Pods and existing resource queries
Raw KSM and standard recording rules share one ownership contract. Radar selects the strategy reaching the earliest observation in the requested range, then the one covering more timestamps; raw wins equal coverage. One strategy is used per response, not a union or current-Pod fill. This handles newly installed rules and unequal retention but cannot recover unretained data. Rightsizing reuses the same ownership builder without changing its scope contract. Istio history needs ownership too: native names do not distinguish Kubernetes kind, and today’s lack of a collision cannot prove past uniqueness. Historical queries also select native destination workload name/namespace. Missing ownership names the KSM prerequisite; waypoint mapping remains unsupported. When ownership stops reporting after deletion, that Pod leaves the aggregate even if its final rate window still has samples. Earlier retained points remain queryable after rollout/zero if historical cluster scope can be established. Where history has no usable samples but current identity does, the affected family can show current Pods only, visibly labeled, never spliced into historical gaps. Current means still present and owned, not only Ready. The existing network, filesystem and restart queries use exact current Pod names and namespace, not fuzzy workload-name prefixes. They still lack the new UID and verified cluster-partition matching: a same-name Pod in a shared store or a prior incarnation can affect historical values. They are separate context, not proof of the new panels’ population. Network/storage stay in the lower, explicitly labeled section for this release. Extending identity checks requires probing each metric family; CPU’s usableid/container labels cannot be assumed on Pod-network or
filesystem series.
Unsupported sources and formats, in plain language
- Ingress metrics: observations made by a reverse proxy before requests reach application Pods. They can be useful without Beyla/mesh, but describe routed traffic and proxy failures, not every request or task handled by the workload. No ingress-controller metric adapter is included here.
- Native-only histograms: another Prometheus representation of latency data,
stored in histogram samples instead of separate
_bucket{le=...}series. The current latency queries use classic buckets. A backend can support PromQL yet lack the classic series these charts need. Separate counters, where present, can still support rate/errors; a native-only Beyla histogram may also lack the_countseries the current request adapter expects. Native support must cover discovery, count extraction, quantiles and population checks together. - Istio waypoints: shared proxies used in ambient mesh. The scraped proxy Pod is not the application Pod whose traffic it observes. The current destination sidecar mapping cannot be reused by simply accepting another reporter value.
Request sources
- Istio destination sidecars:
istio_requests_totalandistio_request_duration_milliseconds_bucket,reporter="destination",request_protocol="http". Requires scrapenamespaceandpodlabels identifying the destination Pod, as well asdestination_workload_namespace. This avoids attributing metrics solely from a workload name, which has no kind or UID. Waypoint reporters cannot use this mapping and are not currently included. - Beyla HTTP server:
http_server_request_duration_seconds_countand_bucket, withk8s_namespace_nameandk8s_pod_name. Automatic attribution discovers an exact job from matching identities, including custom job names. With an explicit scope override, the current-Pod path uses its Beyla/Alloy job discriminator;--beyla-job-selectoraccepts one exact or regexjobmatcher. For workload metrics that override is paired with a verified scope assertion; historical queries otherwise consider matching HTTP-server jobs across the range and withhold ambiguous observation populations instead of picking a job from today’s Pods. For example,--beyla-job-selector 'job="primary-beyla"'selects one observer when paired with an appropriate scope flag. The pre-existing Live Traffic integration also accepts wider matcher fragments; those are not supported by the workload adapter. The flag alone does not filter automatic workload attribution. Do not assert an unverified cluster scope just to select a job. Application-only Beyla works without network metrics: verified with direct exposition in kind and EKS nonprod (Beyla 3.32.0). Default OTLP-to-Alloy conversion was also tested and lacks Pod attributes on the request series; that pipeline is not supported by this adapter.
Workload metrics HTTP API
GET /api/prometheus/workload/{kind}/{namespace}/{name}?range=1h&source=beyla
returns the curated workload dashboard. It does not accept arbitrary PromQL,
backend URLs, credentials or cluster-scope overrides from chart requests.
The response describes both data and coverage:
Panel states are
available, partial, stale, unavailable, error, and
detecting. partial can mean missing coverage; stale means only older usable
evaluations remain. unavailable is not measured zero, and error is not absence
of instrumentation. detecting means identity matching is in progress.
Top-level state summarizes collection and does not aggregate all panel states.
An early identity-detection response can have empty panel/history/comparison
maps and sources, with numeric fields serialized as zero. Read state first:
those zeros are not a measured zero-Pod result or an epoch-zero query window.
reason, scopeNotice, source labels and attribution text are display copy,
not stable machine-readable reason codes.
HTTP errors use {"error":"..."}. A 403 can mean either the caller lacks the
required workload/Pod permissions, or Radar’s own cache cannot cover the Pod
and controller data in that namespace. The latter needs an operator to check
Radar’s Kubernetes permissions and configured namespace scope, not necessarily
change the viewer’s permissions. A 503 means the connection/client/cache is not ready; a
502 can indicate workload-Pod resolution failed. A 409 means the metrics
connection changed during collection: retry against the new connection.
Upstream query failures are generally reported in panel or collection state,
not as HTTP errors; backend authentication failures are distinct from Radar’s
Kubernetes authorization gate.
Live-tested coverage
Validation on 2026-09-13–14 exercised the following combinations. A successful query endpoint does not imply that every chart has the necessary metrics. These checks cover specific versions and collection configurations, not every installation of a backend or a measured percentage of Kubernetes users. The fleet matrix originally validated current-Pod attribution. Historical ownership was additionally validated in the repeatable kind lab: Beyla and Istio retained identical CPU, memory, throttling, request, error and latency values at fixed timestamps after rollout/deletion and scale-to-zero. This is not a retroactive history certification of every cloud row. Historical resource and RED charts were also rechecked on the GKE VictoriaMetrics/Istio fixture and two EKS Mimir/Beyla workloads without scope assertions. The GKE fixture produced positive requests, approximately 25% injected 5xx responses, and defined p50/p95; its worker and Redis retained resource charts without fabricated HTTP charts. Cloud workloads were not rolled or scaled for this check. A real-Prometheus fixture verifies aggregate totals for 128 Pods; the live lab itself remains small.
Non-HTTP workers and Redis correctly retain resource panels when available,
without an empty HTTP chart grid. A network-only VictoriaMetrics store correctly
has no workload observations. Authenticated Mimir rejects missing/wrong
credentials and a missing required tenant in gateway-level checks, not browser
tests of Radar’s auth error state. A separate unhealthy Mimir store
failed real queries despite having Ready Pods; readiness is not query health.
Configuration and remaining gaps
- The tested VictoriaMetrics Kubernetes chart (0.92.1) drops cAdvisor
idand KSMuidby default. Preserve those fields for direct UID / KSM-backed attribution. The successful VM test used that explicit configuration; it is not a stock-default OOTB claim. Copying every GKE node label also exceeded VM’s label-per-series limit in the lab; collector label filtering was needed to prevent dropped data. - The baseline Istio latency gap came from comparing independently updated
request counters and histograms. Latency now matches their complete label
populations and validates each histogram count against its
+Infbucket; it does not require the separate request counter to have advanced equally. Missing populations, mismatched bucket layouts and duplicate jobs/replicas still withhold affected samples. Histograms can briefly trail request counters. - Explicitly configured endpoints can return an empty
upvector and still connect; filtered remote-write stores do not have to retain scrape metadata. Automatic discovery still skips empty candidates. Connected means the query API answered, not that the selected tenant contains this workload’s data. Authentication, backend query errors and invalid query responses have distinct diagnostics. HTTP 405 explains that workload queries require POST through proxies. - Live checks primarily used one or two replicas, including a single-node DaemonSet. Large workloads, HA duplicate populations and adversarial cross-cluster name collisions are not live-certified by this matrix. Separate query tests cover several identity/duplicate cases; that is not equivalent to a fleet test.
- These runs used local Radar with Kubernetes port-forward access. They do not certify Radar Cloud transport, managed-service authentication or every in-cluster deployment path. A full browser sweep was not repeated for each backend.