Docs

Metrics and Grafana

Read your instances' CPU, memory, disk, network and GPU metrics in Grafana or with PromQL.

Our hosts measure every instance from the outside: CPU time, memory, disk and network, with no agent in the guest. A GPU instance can also report its own GPUs, which no host can read once they are passed through to it: utilization, memory, temperature, power, clocks and errors. That is off unless you turn it on for the instance (GPU metrics). Nothing else is collected from inside an instance.

You can read the metrics three ways:

WayWhereNeeds
Your organization's Grafanametrics.confidential.aiSign-in, as in the console
Prometheus query APIhttps://api.cloud.confidential.ai/v1/prometheusA credential with instances:read
MCP toolsquery_metrics, query_metrics_range through the MCP serverAn agent connection

Points are kept every 30 seconds for 3 days, and every 5 minutes for 14 days.

Your organization's Grafana

Open Observability in the console, or go to metrics.confidential.ai. You sign in the same way as in the console and land in your organization's own Grafana organization, which already has an Instances dashboard for all your instances, and a Clusters dashboard of your managed clusters: their nodes, by cluster, pool and node, and each cluster's control plane. An instance's page in the console links to the same dashboard filtered to that instance (Metrics).

Owners and members can edit and add dashboards; viewers can read them. Each Grafana organization reads only its own organization's metrics.

The series

Every series is labelled with instance (the instance's id) and project. Each cloud_instance_gpu_* series is also labelled with gpu, the GPU's index in the instance, and source="guest", since the instance reports it; they exist only for instances with GPU metrics on.

SeriesWhat
cloud_instance_cpu_seconds_totalCPU time the instance's vCPUs ran, in seconds
cloud_instance_cpu_steal_seconds_totalTime its vCPUs were ready to run but waited for the host, in seconds
cloud_instance_memory_domain_bytesMemory the instance was given
cloud_instance_memory_resident_bytesMemory the instance holds on its host
cloud_instance_network_receive_bytes_total, cloud_instance_network_transmit_bytes_totalBytes in and out
cloud_instance_network_receive_packets_total, cloud_instance_network_transmit_packets_totalPackets in and out
cloud_instance_network_receive_errors_total, cloud_instance_network_transmit_errors_totalPackets in and out that failed
cloud_instance_network_receive_drops_total, cloud_instance_network_transmit_drops_totalPackets in and out that were dropped
cloud_instance_disk_read_bytes_total, cloud_instance_disk_write_bytes_totalBytes read from and written to its disks
cloud_instance_disk_read_ops_total, cloud_instance_disk_write_ops_totalRead and write operations
cloud_instance_disk_read_seconds_total, cloud_instance_disk_write_seconds_totalTime its reads and writes took to complete, in seconds; divide by the operations for latency
cloud_instance_disk_flush_ops_total, cloud_instance_disk_flush_seconds_totalCache flushes, and the time they took
cloud_instance_gpusGPUs passed through to the instance
cloud_instance_gpu_utilization_ratio, cloud_instance_gpu_memory_utilization_ratioShare of the time each GPU was computing, and was reading or writing its memory, 0 to 1
cloud_instance_gpu_memory_used_bytes, cloud_instance_gpu_memory_free_bytesEach GPU's memory in use and free
cloud_instance_gpu_temperature_celsius, cloud_instance_gpu_memory_temperature_celsiusEach GPU's temperature and its memory's
cloud_instance_gpu_power_watts, cloud_instance_gpu_energy_joules_totalPower each GPU draws, and the energy it has used
cloud_instance_gpu_sm_clock_hertz, cloud_instance_gpu_memory_clock_hertz, cloud_instance_gpu_throttle_reasonsEach GPU's clocks, and the driver's bitmask of why it is slowed
cloud_instance_gpu_ecc_corrected_errors_total, cloud_instance_gpu_ecc_uncorrected_errors_totalECC errors since the driver loaded
cloud_instance_gpu_remapped_rows, cloud_instance_gpu_row_remap_failedMemory rows remapped, by kind (correctable, uncorrectable), and 1 if a remap failed
cloud_instance_gpu_pcie_replays_total, cloud_instance_gpu_last_xidPCIe replays, and the last XID error the GPU raised since boot (0 for none)

Useful queries:

# CPU in use, in vCPUs
sum by (instance) (rate(cloud_instance_cpu_seconds_total[5m]))

# Network out, in bytes per second
sum by (instance) (rate(cloud_instance_network_transmit_bytes_total[5m]))

# Disk writes per second
sum by (instance) (rate(cloud_instance_disk_write_ops_total[5m]))

# Average GPU utilization of each GPU instance, 0 to 1
avg by (instance) (cloud_instance_gpu_utilization_ratio)

Managed clusters' control planes

A managed cluster's control plane runs outside the confidential boundary, and our hosts measure its etcd, API server, scheduler and controller manager. These series are labelled with cluster_id (the cluster's id) and project, and a key reads them only if it holds clusters:read as well as instances:read:

SeriesWhat
cloud_cluster_etcd_has_leader1 while every member of the cluster's etcd that answers sees a leader; absent while none answers, so watch cloud_cluster_etcd_members_up for that
cloud_cluster_etcd_members_upMembers of its etcd that answer; 0 while none does
cloud_cluster_etcd_db_size_bytes, cloud_cluster_etcd_quota_bytesIts etcd's database size, and the size past which it refuses writes
cloud_cluster_etcd_wal_fsync_seconds_p99How long its slowest member's WAL fsyncs took at the 99th percentile over 5 minutes, in seconds
cloud_cluster_apiserver_members_up, cloud_cluster_scheduler_members_up, cloud_cluster_controller_manager_members_upIts API servers, schedulers and controller managers that answer; 0 while none does
cloud_cluster_scheduler_has_leader, cloud_cluster_controller_manager_has_leader1 while one of its schedulers, or controller managers, leads; only the leader schedules pods or runs controllers
cloud_cluster_apiserver_requests_per_secondRequests a second to its API server over 5 minutes, by verb and HTTP code
cloud_cluster_apiserver_request_seconds_p99How long its API server took to answer at the 99th percentile over 5 minutes, by verb, in seconds, in steps of 0.1, 0.2, 0.4, 1, 2, 5 and 10; watches and the time webhooks take are left out
cloud_cluster_scheduler_pending_podsPods waiting to be scheduled, by the scheduler's queue (unschedulable holds those that fit no node)
cloud_cluster_controller_manager_workqueue_depthChanges waiting for each controller, by its name
cloud_cluster_info1 for each cluster, with its name as cluster_name, to name clusters by in place of cluster_id
# How full each cluster's etcd is, 0 to 1
cloud_cluster_etcd_db_size_bytes / cloud_cluster_etcd_quota_bytes

# Share of each cluster's API server requests that failed, 0 to 1
sum by (cluster_id) (cloud_cluster_apiserver_requests_per_second{code=~"5.."})
  / sum by (cluster_id) (cloud_cluster_apiserver_requests_per_second)

GPU metrics

GPU metrics are off by default. We read an instance's GPU figures only if you turn them on for that instance: set "gpuMetrics": true when you create it, or later with PATCH /v1/instances/{id}, and set it back to false to stop. The change takes effect while the instance runs. Only sizes with GPUs take it: true with a CPU size returns 422 unsupported_combination.

curl -X PATCH https://api.cloud.confidential.ai/v1/instances/$ID \
  -H "Authorization: Bearer $KEY" -H "Api-Version: 2026-10-15" \
  -H "Content-Type: application/json" -d '{"gpuMetrics": true}'

Only an image that ships NVIDIA's GPU metrics exporter serves the figures: confos-gpu from its next version. Its OS reads them from NVIDIA's driver every 30 seconds and serves them on port 9400 inside the instance, so your workload can't listen on 9400 there. With GPU metrics on, the instance's network policy lets our metrics collector in to that port; with them off, it does not. Your own software inside the instance can always read http://localhost:9400/metrics itself, whether or not they are on. On an image without the exporter, turning them on shows no GPU figures. In confidential mode NVIDIA turns off the GPUs' profiling counters, so per-kernel activity, tensor-core use and NVLink bandwidth are not available.

The figures are what the instance says. We use them only to show you your GPUs, never for billing, placement or a host's health.

Query the API

/v1/prometheus is a read-only Prometheus HTTP API. It keeps Prometheus's own paths and parameters, so any Prometheus client can call it:

curl -G https://api.cloud.confidential.ai/v1/prometheus/api/v1/query \
  -H "Authorization: Bearer $CC_API_KEY" -H "Api-Version: 2026-10-15" \
  --data-urlencode 'query=sum by (instance) (rate(cloud_instance_cpu_seconds_total[5m]))'
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [ { "metric": { "instance": "i-9pj5dy9i6cbu" }, "value": [1792460329, "0.0458"] } ]
  }
}
PathDoes
/api/v1/queryEvaluates a PromQL query at one time (now by default)
/api/v1/query_rangeEvaluates it at every step from start to end
/api/v1/seriesLists the series that match match[]
/api/v1/labels, /api/v1/label/{name}/valuesLists label names and values

Every query reads your organization's series and nothing else, whatever it names. Queries run for at most 20 seconds.

Use your own Grafana

To chart the metrics in a Grafana of your own, add a Prometheus data source:

SettingValue
Prometheus server URLhttps://api.cloud.confidential.ai/v1/prometheus
HTTP methodGET
Custom header AuthorizationBearer cc_..., an API key with View instances
Custom header Api-Version2026-10-15

Use a standard key made only for this, with no other permission, and revoke it in the console if the Grafana is ever exposed.

On this page