Metrics and Grafana
Read your instances' CPU, memory, disk, network and GPU metrics in Grafana or with PromQL.
Our hosts measure every instance from the outside: CPU time, memory, disk and network, with no agent in the guest. A GPU instance can also report its own GPUs, which no host can read once they are passed through to it: utilization, memory, temperature, power, clocks and errors. That is off unless you turn it on for the instance (GPU metrics). Nothing else is collected from inside an instance.
You can read the metrics three ways:
| Way | Where | Needs |
|---|---|---|
| Your organization's Grafana | metrics.confidential.ai | Sign-in, as in the console |
| Prometheus query API | https://api.cloud.confidential.ai/v1/prometheus | A credential with instances:read |
| MCP tools | query_metrics, query_metrics_range through the MCP server | An agent connection |
Points are kept every 30 seconds for 3 days, and every 5 minutes for 14 days.
Your organization's Grafana
Open Observability in the console, or go to metrics.confidential.ai. You sign in the same way as in the console and land in your organization's own Grafana organization, which already has an Instances dashboard for all your instances, and a Clusters dashboard of your managed clusters: their nodes, by cluster, pool and node, and each cluster's control plane. An instance's page in the console links to the same dashboard filtered to that instance (Metrics).
Owners and members can edit and add dashboards; viewers can read them. Each Grafana organization reads only its own organization's metrics.
The series
Every series is labelled with instance (the instance's id) and project. Each cloud_instance_gpu_* series is also labelled with gpu, the GPU's index in the instance, and source="guest", since the instance reports it; they exist only for instances with GPU metrics on.
| Series | What |
|---|---|
cloud_instance_cpu_seconds_total | CPU time the instance's vCPUs ran, in seconds |
cloud_instance_cpu_steal_seconds_total | Time its vCPUs were ready to run but waited for the host, in seconds |
cloud_instance_memory_domain_bytes | Memory the instance was given |
cloud_instance_memory_resident_bytes | Memory the instance holds on its host |
cloud_instance_network_receive_bytes_total, cloud_instance_network_transmit_bytes_total | Bytes in and out |
cloud_instance_network_receive_packets_total, cloud_instance_network_transmit_packets_total | Packets in and out |
cloud_instance_network_receive_errors_total, cloud_instance_network_transmit_errors_total | Packets in and out that failed |
cloud_instance_network_receive_drops_total, cloud_instance_network_transmit_drops_total | Packets in and out that were dropped |
cloud_instance_disk_read_bytes_total, cloud_instance_disk_write_bytes_total | Bytes read from and written to its disks |
cloud_instance_disk_read_ops_total, cloud_instance_disk_write_ops_total | Read and write operations |
cloud_instance_disk_read_seconds_total, cloud_instance_disk_write_seconds_total | Time its reads and writes took to complete, in seconds; divide by the operations for latency |
cloud_instance_disk_flush_ops_total, cloud_instance_disk_flush_seconds_total | Cache flushes, and the time they took |
cloud_instance_gpus | GPUs passed through to the instance |
cloud_instance_gpu_utilization_ratio, cloud_instance_gpu_memory_utilization_ratio | Share of the time each GPU was computing, and was reading or writing its memory, 0 to 1 |
cloud_instance_gpu_memory_used_bytes, cloud_instance_gpu_memory_free_bytes | Each GPU's memory in use and free |
cloud_instance_gpu_temperature_celsius, cloud_instance_gpu_memory_temperature_celsius | Each GPU's temperature and its memory's |
cloud_instance_gpu_power_watts, cloud_instance_gpu_energy_joules_total | Power each GPU draws, and the energy it has used |
cloud_instance_gpu_sm_clock_hertz, cloud_instance_gpu_memory_clock_hertz, cloud_instance_gpu_throttle_reasons | Each GPU's clocks, and the driver's bitmask of why it is slowed |
cloud_instance_gpu_ecc_corrected_errors_total, cloud_instance_gpu_ecc_uncorrected_errors_total | ECC errors since the driver loaded |
cloud_instance_gpu_remapped_rows, cloud_instance_gpu_row_remap_failed | Memory rows remapped, by kind (correctable, uncorrectable), and 1 if a remap failed |
cloud_instance_gpu_pcie_replays_total, cloud_instance_gpu_last_xid | PCIe replays, and the last XID error the GPU raised since boot (0 for none) |
Useful queries:
# CPU in use, in vCPUs
sum by (instance) (rate(cloud_instance_cpu_seconds_total[5m]))
# Network out, in bytes per second
sum by (instance) (rate(cloud_instance_network_transmit_bytes_total[5m]))
# Disk writes per second
sum by (instance) (rate(cloud_instance_disk_write_ops_total[5m]))
# Average GPU utilization of each GPU instance, 0 to 1
avg by (instance) (cloud_instance_gpu_utilization_ratio)Managed clusters' control planes
A managed cluster's control plane runs outside the confidential boundary, and our hosts measure its etcd, API server, scheduler and controller manager. These series are labelled with cluster_id (the cluster's id) and project, and a key reads them only if it holds clusters:read as well as instances:read:
| Series | What |
|---|---|
cloud_cluster_etcd_has_leader | 1 while every member of the cluster's etcd that answers sees a leader; absent while none answers, so watch cloud_cluster_etcd_members_up for that |
cloud_cluster_etcd_members_up | Members of its etcd that answer; 0 while none does |
cloud_cluster_etcd_db_size_bytes, cloud_cluster_etcd_quota_bytes | Its etcd's database size, and the size past which it refuses writes |
cloud_cluster_etcd_wal_fsync_seconds_p99 | How long its slowest member's WAL fsyncs took at the 99th percentile over 5 minutes, in seconds |
cloud_cluster_apiserver_members_up, cloud_cluster_scheduler_members_up, cloud_cluster_controller_manager_members_up | Its API servers, schedulers and controller managers that answer; 0 while none does |
cloud_cluster_scheduler_has_leader, cloud_cluster_controller_manager_has_leader | 1 while one of its schedulers, or controller managers, leads; only the leader schedules pods or runs controllers |
cloud_cluster_apiserver_requests_per_second | Requests a second to its API server over 5 minutes, by verb and HTTP code |
cloud_cluster_apiserver_request_seconds_p99 | How long its API server took to answer at the 99th percentile over 5 minutes, by verb, in seconds, in steps of 0.1, 0.2, 0.4, 1, 2, 5 and 10; watches and the time webhooks take are left out |
cloud_cluster_scheduler_pending_pods | Pods waiting to be scheduled, by the scheduler's queue (unschedulable holds those that fit no node) |
cloud_cluster_controller_manager_workqueue_depth | Changes waiting for each controller, by its name |
cloud_cluster_info | 1 for each cluster, with its name as cluster_name, to name clusters by in place of cluster_id |
# How full each cluster's etcd is, 0 to 1
cloud_cluster_etcd_db_size_bytes / cloud_cluster_etcd_quota_bytes
# Share of each cluster's API server requests that failed, 0 to 1
sum by (cluster_id) (cloud_cluster_apiserver_requests_per_second{code=~"5.."})
/ sum by (cluster_id) (cloud_cluster_apiserver_requests_per_second)GPU metrics
GPU metrics are off by default. We read an instance's GPU figures only if you turn them on for that instance: set "gpuMetrics": true when you create it, or later with PATCH /v1/instances/{id}, and set it back to false to stop. The change takes effect while the instance runs. Only sizes with GPUs take it: true with a CPU size returns 422 unsupported_combination.
curl -X PATCH https://api.cloud.confidential.ai/v1/instances/$ID \
-H "Authorization: Bearer $KEY" -H "Api-Version: 2026-10-15" \
-H "Content-Type: application/json" -d '{"gpuMetrics": true}'Only an image that ships NVIDIA's GPU metrics exporter serves the figures: confos-gpu from its next version. Its OS reads them from NVIDIA's driver every 30 seconds and serves them on port 9400 inside the instance, so your workload can't listen on 9400 there. With GPU metrics on, the instance's network policy lets our metrics collector in to that port; with them off, it does not. Your own software inside the instance can always read http://localhost:9400/metrics itself, whether or not they are on. On an image without the exporter, turning them on shows no GPU figures. In confidential mode NVIDIA turns off the GPUs' profiling counters, so per-kernel activity, tensor-core use and NVLink bandwidth are not available.
The figures are what the instance says. We use them only to show you your GPUs, never for billing, placement or a host's health.
Query the API
/v1/prometheus is a read-only Prometheus HTTP API. It keeps Prometheus's own paths and parameters, so any Prometheus client can call it:
curl -G https://api.cloud.confidential.ai/v1/prometheus/api/v1/query \
-H "Authorization: Bearer $CC_API_KEY" -H "Api-Version: 2026-10-15" \
--data-urlencode 'query=sum by (instance) (rate(cloud_instance_cpu_seconds_total[5m]))'{
"status": "success",
"data": {
"resultType": "vector",
"result": [ { "metric": { "instance": "i-9pj5dy9i6cbu" }, "value": [1792460329, "0.0458"] } ]
}
}| Path | Does |
|---|---|
/api/v1/query | Evaluates a PromQL query at one time (now by default) |
/api/v1/query_range | Evaluates it at every step from start to end |
/api/v1/series | Lists the series that match match[] |
/api/v1/labels, /api/v1/label/{name}/values | Lists label names and values |
Every query reads your organization's series and nothing else, whatever it names. Queries run for at most 20 seconds.
Use your own Grafana
To chart the metrics in a Grafana of your own, add a Prometheus data source:
| Setting | Value |
|---|---|
| Prometheus server URL | https://api.cloud.confidential.ai/v1/prometheus |
| HTTP method | GET |
Custom header Authorization | Bearer cc_..., an API key with View instances |
Custom header Api-Version | 2026-10-15 |
Use a standard key made only for this, with no other permission, and revoke it in the console if the Grafana is ever exposed.