> ## Documentation Index
> Fetch the complete documentation index at: https://knowledge.cloudquant.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Grafana

> Connect your Grafana instance to CloudQuant Data Liberator's Prometheus stack using a rotatable Bearer token

# Grafana integration

CloudQuant Data Liberator exposes a curated, read-only slice of its internal Prometheus stack at a Bearer-authenticated endpoint so you can wire your own Grafana (or any Prometheus-API-compatible client) to the same metrics that power the in-product **System Monitoring** view.

This integration is intended for ops teams, on-call rotations, and infrastructure dashboards that need to live alongside metrics from systems outside CloudQuant.

## At a glance

|                      |                                                                                                                                                                                   |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Endpoint**         | `https://<your-liberator-host>/metrics-api-bearer/`                                                                                                                               |
| **Authentication**   | `Authorization: Bearer cqm_<48-hex-chars>`                                                                                                                                        |
| **Protocol**         | Standard Prometheus HTTP API (`/api/v1/query`, `/api/v1/query_range`, `/api/v1/series`, `/api/v1/labels`, `/api/v1/label/<name>/values`, `/api/v1/status/buildinfo`, `/federate`) |
| **Surface**          | Read-only. Admin endpoints (`/api/v1/admin/*`) and remote-write are gated off at the Prometheus server, not just at the gateway.                                                  |
| **Token management** | Liberator UI → **System Monitoring → Grafana Integration** tab                                                                                                                    |

## Prerequisites

* A user with **super-admin** privileges on your Liberator instance (token management is a super-admin operation).
* Network reachability from your Grafana host to your Liberator hostname over HTTPS.
* Grafana 9.x or newer (any version that supports custom HTTP headers on the Prometheus datasource).

## Step 1 — Issue a Bearer token

1. Sign in to the Liberator UI as a super-admin.
2. Open **System Monitoring** from the top navigation.
3. Click the **Grafana Integration** action in the upper-right of the page.
4. Click **Generate token**. The new token is shown **exactly once**, so copy it immediately.

Tokens follow the format `cqm_` + 48 hex characters. Every token issued through this dialog is logged with the issuing user and timestamp; tokens can be listed and revoked from the same dialog.

<Warning>
  The full token value is only visible at issue time. Treat it like a password: store it in your secrets manager and never commit it to source control. If a token is lost, revoke it and issue a new one rather than trying to recover it.
</Warning>

<Tip>
  Issue one token per logical consumer (e.g. `grafana-prod`, `federated-prometheus-eu`) so you can revoke a single consumer's access without disrupting others.
</Tip>

## Step 2 — Configure the Grafana datasource

### Through the Grafana UI

1. **Connections → Data sources → Add data source → Prometheus.**

2. Set the fields below:

   | Field                     | Value                                                   |
   | ------------------------- | ------------------------------------------------------- |
   | **Name**                  | `cloudquant-liberator` (or any label you prefer)        |
   | **Prometheus server URL** | `https://<your-liberator-host>/metrics-api-bearer`      |
   | **HTTP method**           | `POST` (recommended, handles longer queries)            |
   | **Scrape interval**       | `30s` (matches the upstream Prometheus scrape interval) |

3. Under **Custom HTTP Headers**, click **+ Add header** and set:

   * **Header** = `Authorization`
   * **Value** = `Bearer cqm_<your-token>`

4. Click **Save & test**. You should see **"Successfully queried the Prometheus API."**

### Through provisioning (recommended for production)

For reproducible setups, drop this file at `/etc/grafana/provisioning/datasources/cloudquant.yaml`:

```yaml theme={null}
apiVersion: 1
datasources:
  - name: cloudquant-liberator
    uid: cloudquant-liberator
    type: prometheus
    access: proxy
    url: https://<your-liberator-host>/metrics-api-bearer
    isDefault: true
    editable: true
    jsonData:
      httpMethod: POST
      timeInterval: 30s
      prometheusType: Prometheus
      httpHeaderName1: Authorization
    secureJsonData:
      httpHeaderValue1: "Bearer cqm_<your-token>"
```

Restart Grafana (or send `SIGHUP`) to pick up the file. The token lives in `secureJsonData`, which Grafana persists encrypted at rest.

## Step 3 — Run a query

Once the datasource is healthy, you can browse it in **Explore** or build dashboards against it. A few queries to verify end-to-end connectivity:

```promql theme={null}
# All scrape targets currently up
sum(up)

# Request rate by HTTP status class through the Liberator gateway
sum by (envoy_response_code_class) (rate(envoy_http_downstream_rq_xx[5m]))

# Liberator waiting-room active connections
liberator_waitingroom_connections_active

# p95 upstream request latency
histogram_quantile(
  0.95,
  sum by (le) (rate(envoy_cluster_external_upstream_rq_time_bucket[5m]))
)
```

## What's exposed

The Bearer endpoint forwards to the same Prometheus server that powers the in-product **System Monitoring** view, so anything you see there is also queryable here. Notable metric families:

| Prefix            | What it covers                                                                     |
| ----------------- | ---------------------------------------------------------------------------------- |
| `envoy_*`         | Envoy Gateway — request rates, latency histograms, upstream connection health      |
| `liberator_*`     | Liberator application — waiting-room queue, in-flight query times, exporter health |
| `cache_manager_*` | Cache-manager worker pool status and error counters                                |
| `kube_*`          | kube-state-metrics — pods, deployments, nodes, conditions                          |
| `container_*`     | cAdvisor — per-container CPU, memory, filesystem, network                          |
| `node_*`          | node-exporter — host CPU, memory, disk, network                                    |
| `prometheus_*`    | The Prometheus server itself — useful for sizing dashboards                        |

For a complete metric inventory in your environment, query `/api/v1/label/__name__/values` once you've connected.

## Rotating a token

1. Open the **Grafana Integration** dialog as in Step 1.
2. Issue a **new** token for the same logical consumer.
3. Update the new token in Grafana (UI: edit the datasource → swap the `Authorization` header value; provisioning: replace `httpHeaderValue1` and restart).
4. Wait until your Grafana datasource health check passes with the new token.
5. **Revoke** the old token from the same dialog.

This zero-downtime pattern is the same as the [AWS access-key rotation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_RotateAccessKey) workflow.

## Limitations

* **Read-only.** Prometheus admin endpoints (`/api/v1/admin/tsdb/*`) and the remote-write receiver are not exposed. Even tokened callers cannot mutate the TSDB through this endpoint.
* **One Prometheus.** The endpoint serves the Liberator cluster's primary Prometheus only. Federated views across multiple Liberator clusters require Prometheus-side federation on the consumer side; use `/metrics-api-bearer/federate` with appropriate `match[]` parameters.
* **CORS.** Browser-side direct calls are not supported (the gateway only allows server-side calls). Grafana proxies through its backend, so this restriction does not affect dashboard use.

## Troubleshooting

| Symptom                                                  | Diagnosis                                                                                                                    | Resolution                                                                                                                                         |
| -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Save & test` returns *"non-JSON: `<!DOCTYPE html>...`"* | The request is being redirected to the OIDC login page — the `Authorization` header isn't being sent, or the token is wrong. | Verify the header name is exactly `Authorization` and the value begins with `Bearer ` (with a trailing space) before the token.                    |
| `Save & test` returns *401 Unauthorized*                 | Token has been revoked, never issued, or has a typo.                                                                         | Re-issue from the **Grafana Integration** dialog and update the datasource.                                                                        |
| `Save & test` returns *403 Forbidden*                    | The Liberator gateway rejected the request before reaching Prometheus.                                                       | Check with your CloudQuant administrator that the bearer route is enabled on your cluster.                                                         |
| Health check passes but every query returns *"no data"*  | Time range is outside Prometheus's retention window, or the metric name has changed in a recent upgrade.                     | Open **Explore**, hit `/api/v1/label/__name__/values` against the datasource, and confirm the metric exists.                                       |
| Range queries fail with *"context deadline exceeded"*    | Query window × resolution exceeds the gateway's response timeout.                                                            | Use the in-product **System Monitoring** view as a sanity check on the query, then widen your Grafana panel's `interval` or narrow the time range. |