Careful!
You are browsing documentation for the next version of Kuma. Use this version at your own risk.
MeshMetric
This policy uses new policy matching algorithm. Do not combine with Traffic Metrics.
Kuma facilitates consistent traffic metrics across all data plane proxies in your mesh.
You can define metrics configuration for a whole Mesh, and optionally tweak certain parts for individual data plane proxies. For example, you might need to override the default metrics port if it’s already in use on the specified machine.
Kuma provides full integration with Prometheus:
- Each proxy can expose its metrics in Prometheus format.
- Kuma exposes an API called the monitoring assignment service (MADS) which exposes proxies configured by
MeshMetric
.
To collect metrics from Kuma, you need to expose metrics from proxies and applications.
In the rest of this page we assume you have already configured your observability tools to work with Kuma. If you haven’t already read the observability docs.
TargetRef support matrix
TargetRef type | top level | to | from |
---|---|---|---|
Mesh | ✅ | ❌ | ❌ |
MeshSubset | ✅ | ❌ | ❌ |
MeshService | ✅ | ❌ | ❌ |
MeshServiceSubset | ✅ | ❌ | ❌ |
To learn more about the information in this table, see the matching docs.
Configuration
There are three main sections of the configuration: sidecar
, applications
, backends
.
The first two define how to scrape parts of the mesh (sidecar and underlying applications), the third one defines what to do with the data (in case of Prometheus instructs to scrape specific address, in case of OpenTelemetry defines where to push data).
In contrast to Traffic Metrics all configuration is dynamic and no restarts of the Data Plane Proxies are needed.
You can define configuration refresh interval by using KUMA_DATAPLANE_RUNTIME_DYNAMIC_CONFIGURATION_REFRESH_INTERVAL
env var or dataplaneRuntime.dynamicConfiguration.refreshInterval
Helm value.
Sidecar
This part of the configuration applies to the data plane proxy scraping.
In case you don’t want to retrieve all Envoy’s metrics, it’s possible to filter them.
You are able to specify regex
which causes the metrics endpoint to return only matching metrics.
Also, you can set flag usedOnly
that returns only metrics updated by Envoy.
Example section of the configuration:
sidecar:
regex: http2_act.*
usedOnly: true
Applications
In addition to exposing metrics from the data plane proxies, you might want to expose metrics from applications running next to the proxies.
Kuma allows scraping Prometheus metrics from the applications endpoint running in the same Pod
or VM
.
Later those metrics are aggregated and exposed at the same port/path
as data plane proxy metrics.
It is possible to configure it at the Mesh
level, for all the applications in the Mesh
, or just for specific applications.
Here are reasons where you’d want to use this feature:
- Application metrics are labelled with your mesh parameters (tags, mesh, data plane name…), this means that in mixed Universal and Kubernetes mode metrics are reported with the same types of labels.
- Both application and sidecar metrics are scraped at the same time. This makes sure they are coherent (with 2 different scrapers they can end up scraping at different intervals and make metrics harder to correlate).
- If you disable passthrough and your mesh uses mTLS and Prometheus is outside the mesh this is the only way to retrieve these metrics as the app is completely hidden behind the sidecar.
Example section of the configuration:
applications:
- path: "/metrics/prometheus"
address: # optional custom address if the underlying application listens on a different address than the Data Plane Proxy
port: 8888
Backends
backends:
- type: Prometheus
prometheus:
port: 5670
path: /metrics
This tells Kuma to expose an HTTP endpoint with Prometheus metrics on port 5670
and URI path /metrics
.
The metrics endpoint is forwarded to the standard Envoy Prometheus metrics endpoint and supports the same query parameters.
You can pass the filter
query parameter to limit the results to metrics whose names match a given regular expression.
By default, all available metrics are returned.
Secure metrics with TLS
Kuma allows configuring metrics endpoint with TLS.
backends:
- type: Prometheus
prometheus:
port: 5670
path: /metrics
tls:
mode: ProvidedTLS
In addition to the MeshMetric
configuration, kuma-sidecar
requires a provided certificate and key for its operation.
When the certificate and key are available within the container, kuma-sidecar
needs the paths to provided files as the following environment variables:
KUMA_DATAPLANE_RUNTIME_METRICS_CERT_PATH
KUMA_DATAPLANE_RUNTIME_METRICS_KEY_PATH
It’s possible to use a ContainerPatch
to add variables to kuma-sidecar
:
apiVersion: kuma.io/v1alpha1
kind: ContainerPatch
metadata:
name: container-patch-1
namespace: kuma-system
spec:
sidecarPatch:
- op: add
path: /env/-
value: '{
"name": "KUMA_DATAPLANE_RUNTIME_METRICS_CERT_PATH",
"value": "/kuma/server.crt"
}'
- op: add
path: /env/-
value: '{
"name": "KUMA_DATAPLANE_RUNTIME_METRICS_KEY_PATH",
"value": "/kuma/server.key"
}'
activeMTLSBackend
We no longer support activeMTLSBackend, if you need to encrypt and authorize the metrics use Secure metrics with TLS with a combination of one of the authorization methods.
Running multiple prometheus instances
If you need to run multiple instances of Prometheus and want to target different set of Data Plane Proxies you can do this by using Client ID setting on both MeshMetric
(clientId
) and Prometheus configuration (client_id
).
To use clientId
setting you need to be running at least Prometheus 2.49.0
.
Example configurations differentiated by prometheus
tag:
apiVersion: kuma.io/v1alpha1
kind: MeshMetric
metadata:
name: prometheus-one
namespace: kuma-system
labels:
kuma.io/mesh: default
spec:
targetRef:
kind: MeshSubset
tags:
prometheus: one
backends:
- type: Prometheus
prometheus:
clientId: prometheus-one
port: 5670
path: "/metrics"
apiVersion: kuma.io/v1alpha1
kind: MeshMetric
metadata:
name: prometheus-two
namespace: kuma-system
labels:
kuma.io/mesh: default
spec:
targetRef:
kind: MeshSubset
tags:
prometheus: two
backends:
- type: Prometheus
prometheus:
clientId: prometheus-two
port: 5670
path: "/metrics"
And the Prometheus configurations:
scrape_configs:
- job_name: 'kuma-dataplanes'
# ...
kuma_sd_configs:
- server: http://localhost:5676
refresh_interval: 60s # different from prometheus-two
client_id: "prometheus-one"
scrape_configs:
- job_name: 'kuma-dataplanes'
# ...
kuma_sd_configs:
- server: http://localhost:5676
refresh_interval: 20s # different from prometheus-one
client_id: "prometheus-two"
Examples
With custom port, path, clientId, application aggregation and service override
The first policy defines a default MeshMetric
policy for the default
mesh.
The second policy creates an override for workloads tagged with framework: example-web-framework
.
That web framework exposes metrics under /metrics/prometheus
and port 8888
.
apiVersion: kuma.io/v1alpha1
kind: MeshMetric
metadata:
name: metrics-default
namespace: kuma-system
labels:
kuma.io/mesh: default
spec:
targetRef:
kind: Mesh
default:
sidecar:
usedOnly: true
backends:
- type: Prometheus
prometheus:
clientId: main-backend
port: 5670
path: "/metrics"
tls:
mode: ProvidedTLS
apiVersion: kuma.io/v1alpha1
kind: MeshMetric
metadata:
name: metrics-for-mesh-service
namespace: kuma-system
labels:
kuma.io/mesh: default
spec:
targetRef:
kind: MeshSubset
tags:
framework: example-web-framework
default:
applications:
- path: "/metrics/prometheus"
port: 8888