At work we’ve started to instrument our micro-services with Prometheus. The combination of Prometheus and Grafana is pretty amazing.
One of the nice things Prometheus does is auto-discovery of services in a Kubernetes cluster. The docs for this area aren’t bad but I couldn’t find a complete example so here’s the configuration we use.
- job_name: 'kubernetes'
kubernetes_sd_configs:
- api_servers:
- 'https://kubernetes.default.svc'
in_cluster: true
relabel_configs:
- regex: "__meta_kubernetes_role"
action: labelmap
replacement: "kubernetes_role"
- regex: "__meta_kubernetes_node_label_(.*)"
action: labelmap
replacement: "kubernetes_node_label_${1}"
- regex: "__meta_kubernetes_service_namespace"
action: labelmap
replacement: "kubernetes_service_namespace"
- regex: "__meta_kubernetes_service_name"
action: labelmap
replacement: "kubernetes_service_name"
- regex: "__meta_kubernetes_service_label_(.*)"
action: labelmap
replacement: "kubernetes_service_label_${1}"
- regex: "__meta_kubernetes_service_annotation_(.*)"
action: labelmap
replacement: "__meta_kubernetes_service_annotation_${1}"
