At work we’ve started to instrument our micro-services with Prometheus. The combination of Prometheus and Grafana is pretty amazing.

One of the nice things Prometheus does is auto-discovery of services in a Kubernetes cluster. The docs for this area aren’t bad but I couldn’t find a complete example so here’s the configuration we use.

  - job_name: 'kubernetes'
    kubernetes_sd_configs:
      - api_servers:
          -  'https://kubernetes.default.svc'
        in_cluster: true
    relabel_configs:
      - regex: "__meta_kubernetes_role"
        action: labelmap
        replacement: "kubernetes_role"
      - regex: "__meta_kubernetes_node_label_(.*)"
        action: labelmap
        replacement: "kubernetes_node_label_${1}"
      - regex: "__meta_kubernetes_service_namespace"
        action: labelmap
        replacement: "kubernetes_service_namespace"
      - regex: "__meta_kubernetes_service_name"
        action: labelmap
        replacement: "kubernetes_service_name"
      - regex: "__meta_kubernetes_service_label_(.*)"
        action: labelmap
        replacement: "kubernetes_service_label_${1}"
      - regex: "__meta_kubernetes_service_annotation_(.*)"
        action: labelmap
        replacement: "__meta_kubernetes_service_annotation_${1}"