Prometheus automated discovery of services with docker-compose

When using docker-compose, we can use labels to annotate containers with some metadata. Similarly to how this can be done on Kubernetes using annotations.

You might already be using labels if you are using traefik or ofelia.

Prometheus service discovery

Prometheus supports service discovery mechanism. We are going to be using docker_sd to discover docker containers.

We can use a configuration like this. Which will use the docker.sock to get all containers, and then we can do some modification depending on labels, for example we can override the scheme, port and path via labels. As well as checking if prometheus.scrape is set to "true", to only scrape those containers that we explicitly say we want.

prometheus.yaml file would look like this:

scrape_configs:
  - job_name: docker-containers
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
    relabel_configs:
      # Only keep containers that have a `prometheus.scrape` label set to true.
      - source_labels: [__meta_docker_container_label_prometheus_scrape]
        regex: true
        action: keep

      # Only keep those that are in network named metrics.
      - source_labels: [__meta_docker_network_name]
        regex: metrics
        action: keep

      # allow override of http scheme with `promehteus.scheme`
      - action: replace
        regex: (https?)
        source_labels: [__meta_docker_container_label_prometheus_scheme]
        target_label: __scheme__

      # allow override of default /metrics path with `prometheus.path`
      - action: replace
        regex: (.+)
        source_labels: [__meta_docker_container_label_prometheus_path]
        target_label: __metrics_path__

      # allow override of default port with `prometheus.port`
      - action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        source_labels: [__address__, __meta_docker_container_label_prometheus_port]
        target_label: __address__

      # Add all labels
      - {action: labelmap, regex: __meta_docker_container_label_(.+)}

the docker-compose.yaml would look like this:

version: "3"

networks:
  # Create a dedicated network for metrics
  metrics:
    name: metrics

services:
  # Run prometheus service that uses the prometheus.yaml config
  prometheus:
    image: prom/prometheus
    restart: always
    command:
      - "--config.file=/etc/prometheus/prometheus.yaml"
    networks:
      - metrics
    # root is needed to access the docker.sock.
    user: root
    volumes:
      # We need to pass the docker.sock for prometheus to be able
      # to connect to docker engine and discover containers.
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./prometheus.yaml:/etc/prometheus/prometheus.yaml:ro

Using the discovery with an application container

Now that we have Prometheus set up, we can deploy an application and tell the Prometheus to scrape it via the labels.

version: "3"

networks:
  traefik:
    name: traefik
  # We created metrics network in the docker-compose for prometheus,
  # that's why we have set it here to "external".
  metrics:
    external: true

services:
  # Example traefik which is configured to expose metrics in port 8082
  traefik:
    image: traefik:3.0
    restart: always
    networks:
      - traefik
      # The service must be part of metrics network,
      # otherwise the prometheus won't be able to connect to it.
      - metrics
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./traefik.toml:/traefik.toml
      - ./certificates:/certificates/
    labels:
      prometheus.scrape: "true"
      prometheus.port: "8082"
      prometheus.path: "/metrics" # /metrics is default
      prometheus.scheme: "http"   # http is default

When our service is deployed, the Prometheus should automatically discover it and start scraping the metrics.