Istio + Thanos using kube-prometheus-stack (Prometheus Operator)

From Fyzix
Jump to navigation Jump to search

References:

Additional References (for implementing Thanos with EKS + S3):

Problem

Thanos is unable to read the promethus-operated service to fetch metrics less than 2 hours old (i.e. You cannot see Thanos -> Store -> Sidecar with Endpoint prometheus-operated (prometheus-operated.monitoring.svc.cluster.local:10901).

You could disable Istio Sidecar injection entirely for Prometheus. But, this not desired. We want Istio enabled for Prometheus Operator.

I had read a bunch of stuff discussing turning listenLocal: true, but this didn't seem to work as intended, and I could not reach the Prometheus GUI after enabling.

This assumes you have Istio enabled at the namespace level 

---
apiVersion: v1
kind: Namespace
metadata:
  labels:
    name: monitoring
    istio-injection: enabled
    monitoring: "true"
  name: monitoring

Specifically, 

    istio-injection: enabled

Solution

To resolve this issue, add a specific ignore at the Pod level under prometheusSpec.podMetadata in values.yaml

Reference: https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml 

      ## Standard object’s metadata. More info: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#metadata
      ## Metadata Labels and Annotations gets propagated to the prometheus pods.
      ##
      podMetadata:
        annotations:
          traffic.sidecar.istio.io/excludeOutboundPorts: "10901"
          traffic.sidecar.istio.io/excludeInboundPorts: "10901"

This will effectively exclude Istio Sidecar injection into port 10901, allowing Thanos to properly communicate.

Original Issue Reference (lacked context details): https://github.com/thanos-io/kube-thanos/issues/85

Retrieved from "https://www.fyzix.net/index.php?title=Istio_%2B_Thanos_using_kube-prometheus-stack_(Prometheus_Operator)&oldid=33"