Istio + Thanos using kube-prometheus-stack (Prometheus Operator)

From Fyzix
Revision as of 15:08, 19 January 2021 by Fyzix (talk | contribs)
Jump to navigation Jump to search


Additional References (for implementing Thanos with EKS + S3):


Thanos is unable to read the promethus-operated service to fetch metrics less than 2 hours old (i.e. You cannot see Thanos -> Store -> Sidecar with Endpoint prometheus-operated (prometheus-operated.monitoring.svc.cluster.local:10901).

You could disable Istio Sidecar injection entirely for Prometheus. But, this not desired.

I had read a bunch of stuff discussing turning listenLocal: true, but this didn't seem to work as intended, and I could not reach the Prometheus GUI after enabling.

This assumes you have Istio enabled at the namespace level

apiVersion: v1
kind: Namespace
    name: monitoring
    istio-injection: enabled
    monitoring: "true"
  name: monitoring


    istio-injection: enabled


To resolve this issue, add a specific ignore at the Pod level under prometheusSpec.podMetadata in values.yaml


      ## Standard object’s metadata. More info:
      ## Metadata Labels and Annotations gets propagated to the prometheus pods.

This will effectively exclude Istio Sidecar injection into port 10901, allowing Thanos to properly communicate.

Original Issue Reference (lacked context details):