Istio + Thanos using kube-prometheus-stack (Prometheus Operator)
Additional References (for implementing Thanos with EKS + S3):
Thanos is unable to read the promethus-operated service to fetch metrics less than 2 hours old (i.e. You cannot see Thanos -> Store -> Sidecar with Endpoint prometheus-operated (prometheus-operated.monitoring.svc.cluster.local:10901).
You could disable Istio Sidecar injection entirely for Prometheus. But, this not desired. We want Istio enabled for Prometheus Operator.
I had read a bunch of stuff discussing turning listenLocal: true, but this didn't seem to work as intended, and I could not reach the Prometheus GUI after enabling.
This assumes you have Istio enabled at the namespace level
--- apiVersion: v1 kind: Namespace metadata: labels: name: monitoring istio-injection: enabled monitoring: "true" name: monitoring
To resolve this issue, add a specific ignore at the Pod level under prometheusSpec.podMetadata in values.yaml
## Standard object’s metadata. More info: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#metadata ## Metadata Labels and Annotations gets propagated to the prometheus pods. ## podMetadata: annotations: traffic.sidecar.istio.io/excludeOutboundPorts: "10901" traffic.sidecar.istio.io/excludeInboundPorts: "10901"
This will effectively exclude Istio Sidecar injection into port 10901, allowing Thanos to properly communicate.
Original Issue Reference (lacked context details): https://github.com/thanos-io/kube-thanos/issues/85