Istio + Thanos using kube-prometheus-stack (Prometheus Operator)
Additional References (for implementing Thanos with EKS + S3): - https://aws.amazon.com/blogs/opensource/improving-ha-and-long-term-storage-for-prometheus-using-thanos-on-eks-with-s3/ - https://thanos.io/v0.17/thanos/storage.md/#s3 - https://github.com/thanos-io/kube-thanos.git - https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/thanos.md - https://medium.com/@kakashiliu/deploy-prometheus-operator-with-thanos-60210eff172b
Thanos is unable to read the promethus-operated service to fetch metrics less than 2 hours old (i.e. You cannot see Thanos -> Store -> Sidecar with Endpoint prometheus-operated (prometheus-operated.monitoring.svc.cluster.local:10901).
I had read a bunch of stuff talking about utilizing listenLocal: true, but this didn't seem to work as intended, and I could not reach the Prometheus GUI after enabling.
This assumes you have Istio enabled at the namespace level
--- apiVersion: v1 kind: Namespace metadata: labels: name: monitoring istio-injection: enabled monitoring: "true" name: monitoring
To resolve this issue, add a specific ignore at the Pod level under prometheusSpec.podMetadata in values.yaml
## Standard object’s metadata. More info: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#metadata ## Metadata Labels and Annotations gets propagated to the prometheus pods. ## podMetadata: annotations: traffic.sidecar.istio.io/excludeOutboundPorts: "10901" traffic.sidecar.istio.io/excludeInboundPorts: "10901"
This will effectively exclude Istio Sidecar injection into port 10901, allowing Thanos to properly communicate.