Difference between revisions of "Istio + Thanos using kube-prometheus-stack (Prometheus Operator)"

From Fyzix
Jump to navigation Jump to search
(Created page with "References: * https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack * Additional References (for implementing Thanos with EKS + S3): - h...")
 
Line 31: Line 31:
 
     istio-injection: enabled
 
     istio-injection: enabled
 
</pre>
 
</pre>
 
  
 
= Solution =
 
= Solution =
Line 49: Line 48:
  
 
This will effectively exclude Istio Sidecar injection into port 10901, allowing Thanos to properly communicate.
 
This will effectively exclude Istio Sidecar injection into port 10901, allowing Thanos to properly communicate.
 +
 +
Original Issue Reference (lacked context details): https://github.com/thanos-io/kube-thanos/issues/85

Revision as of 15:06, 19 January 2021

References:

Additional References (for implementing Thanos with EKS + S3): - https://aws.amazon.com/blogs/opensource/improving-ha-and-long-term-storage-for-prometheus-using-thanos-on-eks-with-s3/ - https://thanos.io/v0.17/thanos/storage.md/#s3 - https://github.com/thanos-io/kube-thanos.git - https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/thanos.md - https://medium.com/@kakashiliu/deploy-prometheus-operator-with-thanos-60210eff172b

Problem

Thanos is unable to read the promethus-operated service to fetch metrics less than 2 hours old (i.e. You cannot see Thanos -> Store -> Sidecar with Endpoint prometheus-operated (prometheus-operated.monitoring.svc.cluster.local:10901).

I had read a bunch of stuff talking about utilizing listenLocal: true, but this didn't seem to work as intended, and I could not reach the Prometheus GUI after enabling.

This assumes you have Istio enabled at the namespace level

---
apiVersion: v1
kind: Namespace
metadata:
  labels:
    name: monitoring
    istio-injection: enabled
    monitoring: "true"
  name: monitoring

Specifically,

    istio-injection: enabled

Solution

To resolve this issue, add a specific ignore at the Pod level under prometheusSpec.podMetadata in values.yaml

Reference: https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml

      ## Standard object’s metadata. More info: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#metadata
      ## Metadata Labels and Annotations gets propagated to the prometheus pods.
      ##
      podMetadata:
        annotations:
          traffic.sidecar.istio.io/excludeOutboundPorts: "10901"
          traffic.sidecar.istio.io/excludeInboundPorts: "10901"

This will effectively exclude Istio Sidecar injection into port 10901, allowing Thanos to properly communicate.

Original Issue Reference (lacked context details): https://github.com/thanos-io/kube-thanos/issues/85