Difference between revisions of "Istio + Thanos using kube-prometheus-stack (Prometheus Operator)"

From Fyzix
Jump to navigation Jump to search
Line 12: Line 12:
 
Thanos is unable to read the promethus-operated service to fetch metrics less than 2 hours old (i.e. You cannot see Thanos -> Store -> Sidecar with Endpoint prometheus-operated (prometheus-operated.monitoring.svc.cluster.local:10901).
 
Thanos is unable to read the promethus-operated service to fetch metrics less than 2 hours old (i.e. You cannot see Thanos -> Store -> Sidecar with Endpoint prometheus-operated (prometheus-operated.monitoring.svc.cluster.local:10901).
  
You could disable Istio Sidecar injection entirely for Prometheus. But, this not desired.  
+
You could disable Istio Sidecar injection entirely for Prometheus. But, this not desired. We want Istio enabled for Prometheus Operator.  
  
 
I had read a bunch of stuff discussing turning '''listenLocal: true''', but this didn't seem to work as intended, and I could not reach the Prometheus GUI after enabling.
 
I had read a bunch of stuff discussing turning '''listenLocal: true''', but this didn't seem to work as intended, and I could not reach the Prometheus GUI after enabling.

Revision as of 15:09, 19 January 2021

References:

Additional References (for implementing Thanos with EKS + S3):

Problem

Thanos is unable to read the promethus-operated service to fetch metrics less than 2 hours old (i.e. You cannot see Thanos -> Store -> Sidecar with Endpoint prometheus-operated (prometheus-operated.monitoring.svc.cluster.local:10901).

You could disable Istio Sidecar injection entirely for Prometheus. But, this not desired. We want Istio enabled for Prometheus Operator.

I had read a bunch of stuff discussing turning listenLocal: true, but this didn't seem to work as intended, and I could not reach the Prometheus GUI after enabling.

This assumes you have Istio enabled at the namespace level

---
apiVersion: v1
kind: Namespace
metadata:
  labels:
    name: monitoring
    istio-injection: enabled
    monitoring: "true"
  name: monitoring

Specifically,

    istio-injection: enabled

Solution

To resolve this issue, add a specific ignore at the Pod level under prometheusSpec.podMetadata in values.yaml

Reference: https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml

      ## Standard object’s metadata. More info: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#metadata
      ## Metadata Labels and Annotations gets propagated to the prometheus pods.
      ##
      podMetadata:
        annotations:
          traffic.sidecar.istio.io/excludeOutboundPorts: "10901"
          traffic.sidecar.istio.io/excludeInboundPorts: "10901"

This will effectively exclude Istio Sidecar injection into port 10901, allowing Thanos to properly communicate.

Original Issue Reference (lacked context details): https://github.com/thanos-io/kube-thanos/issues/85