Difference between revisions of "Istio + Thanos using kube-prometheus-stack (Prometheus Operator)"

From Fyzix
Jump to navigation Jump to search
Line 1: Line 1:
 
References:
 
References:
* https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
+
* https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack  
*
 
  
 
Additional References (for implementing Thanos with EKS + S3):
 
Additional References (for implementing Thanos with EKS + S3):
- https://aws.amazon.com/blogs/opensource/improving-ha-and-long-term-storage-for-prometheus-using-thanos-on-eks-with-s3/
+
* https://aws.amazon.com/blogs/opensource/improving-ha-and-long-term-storage-for-prometheus-using-thanos-on-eks-with-s3/
- https://thanos.io/v0.17/thanos/storage.md/#s3
+
* https://thanos.io/v0.17/thanos/storage.md/#s3
- https://github.com/thanos-io/kube-thanos.git
+
* https://github.com/thanos-io/kube-thanos.git
- https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/thanos.md
+
* https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/thanos.md
- https://medium.com/@kakashiliu/deploy-prometheus-operator-with-thanos-60210eff172b
+
* https://medium.com/@kakashiliu/deploy-prometheus-operator-with-thanos-60210eff172b
  
 
= Problem =
 
= Problem =
 
Thanos is unable to read the promethus-operated service to fetch metrics less than 2 hours old (i.e. You cannot see Thanos -> Store -> Sidecar with Endpoint prometheus-operated (prometheus-operated.monitoring.svc.cluster.local:10901).
 
Thanos is unable to read the promethus-operated service to fetch metrics less than 2 hours old (i.e. You cannot see Thanos -> Store -> Sidecar with Endpoint prometheus-operated (prometheus-operated.monitoring.svc.cluster.local:10901).
  
I had read a bunch of stuff talking about utilizing ''listenLocal: true'', but this didn't seem to work as intended, and I could not reach the Prometheus GUI after enabling.
+
You could disable Istio Sidecar injection entirely for Prometheus. But, this not desired.
 +
 
 +
I had read a bunch of stuff discussing turning '''listenLocal: true''', but this didn't seem to work as intended, and I could not reach the Prometheus GUI after enabling.
  
 
This assumes you have Istio enabled at the namespace level
 
This assumes you have Istio enabled at the namespace level
Line 27: Line 28:
 
   name: monitoring
 
   name: monitoring
 
</pre>
 
</pre>
 +
 
Specifically,
 
Specifically,
 
<pre>
 
<pre>

Revision as of 15:08, 19 January 2021

References:

Additional References (for implementing Thanos with EKS + S3):

Problem

Thanos is unable to read the promethus-operated service to fetch metrics less than 2 hours old (i.e. You cannot see Thanos -> Store -> Sidecar with Endpoint prometheus-operated (prometheus-operated.monitoring.svc.cluster.local:10901).

You could disable Istio Sidecar injection entirely for Prometheus. But, this not desired.

I had read a bunch of stuff discussing turning listenLocal: true, but this didn't seem to work as intended, and I could not reach the Prometheus GUI after enabling.

This assumes you have Istio enabled at the namespace level

---
apiVersion: v1
kind: Namespace
metadata:
  labels:
    name: monitoring
    istio-injection: enabled
    monitoring: "true"
  name: monitoring

Specifically,

    istio-injection: enabled

Solution

To resolve this issue, add a specific ignore at the Pod level under prometheusSpec.podMetadata in values.yaml

Reference: https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml

      ## Standard object’s metadata. More info: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#metadata
      ## Metadata Labels and Annotations gets propagated to the prometheus pods.
      ##
      podMetadata:
        annotations:
          traffic.sidecar.istio.io/excludeOutboundPorts: "10901"
          traffic.sidecar.istio.io/excludeInboundPorts: "10901"

This will effectively exclude Istio Sidecar injection into port 10901, allowing Thanos to properly communicate.

Original Issue Reference (lacked context details): https://github.com/thanos-io/kube-thanos/issues/85