Collecting Prometheus metrics | User Documentation (2023)

  • Forwarding metrics from Pods
  • Defining prometheus input
  • Default configuration
    • Kubelet
    • API Server
    • Controller
    • etcd
  • Metrics format
  • Links

Most of the components in OpenShift control plane export metrics in Prometheus format. The collector can read these metrics forward them to Splunk Enterprise or Splunk Cloud. Our installation has default configurations for collecting metrics from API Server, Controllers, Kubelets and etcd cluster. In most OpenShift providers you don't need to do additional configuration to see these metrics.

If your applications export metrics in Prometheus format, you can use our collector to forward these metrics as well to Splunk Enterprise or Splunk Cloud.

Forwarding metrics from Pods

Please read our documentation on annotations, to learn how you can define forwarding metrics from Pods.

Defining prometheus input

We deploy collector in 3 different workloads. Depending on where you want to collect your metrics, you should plan to include you Prometheus metrics.

  • 002-daemonset.conf is installed on all nodes (masters and non-masters). Use this configuration if you need to collect metrics from all nodes, from local ports. Example of these metrics is Kubelet metrics.
  • 003-daemonset-master.conf is installed only on master nodes. Use this configuration to collect metrics only from master nodes from local ports. Examples of these metrics are control plane processes, etcd running on masters.
  • 004-addon.conf installed as a deployment and used only once in the whole cluster. Place your Prometheus configuration here, if you want to collect metrics from endpoints or service. Examples of these Prometheus configurations are controller manager and scheduler, which can be accessed only from an internal network and can be discovered with endpoints. Another example is etcd cluster running outside of the OpenShift cluster.

Default configuration

Kubelet

On every node collector reads and forwards kubelet metrics. We deploy this configuration in 002-daemonset.conf.

 1 2 3 4 5 6 7 8 91011121314151617181920212223242526272829303132333435363738
[input.prometheus::kubelet]# disable prometheus kubelet metricsdisabled = false# override typetype = prometheus# specify Splunk indexindex =# Override host (environment variables are supported)host = ${KUBERNETES_NODENAME}# Override sourcesource = kubelet# how often to collect prometheus metricsinterval = 60s# prometheus endpointendpoint = https://127.0.0.1:10250/metrics# token for "Authorization: Bearer $(cat tokenPath)"tokenPath = /var/run/secrets/kubernetes.io/serviceaccount/token# server certificate for certificate validationcertPath = /var/run/secrets/kubernetes.io/serviceaccount/ca.crt# client certificate for authenticationclientCertPath =# Allow invalid SSL server certificateinsecure = true# include metrics help with the events# can be useful to explore prometheus metricsincludeHelp = false

API Server

On master nodes collectors reads and forwards metrics from the API server. We deploy this configuration using 003-daemonset-master.conf.

 1 2 3 4 5 6 7 8 910111213141516171819202122232425262728293031323334353637383940
[input.prometheus::kubernetes-api]# disable prometheus kubernetes-api inputdisabled = false# override typetype = prometheus# specify Splunk indexindex =# override hosthost = ${KUBERNETES_NODENAME}# override sourcesource = kubernetes-api# how often to collect prometheus metricsinterval = 60s# prometheus endpoint# at first trying to get it from localhost (that way avoiding load balancer, if multiple)# as fallback using proxyendpoint.1localhost = https://127.0.0.1:8443/metricsendpoint.2kubeapi = https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}/metrics# token for "Authorization: Bearer $(cat tokenPath)"tokenPath = /var/run/secrets/kubernetes.io/serviceaccount/token# server certificate for certificate validationcertPath = /var/run/secrets/kubernetes.io/serviceaccount/ca.crt# client certificate for authenticationclientCertPath =# Allow invalid SSL server certificateinsecure = true# include metrics help with the eventsincludeHelp = false

Controller

On master nodes collectors reads and forwards metrics from the controller. We deploy this configuration using 003-daemonset-master.conf.

 1 2 3 4 5 6 7 8 91011121314151617181920212223242526272829303132333435363738
[input.prometheus::controller]# disable prometheus controller metricsdisabled = false# override typetype = prometheus# specify Splunk indexindex =# override hosthost = ${KUBERNETES_NODENAME}# override sourcesource = controller# how often to collect prometheus metricsinterval = 60s# prometheus endpointendpoint.https = https://127.0.0.1:8444/metrics# token for "Authorization: Bearer $(cat tokenPath)"tokenPath = /var/run/secrets/kubernetes.io/serviceaccount/token# server certificate for certificate validationcertPath =# client certificate for authenticationclientCertPath =clientKeyPath =# Allow invalid SSL server certificateinsecure = true# include metrics help with the eventsincludeHelp = false

etcd

On master nodes, collectors read and forward metrics from etcd processes. We deploy this configuration using 003-daemonset-master.conf.

 1 2 3 4 5 6 7 8 91011121314151617181920212223242526272829303132333435363738
[input.prometheus::etcd]# disable prometheus etcd metricsdisabled = false# override typetype = prometheus# specify Splunk indexindex =# override hosthost = ${KUBERNETES_NODENAME}# override sourcesource = etcd# how often to collect prometheus metricdinterval = 60s# prometheus endpointendpoint.https = https://:2379/metrics# token for "Authorization: Bearer $(cat tokenPath)"tokenPath =# server certificate for certificate validationcertPath = /rootfs/etc/origin/master/master.etcd-ca.crt# client certificate for authenticationclientCertPath = /rootfs/etc/origin/master/master.etcd-client.crtclientKeyPath = /rootfs/etc/origin/master/master.etcd-client.key# Allow invalid SSL server certificateinsecure = true# include metrics help with the eventsincludeHelp = false

This configuration works when you run etcd cluster with master nodes. With this configuration, collector tries to collect metrics using http scheme at first, and https after that. For https collector uses certPath, clientCertPath and clientKeyPath, which are mounted from the host.

 1 2 3 4 5 6 7 8 91011
... volumeMounts: ... - name: origin-certs mountPath: /rootfs/etc/origin/master/ readOnly: true...volumes:- name: origin-certs hostPath: path: /etc/origin/master/

Verify that these certificates are available, if not, make appropriate changes.

If your etcd cluster is a dedicated set of nodes, you can define prometheus collection in 004-addon.conf.

Metrics format

Prometheus defines several types of metrics.

Each metric value in Splunk has fields:

  • metric_type - one of the types from the Prometheus metric types.
  • metric_name - the name of the metric.
  • metric_help - only if includeHelp is set to true, you will see definition of this metric.
  • metric_label_XXX - if the metric has labels, you will be able to see them attached to the metric values.
  • seed - unique value from the host for specific metric collection.

Based on the metric type you can find various values for the metrics.

  • counter
    • v - current counter value
    • d - the difference with a previous value
    • s - period for which this difference is calculated (in seconds)
    • p - (deprecated) period for which this difference is calculated (in nanoseconds)
  • summary and histogram
    • v - value
    • c - counter specified for this summary or histogram metric
  • All others
    • v - value

If you have specified to include help with the metrics, you can explore all available metrics with the search.

sourcetype="prometheus"| stats latest(_raw) by source, metric_type, metric_name, metric_help
Collecting Prometheus metrics | User Documentation (1)

Links

  • Installation
    • Start monitoring your OpenShift environments in under 10 minutes.
    • Automatically forward host, container and application logs.
    • Test our solution with the embedded 30 days evaluation license.
  • Collector Configuration
    • Collector configuration reference.
  • Annotations
    • Changing index, source, sourcetype for namespaces, workloads and pods.
    • Forwarding application logs.
    • Multi-line container logs.
    • Fields extraction for application and container logs (including timestamp extractions).
    • Hiding sensitive data, stripping terminal escape codes and colors.
    • Forwarding Prometheus metrics from Pods.
  • Audit Logs
    • Configure audit logs.
    • Forwarding audit logs.
  • Prometheus metrics
    • Collect metrics from control plane (etcd cluster, API server, kubelet, scheduler, controller).
    • Configure collector to forward metrics from the services in Prometheus format.
  • Configuring Splunk Indexes
    • Using not default HTTP Event Collector index.
    • Configure the Splunk application to use not searchable by default indexes.
  • Splunk fields extraction for container logs
    • Configure search-time fields extractions for container logs.
    • Container logs source pattern.
  • Configurations for Splunk HTTP Event Collector
    • Configure multiple HTTP Event Collector endpoints for Load Balancing and Fail-overs.
    • Secure HTTP Event Collector endpoint.
    • Configure the Proxy for HTTP Event Collector endpoint.
  • Monitoring multiple clusters
    • Learn how you can monitor multiple clusters.
    • Learn how to set up ACL in Splunk.
  • Streaming OpenShift Objects from the API Server
    • Learn how you can stream all changes from the OpenShift API Server.
    • Stream changes and objects from OpenShift API Server, including Pods, Deployments or ConfigMaps.
  • License Server
    • Learn how you can configure remote License URL for Collectord.
  • Monitoring GPU
  • Alerts
  • Troubleshooting
  • Release History
  • Upgrade instructions
  • Security
  • FAQ and the common questions
  • License agreement
  • Pricing
  • Contact

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.

Top Articles
Latest Posts
Article information

Author: Maia Crooks Jr

Last Updated: 10/11/2022

Views: 5975

Rating: 4.2 / 5 (43 voted)

Reviews: 82% of readers found this page helpful

Author information

Name: Maia Crooks Jr

Birthday: 1997-09-21

Address: 93119 Joseph Street, Peggyfurt, NC 11582

Phone: +2983088926881

Job: Principal Design Liaison

Hobby: Web surfing, Skiing, role-playing games, Sketching, Polo, Sewing, Genealogy

Introduction: My name is Maia Crooks Jr, I am a homely, joyous, shiny, successful, hilarious, thoughtful, joyous person who loves writing and wants to share my knowledge and understanding with you.