Prometheus is a popular solution for monitoring modern application and infrastructure stacks. A key reason Prometheus is so popular -- and versatile -- is the availability of Prometheus Exporters. In simple terms, Prometheus Exporters are utilities that run alongside the application environment that a Prometheus instance monitors. The Prometheus instance scrapes the exporter for crucial application and infrastructure-level metrics.
There are multiple different exporters, and here we’ll take a deep dive on Prometheus Blackbox exporter. We’ll also provide an overview of four other exporters you can use.
Of course, you shouldn’t just use a tool because it’s popular. Tools should solve a problem you have. So, why might you need an exporter like Prometheus Blackbox Exporter?
Exporters are necessary since they are the fundamental Prometheus functionality that collects metrics. Prometheus scrapes these metrics to provide you with application and infrastructure visibility. The diagram below shows how all the pieces, including exporters, fit together in an example architecture.
Prometheus Exporters: A Bird’s Eye View
Now that we know why exporters are useful, let’s look at some of our most popular options and their value.
|Blackbox Exporter||Metrics for “blackbox” probing of endpoints over HTTP, HTTPS, DNS, TCP, and ICMP|
|Kube-state Metrics Exporter||Kubernetes (K8s) add-on agent which provides metrics about various Kubernetes objects, such as deployments, nodes, and pods|
|Node Exporter||Exposes hardware and OS level metrics for *NIX kernels|
|Elasticsearch Exporter||Connects to local or remote Elasticsearch (ES) instance to provide various Elasticsearch metrics in Prometheus format|
|Redis Exporter||Connects to local or remote Redis instance to provide various Redis metrics in Prometheus format|
Now, let’s take a closer look at these exporters, beginning with one of the most popular: Prometheus Blackbox Exporter.
Blackbox Exporter is used for endpoint monitoring and can help generate meaningful uptime and availability metrics. Blackbox Exporter can probe endpoints over HTTP, HTTPS, DNS, TCP, and ICMP.
While most exporters accept static configurations and expose metrics accordingly, Blackbox Exporter works a little differently. Inside the Blackbox Exporter config, you define modules. Then, Prometheus can query each of those modules for a set of specific targets. As a response to that query, Blackbox Exporter generates metrics for the queried endpoint.
To better understand how the Blackbox Exporter works, let’s walk through a practical example.
Here is a sample module defined in the Blackbox Exporter config:
modules: http_2xx: http: fail_if_not_ssl: true ip_protocol_fallback: false method: GET no_follow_redirects: false preferred_ip_protocol: ip4 valid_http_versions: - HTTP/1.1 - HTTP/2.0 prober: http timeout: 15s
The above module is named http_2xx. It is based on the HTTP Probe offered by Blackbox Exporter. Other configuration parameters, and their meaning, are detailed in the table below:
|Fail_if_not_ssl: true||Fail if the endpoint is not SSL secured|
|method: GET||Use HTTP GET method to access the endpoint|
|no_follow_redirects: false||Do not follow HTTP redirects|
|preferred_ip_protocol: ip4||Use IPv4 protocol|
|valid_http_versions||Both HTTP/1.1 and HTTP/2.0 are valid|
|timeout: 15s||Timeout after 15s if no response received|
Now that we have configured the Blackbox Exporter let’s see how Prometheus gets metrics relevant to this module for an endpoint. Inside the Prometheus config, we have a section as follows:
- job_name: 'blackbox-https' scrape_timeout: 15s scrape_interval: 15s metrics_path: /probe params: module: [http_2xx] static_configs: - targets: - https://example.com relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: prometheus-blackbox-exporter:9115
Most of the above Prometheus config is self-explanatory. However, it’s let’s look closer at these two points:
- Note how we chose the Blackbox Export module inside the params section. This module name should correspond to the module name defined in the Blackbox Exporter config (http_2xx in this case).
- Under the static configs section, we provide a target as “https://example.com”. This configuration means that Prometheus will query Blackbox Exporter’s http_2xx module with http://example.com as a parameter.
Stitching all of this together, we should be able to see https metrics for https://example.com by visiting http://prometheus-blackbox-exporter:9115/probe?target=https://example.com&module=http_2xx . The probe_success metric indicates if the probe succeeded. Adding a debug=true parameter will return debug information for that probe.
Let’s look at another example for a TCP-level probe:
modules: tcp_connect: prober: tcp tcp: ip_protocol_fallback: false preferred_ip_protocol: ip4 tls: true tls_config: insecure_skip_verify: true server_name: myserver timeout: 15s
The module defined in the snippet above is named tcp_connect and uses BlackBox Exporter’s TCP Probe. Most of the config options are similar to the http_2xx probe except for some TLS settings. Let’s look at those:
|tls: true||Enable TLS verification for TCP connection|
|insecure_skip_verify: true||Disables SSL certificate verification. This means that your certificate does not need to be a valid one and can be self-signed.|
|server_name: myserver||The presented certificate should have ‘myserver’ added as one of the SANs (Subject Alternative Name) to the certificate|
Similar to our earlier config, we can have another job config added to our Prometheus configuration file. This job uses the tcp_connect module and gets metrics for a TCP endpoint using self-signed certificates.
- job_name: 'blackbox-tcp' scrape_timeout: 15s scrape_interval: 15s metrics_path: /probe params: module: [tcp_connect] static_configs: - targets: - <my_tcp_server>:<my_tcp_server_port> relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: prometheus-blackbox-exporter:9115
Similar to the case above, we should be able to see tcp metrics for endpoint <my_tcp_server>:<my_tcp_server_port> by visiting http://prometheus-blackbox-exporter:9115/probe?target=<my_tcp_server>:<my_tcp_server_port>&module=tcp_connect . The probe_success metric indicates if the probe succeeded. Adding a debug=true parameter will return debug information for that probe.
With these examples, you should now understand how Blackbox Exporter works and how you can configure Prometheus to scrape HTTPS and TCP metrics from your endpoint. Now, let’s learn a bit more about the other exporters.
Kube-state Metrics Exporter
If you want to monitor your Kubernetes workloads using a Prometheus instance, deploying Kube-state Metrics is an absolute must.
This exporter will generate crucial container and pod level metrics for all kinds of K8s workloads. These metrics include vital stats like container CPU/Memory limits and requests, number of replicas for deployment to cronjob, and ingress-level metrics. For more detail, you can find the complete list of metrics here
You can easily deploy Kube-state Metrics by using a helm chart.
Node Exporter provides OS and Hardware level metrics for the VMs. You can deploy Node Exporter as a process running over each VM or as a Daemonset if you want to monitor metrics for K8s nodes.
It provides metrics like disk usage, IOPS, CPU / Memory usage, overall system load, etc. For more detail, you can find the complete list of metrics here.
You can find in-depth installation instructions for different platforms here.
Elasticsearch is one of the most popular document stores and is used extensively for power search-based backends. It also has a K8s operator named ECK for easy orchestration of Elasticsearch workloads over Kubernetes. The Elasticsearch Exporter enables you to use Prometheus with ES.
This exporter can connect to your ES instance and get crucial performance and health metrics about the database. It provides metrics like cluster health, the total number of shards, initializing or unassigned shards, etc. You can find the complete list of metrics here.
You can find a detailed installation guide for different platforms here
Redis is hands down the most popular cache used by modern software applications. Given its crucial position in the tech stack. It is imperative to monitor your Redis instances.
The Redis Exporter connects to your Redis instance like any other Redis client and generates metrics in a Prometheus-friendly format.
Sometimes using existing open-source Prometheus Exporters is not enough. There might be scenarios where you need Prometheus to pull specific metrics relevant to your application. While many standard use cases are covered, there won’t always be an existing exporter to do the job for you. Fortunately, you can simply write a Prometheus Exporter yourself.
Where do you start? The official Prometheus Documentation provides a very detailed guide.
Prometheus Exporters are an excellent means of de-coupling metrics generation and storage in a Prometheus-based monitoring system. While you may not use all of the exporters out there, some are fundamental to monitoring any modern infrastructure and application environment. For example, Prometheus Blackbox Exporter is crucial for endpoint monitoring.
However, simply setting up exporters is not helpful if your Prometheus instance is not scalable, highly available, or fault-tolerant. To solve that problem, we would like to introduce Opsramp.
Based on the scalable Prometheus Cortex technology, OpsRamp off-loads Prometheus maintenance from DevOps teams and allows them to focus on more important tasks like correctly configuring exporters and other metric sources. Opsramp also seamlessly integrates with your public cloud subscription to provide you with cloud provider-specific metrics. You can get started controlling chaos and explore all of Opsramp for free. You can sign-up here.