Monitor your Public Cloud Load Balancer with Prometheus

Learn about the various options available to monitor your Octavia Load Balancer.

The Public Cloud Load Balancer provides two ways to monitor your Load Balancers: either query an API to retrieve usage statistics or configure a Prometheus listener that your Prometheus infrastructure can scrape to collect metrics.

Requirements

A Load Balancer created.

OVHcloud Control Panel Access

Direct link: Public Cloud projects
Navigation path: Public Cloud > Select your project

Instructions

Monitoring using CLI

The Load Balancer service aggregates these statistics and makes them available via the OpenStack API. Those statistics are available at the Load Balancer or listener level.

Load Balancer statistics can be queried using the OpenStack Client. They include metrics for all the listeners configured for the queried Load Balancer.

$ openstack loadbalancer stats show <lb id>
+--------------------+-----------+
| Field              | Value     |
+--------------------+-----------+
| active_connections | 0         |
| bytes_in           | 2236722   |
| bytes_out          | 100973832 |
| request_errors     | 0         |
| total_connections  | 3606      |
+--------------------+-----------+

Individual listener statistics can also be queried using the OpenStack Client.

$ openstack loadbalancer listener stats show <listener id>
+--------------------+-------+
| Field              | Value |
+--------------------+-------+
| active_connections | 0     |
| bytes_in           | 89    |
| bytes_out          | 237   |
| request_errors     | 0     |
| total_connections  | 1     |
+--------------------+-------+

Monitoring with Prometheus

To add a Prometheus endpoint on a Public Cloud Load Balancer, create a listener with a PROMETHEUS protocol. This will enable the /metrics endpoint, which listens on all the interfaces of your Load Balancer by default. This means that if your Load Balancer is using a Floating IP, by default the /metrics endpoint will be opened to the Internet. To avoid that, we advise setting the allowed_cidrs option to protect your metrics, for example with the CIDR of the subnet where the Load Balancer is spawned.

This listener type provides the same features as the "regular" listeners but does not support attaching pools or L7 policies. All metrics will be identified by the Octavia object ID (UUID) of the resources.

Info

Currently, UDP and SCTP-specific metrics are not reported via Prometheus endpoints.

Create a Prometheus listener

OVHcloud Control Panel

OVHcloud API

Horizon

OpenStack CLI

Terraform

Via the OVHcloud Control Panel

Warning

The allowed_cidrs option is not yet available in the OVHcloud Control Panel: you can create the listener here, but to restrict access to the metrics endpoint you must set allowed_cidrs via the OVHcloud API, the OpenStack CLI or Horizon.

Select Public Cloud from the top navigation bar.
In the left-hand menu under Network, select Load Balancer.
Select the name of the load balancer.
Select the Listeners tab.
Click + Add a listener.
Fill in a name, select Prometheus as the Protocol, and choose a Port (different from the already existing listener ports of your Load Balancer).

Info
The default pool is grayed out because the Prometheus listener does not need a pool, unlike the other types of listeners.
Click Add to trigger the listener creation.

The new listener is added to the listeners list.

Network resources - Load balancer monitoring prometheus - step 3

Via the OVHcloud API

You can create the Prometheus listener directly via the OVHcloud API, which lets you set allowedCidrs at creation:

POST/cloud/project/{serviceName}/region/{regionName}/loadbalancing/listener

In the path parameters, serviceName is your Public Cloud project ID and regionName is the region of your Load Balancer (for example US-EAST-VA-1). In the request body, set protocol to prometheus, choose a port, reference your loadbalancerId, and provide allowedCidrs with the subnets allowed to reach the metrics endpoint (for example ["192.0.2.0/24"]).

To change the allowed CIDRs on an existing listener, use the same allowedCidrs property on the update endpoint, where listenerId is the ID of the existing listener:

PUT/cloud/project/{serviceName}/region/{regionName}/loadbalancing/listener/{listenerId}

Via Horizon

Click Network and then Load Balancers. The Load Balancer list is displayed.

Click on the load balancer name. The load balancer details page is then displayed.

Network resources - Load balancer monitoring prometheus - step 4

Click the Listeners tab and then Create Listener.

Network resources - Load balancer monitoring prometheus - step 5

On the listener creation page, fill in the Name and switch the Protocol to PROMETHEUS. The port will be set to a default value that you can change if needed.

You can also fill in the Allowed CIDRs field to restrict access to the metrics endpoint. Enter the CIDR of the subnet where your Prometheus scraper runs (e.g. 192.0.2.0/24). Leave it empty only if your Load Balancer has no Floating IP.

Click Create Listener.

Network resources - Load balancer monitoring prometheus - step 6

Via OpenStack CLI

To create a Prometheus endpoint on port 9100 for Load Balancer lb1, you would run the following command:

$ openstack loadbalancer listener create --name stats-listener --protocol PROMETHEUS --protocol-port 9100 lb1
+-----------------------------+--------------------------------------+
| Field                       | Value                                |
+-----------------------------+--------------------------------------+
| admin_state_up              | True                                 |
| connection_limit            | -1                                   |
| created_at                  | 2023-11-30T01:44:25                  |
| default_pool_id             | None                                 |
| default_tls_container_ref   | None                                 |
| description                 |                                      |
| id                          | fb57d764-470a-4b6b-8820-627452f55b96 |
| insert_headers              | None                                 |
| l7policies                  |                                      |
| loadbalancers               | b081ed89-f6f8-48cb-a498-5e12705e2cf9 |
| name                        | stats-listener                       |
| operating_status            | OFFLINE                              |
| project_id                  | 4c1caeee063747f8878f007d1a323b2f     |
| protocol                    | PROMETHEUS                           |
| protocol_port               | 9100                                 |
| provisioning_status         | PENDING_CREATE                       |
| sni_container_refs          | []                                   |
| timeout_client_data         | 50000                                |
| timeout_member_connect      | 5000                                 |
| timeout_member_data         | 50000                                |
| timeout_tcp_inspect         | 0                                    |
| updated_at                  | None                                 |
| client_ca_tls_container_ref | None                                 |
| client_authentication       | NONE                                 |
| client_crl_container_ref    | None                                 |
| allowed_cidrs               | None                                 |
| tls_ciphers                 | None                                 |
| tls_versions                | None                                 |
| alpn_protocols              | None                                 |
| tags                        |                                      |
+-----------------------------+--------------------------------------+

To restrict access to the metrics endpoint to a specific subnet, add the --allowed-cidr option to the command above:

$ openstack loadbalancer listener create --name stats-listener --protocol PROMETHEUS --protocol-port 9100 --allowed-cidr 192.0.2.0/24 lb1

Replace 192.0.2.0/24 with the CIDR of the subnet where your Prometheus scraper runs. You can repeat the option to allow multiple CIDRs.

Via Terraform

The resource openstack_lb_listener_v2 from the OpenStack provider enables you to configure a Prometheus listener. The following snippet is extracted from a full example available on GitHub that you can adapt to your needs.

resource "openstack_lb_listener_v2" "prom_listener" {
 protocol = "PROMETHEUS"
 protocol_port = 9100
 loadbalancer_id = openstack_lb_loadbalancer_v2.tf_lb.id
 #restrict the access of the listener to the private network subnet
 allowed_cidrs = [openstack_networking_subnet_v2.tf_lb_subnet.cidr]
}

Configure Prometheus to collect your metrics

Once the PROMETHEUS listener is ACTIVE, you can configure Prometheus to collect metrics from the Load Balancer by updating the prometheus.yml file.

[scrape_configs]
- job_name: 'Octavia LB1'
  static_configs:
  - targets: ['loadbalancer-IP:9100']

Warning

NOTE: If your Prometheus server is running within your private network, you can use the virtual IP of the load balancer as the target IP in the prometheus.yml file. Otherwise, you must add a public Floating IP to your load balancer and use it as the target IP.

Example prometheus.yml file:

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    
    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s 
   
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'Octavia LB1'
    static_configs:
      - targets: ['10.xxx.xxx.170:9100']

For more information on setting up Prometheus, see the Prometheus project website.

You can connect Grafana to the Prometheus instance to provide additional graphing and dashboard capabilities. A Grafana dashboard for Octavia Load Balancers is available here.

The metrics have the following naming format:

octavia_loadbalancer_xxx or octavia_memory_pool_xxx metrics are global to the Load Balancer
octavia_listener_xxx metrics are instantiated per listener using the label {listener=ID}
octavia_pool_xxx metrics are instantiated per pool using the label {pool=ID}
octavia_member_xxx metrics are instantiated per member using the label {member=ID}

Metric	Description
octavia_loadbalancer_cpu	Load balancer CPU utilization (percentage).
octavia_loadbalancer_memory	Load balancer memory utilization (percentage).
octavia_memory_pool_allocated_bytes	Total amount of memory allocated in the memory pools (in bytes).
octavia_memory_pool_used_bytes	Total amount of memory used in the memory pools (in bytes).
octavia_memory_pool_failures_total	Total number of failed memory pool allocations.
octavia_loadbalancer_max_connections	Maximum number of concurrent connections.
octavia_loadbalancer_current_connections	Number of active sessions.
octavia_loadbalancer_connections_total	Total number of created sessions.
octavia_loadbalancer_requests_total	Total number of requests (TCP or HTTP).
octavia_loadbalancer_max_ssl_connections	Configured maximum number of concurrent SSL connections.
octavia_loadbalancer_current_ssl_connections	Number of active SSL connections.
octavia_loadbalancer_ssl_connections_total	Total number of opened SSL connections.
octavia_loadbalancer_current_connection_rate	Current number of connections per second over last elapsed second.
octavia_loadbalancer_limit_connection_rate	Configured maximum number of connections per second.
octavia_loadbalancer_max_connection_rate	Maximum observed number of connections per second.
octavia_loadbalancer_current_session_rate	Current number of sessions per second over last elapsed second.
octavia_loadbalancer_limit_session_rate	Configured maximum number of sessions per second.
octavia_loadbalancer_max_session_rate	Maximum observed number of sessions per second.
octavia_loadbalancer_current_ssl_rate	Current number of SSL sessions per second over last elapsed second.
octavia_loadbalancer_limit_ssl_rate	Configured maximum number of SSL sessions per second.
octavia_loadbalancer_max_ssl_rate	Maximum observed number of SSL sessions per second.
octavia_loadbalancer_current_frontend_ssl_key_rate	Current frontend SSL Key computation per second over last elapsed second.
octavia_loadbalancer_max_frontend_ssl_key_rate	Maximum observed frontend SSL Key computation per second.
octavia_loadbalancer_current_backend_ssl_key_rate	Current backend SSL Key computation per second over last elapsed second.
octavia_loadbalancer_max_backend_ssl_key_rate	Maximum observed backend SSL Key computation per second.
octavia_loadbalancer_ssl_cache_lookups_total	Total number of SSL session cache lookups.
octavia_loadbalancer_ssl_cache_misses_total	Total number of SSL session cache misses.
octavia_loadbalancer_http_comp_bytes_in_total	Number of bytes per second over last elapsed second, before http compression.
octavia_loadbalancer_http_comp_bytes_out_total	Number of bytes per second over last elapsed second, after http compression.
octavia_loadbalancer_limit_http_comp	Configured maximum input compression rate in bytes.
octavia_loadbalancer_listeners	Current number of active listeners.
octavia_loadbalancer_dropped_logs_total	Total number of dropped logs.
octavia_listener_status	Current status of the listener. 0=OFFLINE, 1=ONLINE, 2=DEGRADED.
octavia_listener_current_sessions	Current number of active sessions.
octavia_listener_max_sessions	Maximum observed number of active sessions.
octavia_listener_limit_sessions	Configured session limit.
octavia_listener_sessions_total	Total number of sessions.
octavia_listener_limit_session_rate	Configured limit on new sessions per second.
octavia_listener_max_session_rate	Maximum observed number of sessions per second.
octavia_listener_connections_rate_max	Maximum observed number of connections per second.
octavia_listener_connections_total	Total number of connections.
octavia_listener_bytes_in_total	Current total of incoming bytes.
octavia_listener_bytes_out_total	Current total of outgoing bytes.
octavia_listener_requests_denied_total	Total number of denied requests.
octavia_listener_responses_denied_total	Total number of denied responses.
octavia_listener_request_errors_total	Total number of request errors.
octavia_listener_denied_connections_total	Total number of requests denied by connection rules.
octavia_listener_denied_sessions_total	Total number of requests denied by session rules.
octavia_listener_failed_header_rewriting_total	Total number of failed header rewriting rules.
octavia_listener_http_requests_rate_max	Maximum observed number of HTTP requests per second.
octavia_listener_http_requests_total	Total number of HTTP requests received.
octavia_listener_http_responses_total	Total number of HTTP responses.
octavia_listener_intercepted_requests_total	Total number of intercepted HTTP requests.
octavia_listener_http_cache_lookups_total	Total number of HTTP cache lookups.
octavia_listener_http_cache_hits_total	Total number of HTTP cache hits.
octavia_listener_http_comp_bytes_in_total	Total number of HTTP response bytes fed to the compressor.
octavia_listener_http_comp_bytes_out_total	Total number of HTTP response bytes emitted by the compressor.
octavia_listener_http_comp_bytes_bypassed_total	Total number of bytes that bypassed the HTTP compressor (CPU/BW limit).
octavia_listener_http_comp_responses_total	Total number of HTTP responses that were compressed.
octavia_pool_status	Current status of the pool. 0=OFFLINE, 1=ONLINE.
octavia_pool_current_sessions	Current number of active sessions.
octavia_pool_max_sessions	Maximum observed number of active sessions.
octavia_pool_limit_sessions	Configured session limit.
octavia_pool_sessions_total	Total number of sessions.
octavia_pool_max_session_rate	Maximum observed number of sessions per second.
octavia_pool_last_session_seconds	Number of seconds since last session assigned to a member.
octavia_pool_current_queue	Current number of queued requests.
octavia_pool_max_queue	Maximum observed number of queued requests.
octavia_pool_connection_attempts_total	Total number of connection establishment attempts.
octavia_pool_connection_reuses_total	Total number of connection reuses.
octavia_pool_bytes_in_total	Current total of incoming bytes.
octavia_pool_bytes_out_total	Current total of outgoing bytes.
octavia_pool_queue_time_average_seconds	Avg. queue time for last 1024 successful connections.
octavia_pool_connect_time_average_seconds	Avg. connect time for last 1024 successful connections.
octavia_pool_response_time_average_seconds	Avg. response time for last 1024 successful connections.
octavia_pool_total_time_average_seconds	Avg. total time for last 1024 successful connections.
octavia_pool_max_queue_time_seconds	Maximum observed time spent in the queue
octavia_pool_max_connect_time_seconds	Maximum observed time spent waiting for a connection to complete
octavia_pool_max_response_time_seconds	Maximum observed time spent waiting for a member response.
octavia_pool_max_total_time_seconds	Maximum observed total request+response time (request+queue+connect+response+processing)
octavia_pool_requests_denied_total	Total number of denied requests.
octavia_pool_responses_denied_total	Total number of denied responses.
octavia_pool_connection_errors_total	Total number of connection errors.
octavia_pool_response_errors_total	Total number of response errors.
octavia_pool_retry_warnings_total	Total number of retry warnings.
octavia_pool_redispatch_warnings_total	Total number of redispatch warnings.
octavia_pool_failed_header_rewriting_total	Total number of failed header rewriting warnings.
octavia_pool_client_aborts_total	Total number of data transfers aborted by the client.
octavia_pool_member_aborts_total	Total number of data transfers aborted by the server.
octavia_pool_active_members	Current number of active members.
octavia_pool_backup_members	Current number of backup members.
octavia_pool_check_up_down_total	Total number of UP->DOWN transitions.
octavia_pool_check_last_change_seconds	Number of seconds since the last UP<->DOWN transition.
octavia_pool_downtime_seconds_total	Total downtime (in seconds) for the pool.
octavia_pool_loadbalanced_total	Total number of times a pool was selected, either for new sessions, or when redispatching.
octavia_pool_http_requests_total	Total number of HTTP requests received.
octavia_pool_http_responses_total	Total number of HTTP responses.
octavia_pool_http_cache_lookups_total	Total number of HTTP cache lookups.
octavia_pool_http_cache_hits_total	Total number of HTTP cache hits.
octavia_pool_http_comp_bytes_in_total	Total number of HTTP response bytes fed to the compressor.
octavia_pool_http_comp_bytes_out_total	Total number of HTTP response bytes emitted by the compressor.
octavia_pool_http_comp_bytes_bypassed_total	Total number of bytes that bypassed the HTTP compressor (CPU/BW limit).
octavia_pool_http_comp_responses_total	Total number of HTTP responses that were compressed.
octavia_member_status	Current status of the member. 0=ERROR, 1=ONLINE, 2=OFFLINE, 3=DRAIN.
octavia_member_current_sessions	Current number of active sessions.
octavia_member_max_sessions	Maximum observed number of active sessions.
octavia_member_limit_sessions	Configured session limit.
octavia_member_sessions_total	Total number of sessions.
octavia_member_max_session_rate	Maximum observed number of sessions per second.
octavia_member_last_session_seconds	Number of seconds since last session assigned to the member.
octavia_member_current_queue	Current number of queued requests.
octavia_member_max_queue	Maximum observed number of queued requests.
octavia_member_queue_limit	Configured maxqueue for the member (0 meaning no limit).
octavia_member_bytes_in_total	Current total of incoming bytes.
octavia_member_bytes_out_total	Current total of outgoing bytes.
octavia_member_queue_time_average_seconds	Avg. queue time for last 1024 successful connections.
octavia_member_connect_time_average_seconds	Avg. connect time for last 1024 successful connections.
octavia_member_response_time_average_seconds	Avg. response time for last 1024 successful connections.
octavia_member_total_time_average_seconds	Avg. total time for last 1024 successful connections.
octavia_member_max_queue_time_seconds	Maximum observed time spent in the queue
octavia_member_max_connect_time_seconds	Maximum observed time spent waiting for a connection to complete
octavia_member_max_response_time_seconds	Maximum observed time spent waiting for a member response.
octavia_member_max_total_time_seconds	Maximum observed total request+response time (request+queue+connect+response+processing)
octavia_member_connection_attempts_total	Total number of connection establishment attempts.
octavia_member_connection_reuses_total	Total number of connection reuses.
octavia_member_responses_denied_total	Total number of denied responses.
octavia_member_connection_errors_total	Total number of connection errors.
octavia_member_response_errors_total	Total number of response errors.
octavia_member_retry_warnings_total	Total number of retry warnings.
octavia_member_redispatch_warnings_total	Total number of redispatch warnings.
octavia_member_failed_header_rewriting_total	Total number of failed header rewriting warnings.
octavia_member_client_aborts_total	Total number of data transfers aborted by the client.
octavia_member_server_aborts_total	Total number of data transfers aborted by the server.
octavia_member_weight	Member weight.
octavia_member_check_failures_total	Total number of failed check (Only counts checks failed when the member is up).
octavia_member_check_up_down_total	Total number of UP->DOWN transitions.
octavia_member_downtime_seconds_total	Total downtime (in seconds) for the member.
octavia_member_check_last_change_seconds	Number of seconds since the last UP<->DOWN transition.
octavia_member_current_throttle	Current throttle percentage for the member, when slowstart is active, or no value if not in slowstart.
octavia_member_loadbalanced_total	Total number of times a member was selected, either for new sessions, or when redispatching.
octavia_member_http_responses_total	Total number of HTTP responses.
octavia_member_idle_connections_current	Current number of idle connections available for reuse
octavia_member_idle_connections_limit	Limit on the number of available idle connections

Go further

If you want to automate this scenario with Terraform and our managed Grafana service, you can find a ready-to-run example on this GitHub repository.

Browse all the Networking guides.

Browse the rest of the OVHcloud US documentation for guides on our other products and services.

Learn more about Public Cloud Network on the OVHcloud US website.

If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for a custom analysis of your project.