--- title: "OpenSearch™ - Monitor your infra (with Logstash or Fluent Bit)" description: "Learn how to configure two famous collectors, Logstash and Fluent Bit, to connect and forward data logs to your Cloud Analytics with OpenSearch™ service." url: https://docs.us.ovhcloud.com/en/public-cloud/data-analytics/opensearch-logstash lang: en --- > For AI agents: the complete documentation index is available at https://docs.us.ovhcloud.com/en/llms.txt, the full documentation bundle is available at https://docs.us.ovhcloud.com/en/llms-full.txt. # OpenSearch™ - Monitor your infra (with Logstash or Fluent Bit) **Learn how to configure two famous collectors, Logstash and Fluent Bit, to connect and forward data logs to your Cloud Analytics with OpenSearch™ service.** OpenSearch™ is an open-source search and analytics suite. Several methods are available to push relevant data to OpenSearch™. You can upload a file easily, but for real-time data such as metrics and logs, collectors are required.   ## Requirements This tutorial requires: - Access to the [OVHcloud Control Panel](https://manager.us.ovhcloud.com/) - A [Public Cloud project](https://us.ovhcloud.com/public-cloud/) in your OVHcloud account, running at least: - one Cloud Analytics with OpenSearch™ service running; - one or more additional compute instances running Linux, acting as a data injector. It can also be your computer, a Kubernetes cluster, etc. **Warning:** **NOTE:** Make sure that you have a user account with enough privileges to write to the OpenSearch™ service, and that the IP addresses that you will use to run your agents (Logstash and Fluent Bit) are part of the authorized list. Check our [Getting started](https://docs.us.ovhcloud.com/en/public-cloud/databases/getting-started.md) guide for more information.   ### Software environment The main software components used to create this tutorial are: - OpenSearch™ 1.0 Public Cloud DB instance - Linux Ubuntu 21.10 / Apache2 2.4.48 to run: - Logstash 7.13.2 - Fluent Bit 1.8   ### OpenSearch™ Agents and ingestion tools compatibility Matrix Due to the fork from ElasticSearch, it is recommended to verify the software version that you will use: [OpenSearch™ Compatibility Matrices](https://opensearch.org/docs/latest/clients/agents-and-ingestion-tools/index/#compatibility-matrices).   ## Instructions ### Step 1: Logstash as the data log source #### Install Logstash on client sources To collect data, you need to install the collector agent on the data source. In our case, it's a Linux Ubuntu virtual machine, but it can be anything. Please refer to the official [OpenSearch™ Logstash installation documentation](https://opensearch.org/docs/latest/clients/logstash/index/). As detailed in the compatibility matrix, select and download a compatible release from ElasticSearch. Click [here](https://www.elastic.co/fr/downloads/past-releases#logstash-oss) to download Logstash OSS past releases.   #### Configure Logstash The default configuration is sufficient for this tutorial, but in case you need to review the configuration, go to _/etc/logstash_ and look at the configuration files (_logstash.yml_).   #### Configure a pipeline Every pipeline is based on three phases: inputs, filters, and outputs. For this tutorial, let's assume an Apache service is running. We will collect Apache access logs to configure a pipeline with: - the Apache access.log file as the input; - the filter is based on the predefined grok; - the OpenSearch™ instance as the target output (modified with your values). Configuration file: _/etc/logstash/conf.d/apache2.conf_ ``` input { file { path => "/var/log/apache2/access.log" } } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] } } output { opensearch { hosts => ["https://opensearch-682faf00-682faf00.database.cloud.ovh.us:20184"] index => "index_test-%{+YYYY.MM.dd}" user => "admin" password => "2fakeSVV5wvyPykF" } } ```   #### Grok patterns and debugger As an example of the main pattern matching that is currently used, or to test some more specific ones, you can use some tools like [Javainuse Grok Debugger](https://www.javainuse.com/grok):   ### Step 2: Visualize Logstash data in the OpenSearch™ Dashboard #### Validate that the index has been created and is now populated Now that we have Logstash running and parsing the Apache access log file, let's do a different connection to the Apache service and validate that Logstash is forwarding all access data to OpenSearch™. Let's also connect to the OpenSearch™ Dashboard to check that the corresponding index has been created and that some first documents have been populated. Use the `Dev Tools` to query the indices: Then, in the console, execute a _GET /\_cat/indices_ command to get the index list with the number of documents for each:   #### Create an index pattern To aggregate all corresponding daily indices, we have to create an index pattern from the `Dashboard Management` menu: Click `+ Create index pattern`. As we want to consolidate all our daily indices, let's define the pattern as _index\_test_: Then define the timestamp: OpenSearch™ will then define all fields available and their type directly from stored documents in the index:   #### Create a new dashboard ##### **Create a new visualization** In the OpenSearch™ Dashboard, click `Visualize`, then `+ Create new visualization`. Select the visualization type: Then select the data source (the index pattern you just created): You now have to define all the criteria that you need to visualize: - the period (last 7 days in our example); - for the metrics, the Y-axis data (aggregation/field); - and the same for Buckets with time intervals in case of data aggregation. Click `Update` after making changes to the data definition, or refresh if you change the data time scale. Save your visualization:   ##### **Add visualizations to a dashboard** Let's make a dashboard with the visualization you've just saved. Create a new dashboard: As we have already created a visualization, _Add an existing object_: Search and select the one we just created: Reorganize your panels, and add new or existing ones as wanted: We now have a real-time dashboard showing us the unique IP addresses per day, found in Apache logs.   ### Step 3: Fluent Bit alternative as the data log source Multiple agents exist. [Fluent Bit](https://fluentbit.io/) is a strong, open-source alternative to Logstash, and works natively with OpenSearch™. We propose now to parse the Apache access logs and push this information to the OpenSearch™ instance for this agent.   #### Install Fluent Bit To install Fluent Bit on our Linux Ubuntu instance, we will install the td-agent-bit package, as described in the Fluent Bit installation process from their [official documentation](https://docs.fluentbit.io/manual/installation/linux). Check that the service is running and is enabled to start automatically when the system boots: ```bash sudo systemctl enable td-agent-bit.service sudo systemctl status td-agent-bit.service ```   #### Configure Fluent Bit Let's modify the configuration file /etc/td-agent-bit/td-agent-bit.conf for the INPUT and OUTPUT section : ``` [INPUT] Name tail Tag test.file Path /var/log/apache2/access.log DB /var/log/apache2_access.db Path_Key filename Parser apache2 Mem_Buf_Limit 8MB Skip_Long_Lines On Refresh_Interval 30 [OUTPUT] Name es Match * Host opensearch-682faf00-682faf00.database.cloud.ovh.us Port 20184 tls On tls.verify off HTTP_user admin HTTP_Passwd 2FakeSVV5wvyPykF Logstash_Format True Logstash_Prefix my-fluent ``` If required, please refer to the parameters in the [Fluent Bit official documentation](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit). All Apache access logs will now be pushed to the OpenSearch™ instance under the "my-fluent" index.   ### Step 4: Visualize Fluent Bit data in OpenSearch™ Dashboard As we did for Logstash, you can create an index pattern to aggregate all the daily data logs stored in OpenSearch™, then visualize those data or create dashboards. In this example, the indices will be _my-fluent-YYYY.MM.DD_, so you can define a pattern like _my-fluent-_\*. Congratulations, you are now able to collect data from multiple sources and push them to Cloud Analytics with OpenSearch™!   ## Go further [OpenSearch™ official documentation](https://opensearch.org/docs/latest) [OpenSearch™ Dashboard official documentation](https://opensearch.org/docs/latest) [OpenSearch™ Project GitHub](https://github.com/opensearch-project) [Fluent Bit official documentation](https://docs.fluentbit.io/manual/) Browse all the [Analytics guides](https://docs.us.ovhcloud.com/en/public-cloud/data-analytics/landing-page-data-analytics.md). Browse the rest of the [OVHcloud US documentation](https://docs.us.ovhcloud.com/en/index.md) for guides on our other products and services. Learn more about [Cloud Analytics](https://us.ovhcloud.com/public-cloud/cloud-analytics/) on the OVHcloud US website.