--- title: "Logs Data Platform - Dedicated Input (Logstash)" description: "Learn how to deploy a personalized Logstash having a specific configuration and send logs from any source to your stream directly on the Logs Data Platform." url: https://docs.us.ovhcloud.com/en/manage-and-operate/logs-data-platform/dedicated-input-logstash lang: en --- > For AI agents: the complete documentation index is available at https://docs.us.ovhcloud.com/en/llms.txt, the full documentation bundle is available at https://docs.us.ovhcloud.com/en/llms-full.txt. # Logs Data Platform - Dedicated Input (Logstash) **Learn how to deploy a personalized Logstash having a specific configuration and send logs from any source to your stream directly on the Logs Data Platform.** [Logstash](https://github.com/elastic/logstash) is an open-source software developed by Elastic. Based on its features, it is possible to send messages from several inputs to different types of output using a variety of codecs, while processing them and transforming them in the process. You can learn a lot more about it on [the official website](https://www.elastic.co/products/logstash).   ## Topics - [What is Logstash?](#what-is-logstash) - [How do I configure Logstash Collector on Logs Data Platform?](#host-a-logstash-collector-on-logs-data-platform) - [What is special about our hosted Logstash? Available plugins, SSL Certificates, and more.](#additional-information)   ## Requirements If you are new to Logs Data Platform, completion of the [Quick Start Tutorial](https://docs.us.ovhcloud.com/en/manage-and-operate/logs-data-platform/getting-started.md) is highly recommended.   ## Instructions ### What is Logstash Logstash is a data processing pipeline that allows you to receive information from several sources, transforming it if necessary, before sending it to a variety of software. In the configuration file, there are three main parts to configure: - The Input part: details where your logs come from. For instance, some inputs may wait for logs and in some cases, inputs can fetch them from a database. - The Filter part: details how Logstash should parse and transform your logs messages. - The Output part: specifies the format and their destination. If you are hosting Logstash on your server, here are some sample configuration files. Below is a small configuration file example: ``` #input #The input part launches tcp server that will listen on port 5000 #Each incoming line of log will be tagged as a syslog type. input { tcp { port => 6514 type => syslog } } #Filter #The filter part applies a grok filter that will parse the message #It will also create new fields with information contained in the message. filter { grok { match => { "message" => "%{SYSLOGBASE}" } } date { match => [ "timestamp", "MMM dd HH:mm:ss" ] target => "timestamp" timezone => "Europe/Paris" } } #Output #The output part will then encode everything in a gelf format and send it to Logs Data Platform on the TCP GELF endpoint #Note that you need to change the token to use the one of your stream. output { gelf { host => ".logs.ovh.com" protocol => "TCP" port => 2202 custom_fields => ['X-OVH-TOKEN', 'da819874-5562-4a5f-b34e-fa8e708e8f16'] } } ``` This is one of many configuration examples you can use in your own Logstash to send syslog type logs to the Logs Data Platform. There are a lot of filters and plugins available in Logstash, and the list is growing! For example, by sending these kinds of lines into this Logstash: ``` Mar 25 13:20:36 bdiasse-wsant docker[7406]: time="2016-03-25T13:20:36.038884464+01:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]" Mar 25 13:20:36 bdiasse-wsant systemd[1]: Started docker container 5164ae90f489bc4479081c788b69b533a1e7a0be7e456f123cb05bb0542fc71b. Mar 25 13:20:36 bdiasse-wsant systemd[1]: Starting docker container 5164ae90f489bc4479081c788b69b533a1e7a0be7e456f123cb05bb0542fc71b. Mar 25 13:20:36 bdiasse-wsant avahi-daemon[743]: Withdrawing workstation service for veth4c25f95. Mar 25 13:20:36 bdiasse-wsant systemd[1]: Stopping ifup for veth4c25f95... Mar 25 13:20:36 bdiasse-wsant kernel: [337824.157113] eth0: renamed from veth4c25f95 Mar 25 13:20:36 bdiasse-wsant ifdown[2091]: /sbin/ifdown: interface veth4c25f95 not configured Mar 25 13:20:36 bdiasse-wsant systemd[1]: Stopped ifup for veth4c25f95. Mar 25 13:20:36 bdiasse-wsant NetworkManager[777]: devices removed (path: /sys/devices/virtual/net/veth4c25f95, iface: veth4c25f95) Mar 25 13:20:36 bdiasse-wsant kernel: [337824.172875] IPv6: ADDRCONF(NETDEV_CHANGE): vethd7be492: link becomes ready Mar 25 13:20:36 bdiasse-wsant kernel: [337824.172973] docker0: port 1(vethd7be492) entered forwarding state Mar 25 13:20:36 bdiasse-wsant kernel: [337824.172992] docker0: port 1(vethd7be492) entered forwarding state Mar 25 13:20:36 bdiasse-wsant NetworkManager[777]: (vethd7be492): enslaved to unknown device 3 docker0 Mar 25 13:20:36 bdiasse-wsant NetworkManager[777]: (vethd7be492): link connected Mar 25 13:20:36 bdiasse-wsant docker[7406]: time="2016-03-25T13:20:36.478880100+01:00" level=info msg="POST /v1.21/containers/5164ae90f489bc4479081c788b69b533a1e7a0be7e456f123cb05bb0542fc71b/resize?h=57&w=207" Mar 25 13:20:37 bdiasse-wsant avahi-daemon[743]: Joining mDNS multicast group on interface vethd7be492.IPv6 with address fe80::8808:c4ff:fea6:becc. Mar 25 13:20:37 bdiasse-wsant avahi-daemon[743]: New relevant interface vethd7be492.IPv6 for mDNS. Mar 25 13:20:37 bdiasse-wsant avahi-daemon[743]: Registering new address record for fe80::8808:c4ff:fea6:becc on vethd7be492.*. Mar 25 13:20:51 bdiasse-wsant kernel: [337839.195284] docker0: port 1(vethd7be492) entered forwarding state Mar 25 13:24:02 bdiasse-wsant puppet-agent[1377]: Skipping run of Puppet configuration client; administratively disabled (Reason: 'Disabled by default on new installations'); Mar 25 13:24:02 bdiasse-wsant puppet-agent[1377]: Use 'puppet agent --enable' to re-enable. ``` You will have the following result in your stream in Graylog: So, let's suppose you have a lot of hosts and you want to send all your syslog outputs to Logstash. One way to do it is to put one Logstash on one of your machines and send all the logs to this hosts. This will work but what happens if your hosts go down? Or is not fast enough? Or if it isn't compatible with Java? Don't worry, the best thing about the Logs Data Platform is that we can host your Logstash for you. We also allow you to configure the input and the filter as you wish and you can even define custom Groks. The next section will tell you how to do it!   ### Host a Logstash collector on Logs Data Platform The Logstash Collector comes with many advantages: - The output part of the Logstash is automatically configured by us. - You have your own certificate to enable SSL. This certificate can be used for example to trust only the TCP Inputs of your Logstash. This certificate and its associated key can be found at the following paths: **/etc/ssl/private/server.crt** for the cert and **/etc/ssl/private/server.key** for the key. The CA used to create these inputs is at the following location **/etc/ssl/private/ca.crt**. You will find this SSL CA for reference at the end of this document. For the http input with SSL enabled, a keystore with the key+cert+CA in PKCS12 format is present at the following location **/etc/ssl/private/keystore.jks** with keystore password "changeit". - And finally, we ensure that your collector is always up and running 24/7. To host this collector in the Logs Data Platform, you will need to configure it in the Logs Data Platform manager: - Check the **Data-gathering tools** tab and click on the button `Add a new data-gathering tool`. - The Manager will then ask you to provide an elegant name and a wonderful description. - Select your Software: Select **Logstash 9.x**. - Fill the main port of your input. We need to know which port you will use in your input configuration. We will put **6514** for now. This is also a setting you can change later. - If you want to restrict the Logstash to specific IP Addresses, you can use the **Trusted Networks** field to firewall your instances. Add every IP Address or subnet you want to allow, separated by comma. Leave it empty to allow any IP to contact it. Your collector will have a public IP Address, you can use this feature to prevent people to send you false information in it. - Select one of the streams you have to attached this collector to the stream you have. That means that every message that goes out of this collector is automatically routed to the stream. - Finally, select the number of instances you would need to handle your load. We recommend to have at least two instances to ensure the high availability of your Logstash. However, some use cases require you to limit the number of instances to one. **Note:** Note that you can also choose to _automatically scale the number of instances_. In this case, you need to select a minimum and maximum number of instances. Your input will then be started with the selected minimum number of instances. If these instances face a heavy load for a certain amount of time (more than 1 minute), your input will be automatically scaled up — one new instance at the time — until it reaches the selected maximum number of instances. On the other hand, if the load decreases, this number of instances will be scaled down until it reaches the minimum number of instances. Click on `Next` button to display the Logstash Configuration Page.   #### Logstash Collector configuration There are three sections in this page: - The Input section where you put the Input part of your configuration file. - The Filter section where you put the filter part. - The grok patterns sections where you can create your own grok patterns. All the inputs have preconfigured SSL certificates at the fixed locations used in the configuration below. If we enable the SSL encryption, the configuration needs five extra parameters: - **ssl\_enabled**: that allows to enable SSL. Must be set to `true`. - **ssl\_client\_authentication**: this deactivates the client certificate verification process that needs a trusted certificate on client side. - **ssl\_certificate**: the location of the auto-generated server certificate. - **ssl\_key**: the location of the associated key. - **ssl\_extra\_chain\_certs**: the array contains the path to the CA certificate. Note that the input grammar configuration is already on the screen. input \{ 'Input' section ``` tcp { port => 6514 type => syslog ssl_enabled => true ssl_client_authentication => "none" ssl_certificate => "/etc/ssl/private/server.crt" ssl_key => "/etc/ssl/private/server.key" ssl_extra_chain_certs => ["/etc/ssl/private/ca.crt"] } ``` } For the filter, you would use the same configuration in the example at the beginning of this guide. filter \{ 'Filter' section ``` grok { match => { "message" => "%{SYSLOGBASE}" } } date { match => [ "timestamp", "MMM dd HH:mm:ss" ] target => "timestamp" timezone => "Europe/Paris" } ``` } The last section of the configuration is dedicated to custom Grok Filters. If you know about Grok, you can use this text area to create your own grok that you can use in the filter section above. It is an advanced usage of Logstash. Head to the [end of the document](#useful-resources) to get some reference links that can help you use this functionality. To validate the settings, you will have to click on `Test the configuration` button. After some calculations you should see on the screen a green confirmation. You can then click on `Finish the configuration` button to save your configuration!   #### Starting the input Starting the input is done by clicking the _more options_ `...` button to the right of your input and selecting `Start`. This will deploy your collector on Logs Data Platform.   #### Collector useful information Once your collector is started, you can copy several pieces of information to your clipboard. - The hostname - The public IP - The SSL Certificate The hostname is in the following form: `-XXXXXXXXXXXXXXXXXXXXXXX..logs.ovh.com` This is an address of your collector for the cluster on Logs Data Platform. Send your logs to your address to have them processed by your collector using the following: ``` echo -e '{"version":"1.1",  "_X-OVH-TOKEN":"", "host": "example.org", "short_message": "A short message that helps you identify what is going on", "full_message": "Backtrace here\n\nmore stuff","some_metric_num": 42.0 }\0' | \ openssl s_client -quiet -no_ign_eof -connect -XXXXXXXXXXXXXXXXXXXXXXX..logs.ovh.com:6514 ``` If you head to Graylog, you will find your Logs in the attached Stream just as before.   ### Additional information #### Logstash Version The version hosted by Logs Data Platform is the latest Logstash 8. Of course we will update to the new versions as soon as they become available.   #### Logstash Plugins Below is a list of Logstash plugins we support. **Inputs plugins** ``` logstash-input-beats logstash-input-couchdb_changes logstash-input-dead_letter_queue logstash-input-elastic_serverless_forwarder logstash-input-elasticsearch logstash-input-ganglia logstash-input-gelf logstash-input-generator logstash-input-graphite logstash-input-heartbeat logstash-input-http logstash-input-http_poller logstash-input-jms logstash-input-opensearch logstash-input-redis logstash-input-relp logstash-input-stdin logstash-input-syslog logstash-input-tcp logstash-input-twitter logstash-input-cloudwatch logstash-input-s3 logstash-input-sqs logstash-input-jdbc logstash-input-logstash logstash-input-rabbitmq logstash-input-snmp logstash-input-snmptrap ``` **Input GELF plugin** We patched the GELF input to support TCP+TLS. Example Input Section: ``` tcp { port => 12202 type => gelf ssl_enabled => true ssl_client_authentication => "none" ssl_certificate => "/etc/ssl/private/server.crt" ssl_key => "/etc/ssl/private/server.key" ssl_extra_chain_certs => ["/etc/ssl/private/ca.crt"] codec => gelf { delimiter => "\x00" } } ```   #### filter plugins ``` logstash-filter-age logstash-filter-aggregate logstash-filter-anonymize logstash-filter-cidr logstash-filter-clone logstash-filter-csv logstash-filter-date logstash-filter-de_dot logstash-filter-dissect logstash-filter-dns logstash-filter-drop logstash-filter-elasticsearch logstash-filter-fingerprint logstash-filter-geoip logstash-filter-grok logstash-filter-hex logstash-filter-http logstash-filter-json logstash-filter-kv logstash-filter-math logstash-filter-memcached logstash-filter-metrics logstash-filter-mutate logstash-filter-opensearch logstash-filter-prune logstash-filter-sleep logstash-filter-split logstash-filter-syslog_pri logstash-filter-throttle logstash-filter-translate logstash-filter-truncate logstash-filter-urldecode logstash-filter-useragent logstash-filter-uuid logstash-filter-xml logstash-filter-jdbc_streaming logstash-filter-jdbc_static ```   #### codec plugins ``` logstash-codec-avro logstash-codec-cef logstash-codec-collectd logstash-codec-dots logstash-codec-edn logstash-codec-edn_lines logstash-codec-es_bulk logstash-codec-fluent logstash-codec-gelf logstash-codec-graphite logstash-codec-json logstash-codec-json_lines logstash-codec-line logstash-codec-msgpack logstash-codec-multiline logstash-codec-netflow logstash-codec-plain logstash-codec-rubydebug logstash-codec-cloudfront logstash-codec-cloudtrail ``` The following plugins are disabled for security reasons: ``` logstash-input-exec logstash-input-file logstash-input-kafka logstash-input-pipe logstash-input-unix logstash-filter-ruby ```   **SSL CA certificate** The CA certificate you can use to verify the authority presented by our hosted collectors can be copied from the homepage of the Logs Data Platform manager.   **Console output** Last but not least, if you have concerns about the way your logstash is handling your logs, you may be interested to follow the messages coming from your collector. To do this, please go to the dedicated page by clicking on the `Console output` button.   #### Useful Resources Here are some links to help you go further with Logstash: - [Logstash official documentation](https://www.elastic.co/guide/en/logstash/current/index.html) - [Grok filters documentation](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html) - [Pushing Logs with a Forwarder - Filebeat (Linux)](https://docs.us.ovhcloud.com/en/manage-and-operate/logs-data-platform/filebeat-logs.md) - [Grok Constructor](http://grokconstructor.appspot.com/do/match) - [A Ruby regular expression editor](https://rubular.com/)   ## Go further Browse all the [Logs Data Platform guides](https://docs.us.ovhcloud.com/en/manage-and-operate/logs-data-platform/landing-page-logs-data-platform.md). Browse the rest of the [OVHcloud US documentation](https://docs.us.ovhcloud.com/en/index.md) for guides on our other products and services. Learn more about [Logs Data Platform](https://us.ovhcloud.com/identity-security-operations/logs-data-platform/) on the OVHcloud US website.