Pushing Logs with a Logging Library - Python 3.x (logging-ldp)

Learn how to push your logs to Logs Data Platform using Python 3.x.

Logging-ldp is intended to be a high-performance logging formatter and handler to send log entries into Logs Data Platform.

This package includes:

a TCP/TLS handler to send log entries over TCP with TLS support.
a formatter to convert logging records into GELF(1.1).
a facility to ensure fields suit the LDP naming conventions.

Requirements

Python 3, we recommend installing pip
an activated Logs Data Platform account
at least one Stream and its token

Instructions

Install

Using pip

You can use pip to install logging-ldp, make sure you have the latest version:

$ pip3 install --upgrade pip 
[...]
Successfully installed pip-<version>

$ pip3 install --upgrade logging-ldp 
[...]
Successfully installed logging-ldp-<version> setuptools-18.3.1

Using sources

Instructions for installing logging-ldp are available on the OVH GitHub repository.

How to send logs

The following example shows how to send log in Graylog TCP input:

import logging
from logging_ldp.formatters import LDPGELFFormatter
from logging_ldp.handlers import LDPGELFTCPSocketHandler

def setup_logging():
    handler = LDPGELFTCPSocketHandler(hostname="<your_cluster>.logs.ovh.com")
    handler.setFormatter(LDPGELFFormatter(token="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"))
    logging.getLogger().addHandler(handler)
    logging.getLogger().setLevel(logging.INFO)

if __name__ == '__main__':
    setup_logging()

    logging.info("Test !")

Send additional static meta data

To automatically append meta data on all your logs, you can implement an alternate Schema:

import logging
from marshmallow import fields
from logging_ldp.formatters import LDPGELFFormatter
from logging_ldp.handlers import LDPGELFTCPSocketHandler
from logging_ldp.schemas import LDPSchema
def setup_logging():
    # Load your config there
    config = dict(name="myapp", version="0.0.1")
    # Define a custom Schema
    class MyApp(LDPSchema):
        app_name = fields.Constant(config['name'])
        app_version = fields.Constant(config['version'])

    handler = LDPGELFTCPSocketHandler(hostname="<your_cluster>.logs.ovh.com")
    handler.setFormatter(LDPGELFFormatter(token="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX", schema=MyApp))
    logging.getLogger().addHandler(handler)
    logging.getLogger().setLevel(logging.INFO)

if __name__ == '__main__':
    setup_logging()
    logging.info("Test !")

The log entry sent to Graylog will be something like this:

{
  "_app_name": "myapp",
  "_app_version": "0.0.1",
  "_X-OVH-TOKEN": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
  "file": "test_thread.py",
  "host": "cdumay-desk",
  "level": 6,
  "line": 36,
  "short_message": "Test !",
  "timestamp": 1556036745.6493497,
  "version": "1.1"
}

Send additional intermittent metadata

To define occasional metadata, you can define a Schema with Nested sub-items:

import logging
from marshmallow import Schema, fields
from logging_ldp.formatters import LDPGELFFormatter
from logging_ldp.handlers import LDPGELFTCPSocketHandler
from logging_ldp.schemas import LDPSchema

# Define a sub-schema
class User(Schema):
    name = fields.String(required=True)
    age = fields.Integer()

# Define a custom schema to apply on log entries
class AccountInfo(LDPSchema):
    user = fields.Nested(User, required=True)
    manager = fields.Nested(User)

def setup_logging():
    handler = LDPGELFTCPSocketHandler(hostname="<your_cluster>.logs.ovh.com")
    handler.setFormatter(LDPGELFFormatter(
        token="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
        schema=AccountInfo
    ))
    logging.getLogger().addHandler(handler)
    logging.getLogger().setLevel(logging.INFO)

if __name__ == '__main__':
    setup_logging()

    current_user = dict(name="John Doe")
    boss = dict(name="Roger Smith", age=51)
    logging.info("User has logged", extra=dict(user=current_user, manager=boss))

The log entry sent will be:

{
  "_manager_age_int": 51,
  "_manager_name": "Roger Smith",
  "_user_name": "John Doe",
  "_X-OVH-TOKEN": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
  "file": "test_thread.py",
  "host": "cdumay-desk",
  "level": 6,
  "line": 40,
  "short_message": "User has logged",
 "timestamp": 1556037587.4444475,
  "version": "1.1"
}

As we can see:

Objects are transformed to flatten dictionaries: manager.name is renamed manager_name.
Fields are types using the LDP naming convention: manager.age is renamed manager_age_int
Null values are not sent: user.age.
You can set required=True to force meta data value or default=xxx to add data automatically.

Go further

Browse all the Logs Data Platform guides.

Browse the rest of the OVHcloud US documentation for guides on our other products and services.

Learn more about Logs Data Platform on the OVHcloud US website.