Log Courier Suite

The Log Courier Suite is a set of lightweight tools created to ship and process log files speedily and securely, with low resource usage, to Elasticsearch or Logstash instances.

• Log Courier Suite
  • Log Courier
  • Log Carver
  • Compatible Logstash Versions
  • Philosophy
  • Documentation
    • Installation
    • Reference
  • Upgrading from 1.x to 2.x

Log Courier

Log Courier is a lightweight shipper. It reads from log files and transmits events over the Courier protocol to a remote Logstash or Log Carver instance.

• Reads from files or the program input (stdin)
• Follows log file rotations and movements
• Compliments log events with extra fields
• Reloads configuration without restarting
• Transmits securely using TLS with server and (optionally) client verification
• Monitors shipping speed and status which can be read using the Administration utility
• Pre-processes events on the sending side using codecs (e.g. Multiline, Filter)

Log Carver

(Beta)

Log Carver is a lightweight event processor. It receives events over the Courier protocol and performs actions against them to manipulate them into the required format for storage within Elasticsearch, or further processing in Logstash.

• Receives events securely using TLS with optional client verification
• Supports Common Expression Language (CEL) conditional expressions in If/ElseIf/Else target different actions against different events
• Provides several actions: date, geoip, user_agent, kv, add_tag, remove_tag, set_field, unset_field
• The set_field action supports Common Expression Language (CEL) for type conversions and string building
• Transmits events to Elasticsearch using the bulk API
• A small example configuration can be found here

Compatible Logstash Versions

Log Courier is compatible with most Logstash versions with a single exception.

• >=7.4.0 and <7.6.0 use a version of JRuby that has a bug making it incompatible and causes log-courier events to stop processing after an indeterminable amount of time (see #370) - please upgrade to 7.6.0 which updates JRuby to a compatible version.

Philosophy

• At-least-once delivery of events, a Log Courier crash should never lose events
• Be efficient, reliable and scalable
• Keep resource usage low
• Be easy to use

Documentation

Installation

• Public Repositories
• Building from Source

Reference

• Administration Utility
• Command Line Arguments
• Configuration
• Logstash Integration
• SSL Certificate Utility
• Change Log

Upgrading from 1.x to 2.x

There are many breaking changes in the configuration between 1.x and 2.x. Please check carefully the list of breaking changes here: Change Log.

Packages also now default to using a log-courier user. If you require the old behaviour of root, please be sure to modify the /etc/sysconfig/log-courier (CentOS/RedHat) or /etc/default/log-courier (Ubuntu) file.