Commit 59395bf0 authored by ale's avatar ale
Browse files

Merge branch 'extend-log-collector' into 'master'

Add drop-in extension hooks to the log-collector rsyslog configuration

Closes #85

See merge request !207
parents 8fd2ea8e 9720fb16
Pipeline #16647 passed with stage
in 5 minutes and 53 seconds
......@@ -93,3 +93,18 @@ To add a new dashboard, or update an existing one, use Kibana's
dashboard API to download the dashboard and its related
visualizations. The API is available at
`/api/kibana/dashboards/export?dashboard=DASHBOARD_ID`.
## Customization of collection rules
The log-collector rsyslog configuration provides a number of extension
points to support drop-in customization of the rulesets. This is
achieved by placing RainerScript files (with a `.conf` extension) in
the following directories on the log-collector hosts:
* /etc/rsyslog-collector/templates.d/ - for directives and code
snippets to be included at the configuration top-level. Useful to
define templates, load modules, etc.
* /etc/rsyslog-collector/rules-structured.d/,
/etc/rsyslog-collector/rules-unstructured.d/ - modify the *incoming*
ruleset for structured and unstructured log types respectively. The
code snippets should include action() directives and conditionals.
......@@ -10,25 +10,33 @@
state: directory
owner: "docker-log-collector"
mode: 0700
- name: Create the rsyslog-collector spool dir
file:
path: "/var/spool/rsyslog-collector"
state: directory
owner: docker-log-collector
mode: 0700
- name: Install logrotate script for local files
template:
src: "log-collector.logrotate.j2"
dest: "/etc/logrotate.d/log-collector"
- name: Install rsyslog-collector.conf
template:
src: "rsyslog-collector.conf.j2"
dest: "/etc/rsyslog-collector.conf"
notify: "restart rsyslog-collector"
- name: Create rsyslog-collector include dirs
file:
path: "/etc/rsyslog-collector/{{ item }}"
state: directory
loop:
- templates.d
- rules-structured.d
- rules-unstructured.d
- name: Create the rsyslog-collector normalization rules dir
file:
path: /etc/rsyslog-collector-lognorm
state: directory
- name: Install rsyslog-collector normalization rules
copy:
src: "{{ item }}"
......@@ -37,3 +45,9 @@
- files/lognorm/*
notify: "restart rsyslog-collector"
- name: Configure rsyslog-collector
template:
src: "rsyslog-collector.conf.j2"
dest: "/etc/rsyslog-collector.conf"
notify: "restart rsyslog-collector"
......@@ -132,6 +132,13 @@ module(
load="mmanon"
)
# Extension point for top-level directives and definitions (templates,
# module loading, etc).
include(
file="/etc/rsyslog-collector/templates.d/*.conf"
mode="optional"
)
# For incoming logs:
# - dump everything to file for debugging purposes
# - autodetect Lumberjack structured logs and parse them
......@@ -179,6 +186,12 @@ ruleset(name="incoming"){
queue.spoolDirectory="/var/spool/rsyslog"
action.resumeretrycount="-1")
} else {
# Extension point for rules applying to structured logs.
include(
file="/etc/rsyslog-collector/rules-structured.d/*.conf"
mode="optional"
)
# Normal structured log present in the default syslog flow. Send
# straight to Elasticsearch, skipping the log normalization step.
action(type="omelasticsearch"
......@@ -239,6 +252,12 @@ ruleset(name="incoming"){
}
{% endfor %}
# Extension point for rules applying to unstructured logs.
include(
file="/etc/rsyslog-collector/rules-unstructured.d/*.conf"
mode="optional"
)
action(type="mmnormalize"
rulebase="/etc/rsyslog-collector-lognorm/audit.rb")
action(type="mmnormalize"
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment