Skip to content
Snippets Groups Projects
Commit 723c3297 authored by Václav Bartoš's avatar Václav Bartoš
Browse files

doc: added instructions to process syslog data

parent 381a1590
No related branches found
No related tags found
No related merge requests found
# Data ingestion - syslog
Syslog messages can be forwarded and stored into SOCtools. A simple NiFi pipeline can be configured to receive and store any syslog message sent to a specific port.
## 1. NiFi pipeline
A simple pipeline listening for rsyslog messages on port 6010.
Create the following pipeline in Data processing / Data input / Custom data inputs:
<img src="images/syslog-pipeline.png" width=359>
Configuration of the "ListenSyslog" processor:
<img src="images/syslog-listener.png" width=608>
Increase "Max Number of TCP Connections" if you are going to send data from many sources.
Configuration of the "AttributesToJSON" processor:
<img src="images/syslog-attr2json.png" width=608>
Configuration of the "UpdateAttributes" processor:
<img src="images/syslog-setinted.png" width=608>
Custom parsing of message body can be done by additional processors.
(TODO add data type conversion)
## 2. rsyslog configuration
Configure rsyslog on the source machine to send all (or selected) logs to `<soctoolsproxy>:6010`.
This can be usually done by creating a file `/etc/rsyslog.d/soctools.conf` with the following content.
```
# Send all logs to SOCtools server
# Send the fully-qualified domain name instead of the hostname
$PreserveFQDN on
*.* @@<CHANGEME:soctoolsproxy>:6010
```
Then just restart rsyslog:
```
sudo systemctl restart rsyslog
```
## 3. Kibana
When some syslog data are succesfully received, an index pattern must be created in Kibana to be able to see it.
Go to Kibana/Management/Index patterns, click on "Create index pattern" and create the pattern `syslog-*`.
Then, the data will be available on Discover page when `syslog-*` index pattern is selected. A saved search and/or dashboard can be created to show the data in user's preferred way.
doc/images/syslog-attr2json.png

12.3 KiB

doc/images/syslog-listener.png

31.4 KiB

doc/images/syslog-pipeline.png

65.7 KiB

doc/images/syslog-setindex.png

13.4 KiB

0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment