diff --git a/doc/dataingestion_syslog.md b/doc/dataingestion_syslog.md new file mode 100644 index 0000000000000000000000000000000000000000..b64e14433139ef9c3f2cc43768fe2f072641634d --- /dev/null +++ b/doc/dataingestion_syslog.md @@ -0,0 +1,59 @@ +# Data ingestion - syslog + +Syslog messages can be forwarded and stored into SOCtools. A simple NiFi pipeline can be configured to receive and store any syslog message sent to a specific port. + +## 1. NiFi pipeline + +A simple pipeline listening for rsyslog messages on port 6010. + +Create the following pipeline in Data processing / Data input / Custom data inputs: + +<img src="images/syslog-pipeline.png" width=359> + +Configuration of the "ListenSyslog" processor: + +<img src="images/syslog-listener.png" width=608> + +Increase "Max Number of TCP Connections" if you are going to send data from many sources. + +Configuration of the "AttributesToJSON" processor: + +<img src="images/syslog-attr2json.png" width=608> + +Configuration of the "UpdateAttributes" processor: + +<img src="images/syslog-setinted.png" width=608> + +Custom parsing of message body can be done by additional processors. + +(TODO add data type conversion) + +## 2. rsyslog configuration + +Configure rsyslog on the source machine to send all (or selected) logs to `<soctoolsproxy>:6010`. +This can be usually done by creating a file `/etc/rsyslog.d/soctools.conf` with the following content. + +``` +# Send all logs to SOCtools server + +# Send the fully-qualified domain name instead of the hostname +$PreserveFQDN on + +*.* @@<CHANGEME:soctoolsproxy>:6010 + +``` + +Then just restart rsyslog: + +``` +sudo systemctl restart rsyslog +``` + +## 3. Kibana + +When some syslog data are succesfully received, an index pattern must be created in Kibana to be able to see it. + +Go to Kibana/Management/Index patterns, click on "Create index pattern" and create the pattern `syslog-*`. + +Then, the data will be available on Discover page when `syslog-*` index pattern is selected. A saved search and/or dashboard can be created to show the data in user's preferred way. + diff --git a/doc/images/syslog-attr2json.png b/doc/images/syslog-attr2json.png new file mode 100644 index 0000000000000000000000000000000000000000..4c7b75a183455c31056fb3becdebff0f790281ed Binary files /dev/null and b/doc/images/syslog-attr2json.png differ diff --git a/doc/images/syslog-listener.png b/doc/images/syslog-listener.png new file mode 100644 index 0000000000000000000000000000000000000000..ac4ad12751dce34f72e4f088bc8fd392a1ecc6c0 Binary files /dev/null and b/doc/images/syslog-listener.png differ diff --git a/doc/images/syslog-pipeline.png b/doc/images/syslog-pipeline.png new file mode 100644 index 0000000000000000000000000000000000000000..0ef17cab5efbabe337a4c02530d969b1b5b5bfbe Binary files /dev/null and b/doc/images/syslog-pipeline.png differ diff --git a/doc/images/syslog-setindex.png b/doc/images/syslog-setindex.png new file mode 100644 index 0000000000000000000000000000000000000000..a3cadf7b2488be13fec99eb4b86d30ef8d0b792d Binary files /dev/null and b/doc/images/syslog-setindex.png differ