diff --git a/doc/dataingestion.md b/doc/dataingestion.md index 8b137891791fe96927ad78e64b0aad7bded08bdc..128a444f240654cb0f6deb6cf17f5ff6a05a9023 100644 --- a/doc/dataingestion.md +++ b/doc/dataingestion.md @@ -1 +1,69 @@ +# Data ingestion +SOCTools monitors itself which means that there is already support for receiving and parsing the following data: +* Misp +* Haproxy +* Kibana +* Keycloak +* Mysql +* Zookeeper +* Nifi +* Elasticsearch + +In addtion there is also support for: +* Suricata EVE logs +* Zeek logs + +Additional logs can be sent to the SOCTools server on port 6000 using Filebeat. The typical configuration is: + +``` +filebeat.inputs: +- type: log + paths: + - /opt/nifi/nifi-current/logs/nifi-app.log + fields: + log_type: nifi + +output.logstash: + hosts: ["soctools.example.com:6000"] + workers: 3 + loadbalance: true + ``` + +The extra filed log_type tells Nifi how it should route the data to the correct parser. The following values are currently supported: +* elasticsearch +* haproxy +* keycloak +* kibana +* misp +* mysql +* nifi +* suricata +* zeek +* zookeeper + +## New log types + +New unsupported log types can be sent to SOCTools port 6006 using Filebeat. Similar configuration as above. By default new data types will be sent to the index logs-custom-unknown. Proper parsing of new log types can be added to the process group "Custom data inputs". + +To specify fields that should be enriched, the following attributes can be added to the flow records: +* enrich_ip1 and enrich_ip2 +* enrich_domain1 and enrich_domain2 +* enrich_fqdn1 and enrich_fqdn2 + +Each attribute should be set to the [NiFi RecordPath](https://nifi.apache.org/docs/nifi-docs/html/record-path-guide.html) of the field to be enriched. + +### Enrichment example +Assume you have the following log data: + +``` +{ + "timestamp" : "2021-02-05T10:05:09.000Z", + "client" : { + "ip" : "172.22.0.1" + }, + "verb" : "GET" +} +``` + +You want to enrich the client IP so you set the attribute enrich_ip1 to the value "/client/ip". To see more example and to see how logs are parsed, take a look at the process group "Data processing"->"Data input"->"SOCTools" in the NiFi GUI.