Skip to content
Snippets Groups Projects
Commit 3edb8c40 authored by Arne Øslebø's avatar Arne Øslebø
Browse files

Update doc/dataingestion.md

parent 80b004bb
No related branches found
No related tags found
No related merge requests found
# Data ingestion
SOCTools monitors itself which means that there is already support for receiving and parsing the following data:
* Misp
* Haproxy
* Kibana
* Keycloak
* Mysql
* Zookeeper
* Nifi
* Elasticsearch
In addtion there is also support for:
* Suricata EVE logs
* Zeek logs
Additional logs can be sent to the SOCTools server on port 6000 using Filebeat. The typical configuration is:
```
filebeat.inputs:
- type: log
paths:
- /opt/nifi/nifi-current/logs/nifi-app.log
fields:
log_type: nifi
output.logstash:
hosts: ["soctools.example.com:6000"]
workers: 3
loadbalance: true
```
The extra filed log_type tells Nifi how it should route the data to the correct parser. The following values are currently supported:
* elasticsearch
* haproxy
* keycloak
* kibana
* misp
* mysql
* nifi
* suricata
* zeek
* zookeeper
## New log types
New unsupported log types can be sent to SOCTools port 6006 using Filebeat. Similar configuration as above. By default new data types will be sent to the index logs-custom-unknown. Proper parsing of new log types can be added to the process group "Custom data inputs".
To specify fields that should be enriched, the following attributes can be added to the flow records:
* enrich_ip1 and enrich_ip2
* enrich_domain1 and enrich_domain2
* enrich_fqdn1 and enrich_fqdn2
Each attribute should be set to the [NiFi RecordPath](https://nifi.apache.org/docs/nifi-docs/html/record-path-guide.html) of the field to be enriched.
### Enrichment example
Assume you have the following log data:
```
{
"timestamp" : "2021-02-05T10:05:09.000Z",
"client" : {
"ip" : "172.22.0.1"
},
"verb" : "GET"
}
```
You want to enrich the client IP so you set the attribute enrich_ip1 to the value "/client/ip". To see more example and to see how logs are parsed, take a look at the process group "Data processing"->"Data input"->"SOCTools" in the NiFi GUI.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment