dataingestion.md

Data ingestion

SOCTools monitors itself which means that there is already support for receiving and parsing the data from the following systems:

• Misp
• Haproxy
• OpenSearch Dashboards
• Keycloak
• Mysql
• Zookeeper
• Nifi
• OpenSearch

In addition, there is also support for:

• Suricata EVE logs
• Zeek logs

Additional logs of this type can be sent to the SOCTools server on ports 6000 or 6001 using Filebeat. The typical configuration is:

filebeat.inputs:
- type: log
  paths:
      - /opt/nifi/nifi-current/logs/nifi-app.log
   fields:
    log_type: nifi

output.logstash:
  hosts: ["soctools.example.com:6000"]
  workers: 3
  loadbalance: true

The extra field log_type tells Nifi how it should route the data to the correct parser. The following values are currently supported:

• elasticsearch
• haproxy
• keycloak
• kibana
• misp
• mysql
• nifi
• suricata
• zeek
• zookeeper

If any other type of logs is received on port 6000 or 6001, it will be sent to the index logs-custom-unknown.

Support for shipping logs over TLS will be added in a future version of SOCTools.

New log types

New unsupported log types can be sent to SOCTools port 6006 using Filebeat. They will be received by the Data processing -> Data input -> Common ListenBeats process group. Proper parsing of new log types can be added to the process group Custom data inputs and output of Common ListenBeats should be connected to Common ListenBeats.

If enrichment is desired on these logs, the following attributes should be added (use UpdateAttribute processor) to the flow records to specify fields that should be enriched:

• enrich_ip1 and enrich_ip2
• enrich_domain1 and enrich_domain2
• enrich_fqdn1 and enrich_fqdn2

Each attribute should be set to the NiFi RecordPath of the field to be enriched.

Enrichment example

Assume you have the following log data:

{
    "timestamp" : "2021-02-05T10:05:09.000Z",
    "client" : {
      "ip" : "172.22.0.1"
    },
    "verb" : "GET"
}

You want to enrich the client IP, so you set the attribute enrich_ip1 to the value /client/ip. To see more examples and to see how logs are parsed, take a look at the process group Data processing -> Data input -> SOCTools in the NiFi GUI.