Skip to content
Snippets Groups Projects

Data ingestion

SOCTools monitors itself which means that there is already support for receiving and parsing the following data:

  • Misp
  • Haproxy
  • Kibana
  • Keycloak
  • Mysql
  • Zookeeper
  • Nifi
  • Elasticsearch

In addtion there is also support for:

  • Suricata EVE logs
  • Zeek logs

Additional logs can be sent to the SOCTools server on port 6000 using Filebeat. The typical configuration is:

filebeat.inputs:
- type: log
  paths:
      - /opt/nifi/nifi-current/logs/nifi-app.log
   fields:
    log_type: nifi

output.logstash:
  hosts: ["soctools.example.com:6000"]
  workers: 3
  loadbalance: true

The extra filed log_type tells Nifi how it should route the data to the correct parser. The following values are currently supported:

  • elasticsearch
  • haproxy
  • keycloak
  • kibana
  • misp
  • mysql
  • nifi
  • suricata
  • zeek
  • zookeeper

Support for shipping logs over TLS will be added in a future version of SOCTools.

New log types

New unsupported log types can be sent to SOCTools port 6006 using Filebeat. Similar configuration as above. By default new data types will be sent to the index logs-custom-unknown. Proper parsing of new log types can be added to the process group "Custom data inputs".

To specify fields that should be enriched, the following attributes can be added to the flow records:

  • enrich_ip1 and enrich_ip2
  • enrich_domain1 and enrich_domain2
  • enrich_fqdn1 and enrich_fqdn2

Each attribute should be set to the NiFi RecordPath of the field to be enriched.

Enrichment example

Assume you have the following log data:

{
    "timestamp" : "2021-02-05T10:05:09.000Z",
    "client" : {
      "ip" : "172.22.0.1"
    },
    "verb" : "GET"
}

You want to enrich the client IP so you set the attribute enrich_ip1 to the value "/client/ip". To see more example and to see how logs are parsed, take a look at the process group "Data processing"->"Data input"->"SOCTools" in the NiFi GUI.