# Data ingestion

SOCTools monitors itself which means that there is already support for receiving and parsing the data from the following systems:
* Misp
* Haproxy
* OpenSearch Dashboards
* Keycloak
* Mysql
* Zookeeper
* Nifi
* OpenSearch

In addition, there is also support for:
* Suricata EVE logs
* Zeek logs

Additional logs of this type can be sent to the SOCTools server on ports 6000 or 6001 using Filebeat. The typical configuration is:

```
filebeat.inputs:
- type: log
  paths:
      - /opt/nifi/nifi-current/logs/nifi-app.log
   fields:
    log_type: nifi

output.logstash:
  hosts: ["soctools.example.com:6000"]
  workers: 3
  loadbalance: true
  ```

The extra field `log_type` tells Nifi how it should route the data to the correct parser. The following values are currently supported:
* elasticsearch
* haproxy
* keycloak
* kibana
* misp
* mysql
* nifi
* suricata
* zeek
* zookeeper

If any other type of logs is received on port 6000 or 6001, it will be sent to the index `logs-custom-unknown`.

Support for shipping logs over TLS will be added in a future version of SOCTools.

## New log types

New unsupported log types can be sent to SOCTools port 6006 using Filebeat. 
They will be received by the `Data processing -> Data input -> Common ListenBeats` process group.
Proper parsing of new log types can be added to the process group `Custom data inputs` and output of `Common ListenBeats`
should be connected to `Common ListenBeats`.  

If enrichment is desired on these logs, the following attributes should be added (use `UpdateAttribute` processor) to the flow records to specify fields that should be enriched:
* `enrich_ip1` and `enrich_ip2`
* `enrich_domain1` and `enrich_domain2`
* `enrich_fqdn1` and `enrich_fqdn2`

Each attribute should be set to the [NiFi RecordPath](https://nifi.apache.org/docs/nifi-docs/html/record-path-guide.html) of the field to be enriched.

### Enrichment example
Assume you have the following log data:

```
{
    "timestamp" : "2021-02-05T10:05:09.000Z",
    "client" : {
      "ip" : "172.22.0.1"
    },
    "verb" : "GET"
}
```

You want to enrich the client IP, so you set the attribute `enrich_ip1` to the value `/client/ip`. To see more examples and to see how logs are parsed, take a look at the process group `Data processing -> Data input -> SOCTools` in the NiFi GUI.