diff --git a/doc/administration.md b/doc/administration.md index 69b35eaf3de9cffbba513f7e37dd8dabe5faa648..d4131c86627fe5a9225773f2b5a04a378ec498d4 100644 --- a/doc/administration.md +++ b/doc/administration.md @@ -1,17 +1,77 @@ # SOCtools Administration Guide -TODO: -Describe what components are there and how they work together, how data flow. (There already is architecture.md, review/update it) +SOCtools consists of several components - data are received, processed and enriched by NiFi and stored into OpenSearch. +A separate index is used for each type of log. +Stored data can be searched and visualized in OpenSearch Dashboards. +There is also an instance of MISP for collecting (and possibly sharing) threat intelligence, +The Hive for case management and Cortex for enrichment of data in The Hive. +More information can be found in `architecture.md`. + +All these components run in separate docker containers on the SOCtools server. +In most containers, the main process runs under [supervisord](supervisord.org/) (so it can be controlled by `supervisorctl` command). + +Connections to web interfaces of all the components are handled by [haproxy](https://www.haproxy.org/) (which runs in its own container). +Each component is made available via HTTPS on a different port (see `quickstart.md` for details). + +In case of problems with connections to some interfaces, haproxy stats page can be used to get more info: +`https://{soctoolsproxy}:8888/` + + +## Demonstration data + +Default installation of SOCtools contains some demonstration data - logs generated inside NiFi simulating alerts and +information about TLS connections from Suricata, and an example event in MISP. + +To get your SOCtools installation to a production-ready state, remove such data using the following steps: +- In NiFi, stop (or delete) the `Data processing / Data input / Test data` process group. +- In OpenSearch Dashboards, remove indices `logs-suricata-alerts-*` and `logs-suricata-tls-*` and related dashboards + (but you can keep them if you plan to send real Suricata data into SOCtools) +- In MISP, remove the `testevent`. + ## User management -How user management is handled (Keycloak)? Is everything centralized or are there "local" users in some applications? -How to add/edit user accounts +Keycloak is used to provide single sign on (primarily using x509 certificates) to web interfaces of all the components. + +The same set of user accounts should be configured in each of the components. +Because each system uses a different way to manage users, there is a web GUI developed specially for SCOtools, which +should be used to add or edit user accounts, and which ensures that a user account is correctly set up in Keycloak and +all the SOCtools components. +This GUI is called SOCtools user management and is available at `https://{soctoolsproxy}:5443/` + +The GUI shows the list of users defined in Keycloak (the primary source of data) in the top table. +There are also buttons to edit or delete the user accounts. +Below this, there are tables showing user accounts in individual components. +If everything works well, all tables should show the same set of users. + +There is no support for different access levels or roles for different users in the current version, +i.e. all users have the same permissions in all services, generally allowing read and write access to all data. + +### Add user + +To add a new user, use "Add new user ..." link on the top of the page and fill in its username and email +(both are mandatory and must be unique, some services use email as the primary identifier). +First and last names are optional. +The user account is created in Keycloak and all the services, and a new client certificate is generated. +An email message is sent to the user's address which contains a unique link allowing the user to download the certificate and private key. +He/she can use the certificate to authenticate to any of the services via Keycloak. + +Note: Currently, only certificate authentication is supported, password can't be set via the web GUI. +If needed, it can be set in Keycloak administration console (`https://{soctoolsproxy}:12443/`) - log in as `admin`, +go to "Users" page, select desired user and click "Edit", go to "Credentials" tab and set the password. +If the "temporary" switch is "on", the user will have to change the password on the next login. + +### Edit or delete user + +User can be edited (name and email can be changed) or deleted using buttons in the first table of the user management +page. +When a user account is deleted, it's removed from all the services (except Cortex, which doesn't allow to remove an +account, it's just locked instead) and the corresponding certificate is revoked. ## Data ingestion -How to forward logs from some servers/applications to SOCtools, what must be set up to in NiFi. +TODO: How to forward logs from some servers/applications to SOCtools, what must be set up to in NiFi. Other data sources except logs? Emails? @@ -20,10 +80,45 @@ How to set up data feeds in MISP and analyzers in Cortex. ## Data processing in NiFi -What the current NiFi pipeline does. How to reconfigure it. +TODO: What the current NiFi pipeline does. How to reconfigure it. + + +---- + +## Troubleshooting + +### Server restart + +When the server is restarted, SOCtools components are not started automatically in default installation. +Run `ansible-playbook -i inventories soctools.yml -t start-docker-containers` to get everything up again. + +### Component not running + +If any of the services (components) is not running, check that its docker container is running (`docker ps`) and that the processes +inside it are running (`docker exec -it <container_name> bash`, `supervisorctl status`). + +To start any stopped containers, run `ansible-playbook -i inventories soctools.yml -t start-docker-containers`. +To run a stopped process inside a container, run either `ansible-playbook -i inventories soctools.yml -t start -l <container-name>` +or `docker exec -it <container_name> bash`, `supervisorctl start all`. +Similar commands can be used to restart a running container/process. -## Other tools? +The following docker containers should be running: +``` +soctools-misp +soctools-mysql (needed by misp) +soctools-cortex +soctools-thehive +soctools-cassandra (needed by thehive) +soctools-haproxy +soctools-opensearch-dashboards +soctools-opensearch-1 +soctools-opensearch-2 +soctools-nifi-1 +soctools-nifi-2 +soctools-nifi-3 +soctools-zookeeper (dependency of nifi) +soctools-keycloak +``` -Is there anything in OpenSearch, OpenSearch Dahsboards, MISP, The Hive, etc., which is specific to SOCtools and should be described (i.e. can't be found in official documentation of these tools)?