Revert "thehive integration with keycloak"

This reverts commit d0fc936f

HOWTOS.md

+Howto's
+=======
+
+Modify main NiFi pipeline
+-------------------------
+
+To make modifications to the main NiFi pipeline and add it to the Ansible playbook, do the following in the soctool directory:
+
+* Make necesarry to the pipeline in the NiFi GUI
+* Copy flow.xml.gz file from one of the NiFi containers:  
+  `docker cp soctools-nifi-1:/opt/nifi/nifi-current/conf/flow.xml.gz .`
+* Convert flowx.xml.gz to new template  
+  `utils/flow2template.py flow.xml.gz roles/nifi/templates/flow.xml.j2`
+
+
+Update configuration files in docker containers using Ansible
+-------------------------------------------------------------
+To update configuration files for all docker containers together, run the following command:
+	ansible-playbook -i inventories soctools.yml -t update-config
+To update configuration files only for specific services, run the following commands:
+	ansible-playbook -i inventories soctools.yml -t update-keycloak-config
+	ansible-playbook -i inventories soctools.yml -t update-thehive-config
+        ansible-playbook -i inventories soctools.yml -t update-cortex-config
+        ansible-playbook -i inventories soctools.yml -t update-cassandra-config
+        ansible-playbook -i inventories soctools.yml -t update-haproxy-config
+        ansible-playbook -i inventories soctools.yml -t update-filebeat-config
+        ansible-playbook -i inventories soctools.yml -t update-nifi-config
+        ansible-playbook -i inventories soctools.yml -t update-odfees-config
+        ansible-playbook -i inventories soctools.yml -t update-odfekibana-config
+
+
+Restart services inside docker containers using Ansible
+-------------------------------------------------------
+To restart services for all docker containers together, run the following command:
+        ansible-playbook -i inventories soctools.yml -t restart
+To restart services only for specific docker containers, run the following commands:
+        ansible-playbook -i inventories soctools.yml -t restart-keycloak
+        ansible-playbook -i inventories soctools.yml -t restart-thehive
+        ansible-playbook -i inventories soctools.yml -t restart-cortex
+        ansible-playbook -i inventories soctools.yml -t restart-cassandra
+        ansible-playbook -i inventories soctools.yml -t restart-haproxy
+        ansible-playbook -i inventories soctools.yml -t restart-filebeat
+        ansible-playbook -i inventories soctools.yml -t restart-misp
+        ansible-playbook -i inventories soctools.yml -t restart-mysql
+        ansible-playbook -i inventories soctools.yml -t restart-nifi
+        ansible-playbook -i inventories soctools.yml -t restart-odfees
+        ansible-playbook -i inventories soctools.yml -t restart-odfekibana
+
+Stop services inside docker containers using Ansible
+----------------------------------------------------
+To stop services for all docker containers together, run the following command:
+        ansible-playbook -i inventories soctools.yml -t stop
+To stop services only for specific docker containers, run the following commands:
+        ansible-playbook -i inventories soctools.yml -t stop-keycloak
+        ansible-playbook -i inventories soctools.yml -t stop-thehive
+        ansible-playbook -i inventories soctools.yml -t stop-cortex
+        ansible-playbook -i inventories soctools.yml -t stop-cassandra
+        ansible-playbook -i inventories soctools.yml -t stop-haproxy
+        ansible-playbook -i inventories soctools.yml -t stop-filebeat
+        ansible-playbook -i inventories soctools.yml -t stop-misp
+        ansible-playbook -i inventories soctools.yml -t stop-mysql
+        ansible-playbook -i inventories soctools.yml -t stop-nifi
+        ansible-playbook -i inventories soctools.yml -t stop-odfees
+        ansible-playbook -i inventories soctools.yml -t stop-odfekibana
+
+Restart services inside docker containers manually
+--------------------------------------------------
+To restart services inside docker containers after changes in configuration files:
+	1. Attache container: docker exec -it container_id_or_name bash (example: docker exec -it soctools-keycloak bash)
+	2. List  services and their statuses: supervisorctl status
+	3. Restart service: supervisorctl restart supervisor_service_name (example: supervisorctl restart keycloak)
+	4. Detach from container: exit
+

README.md

@@ -13,36 +13,45 @@ Log in and install ansible:
 `yum -y install ansible git`  
 `ansible-galaxy collection install ansible.posix`
 
-Clone soctools:  
-`git clone https://scm.uninett.no/geant-wp8-t3.1/soctools.git`  
+Clone soctools:
+Temporary solution: Upload your ssh key to gitlab.geant.org
+`git clone git@gitlab.geant.org:gn4-3-wp8-t3.1-soc/soctools.git`
 `cd soctools`
 
 Install soctools:
-Edit group_vars/all/main.yml and change 'dslproxy' so that it point to the FQDN of the server.  
+Edit group_vars/all/main.yml and change 'soctoolsproxy' so that it point to the FQDN of the server.  
 `vi group_vars/all/main.yml`  
-The first entry in the soctools_users variable is the user with full admin privileges in NiFi and Kibana.
+Users are specified in the file:  
+`group_vars/all/users.yml`  
 
 To configure the server running soctools, run the ansible playbook:  
-`ansible-playbook -i soctools-inventory soctools_server.yml`
+`ansible-playbook -i inventories soctools_server.yml`
 
 To build the Docker images needed, run the ansible playbook:  
-`ansible-playbook -i soctools-inventory buildimages.yml`
+`ansible-playbook -i inventories buildimages.yml`
 
 To build the CA needed for host and user certificates, run the ansible playbook:  
-`ansible-playbook -i soctools-inventory buildca.yml`
+`ansible-playbook -i inventories buildca.yml`
 
-User certificates are can be found in the directory roles/ca/files/CA/private. Import into browser for authentication.
+If using soctools CA certificates provided with this installation, you first need to download and import root certificate found in secrets/CA/ca.crt   
+For Windows, CA certificate should be installed in Trusted Root Certification Authorities store. 
+
+User certificates are can be found in the directory secrets/certificates. Import into browser for authentication.
+For Windows, user certificate should be installed in Personal store. Passwords for the certificates can be found in the directory secrets/passwords.   
 
 To start the cluster, run the ansible playbook soctools.yml:  
-`ansible-playbook -i soctools-inventory soctools.yml -t start`
+`ansible-playbook -i inventories soctools.yml -t start`
 
 To stop the cluster, run the ansible playbook soctools.yml:  
-`ansible-playbook -i soctools-inventory soctools.yml -t stop`
-
-The NiFi interface should now be available on port 9443 on the server.  
-The OpenDistro for Elasticsearch interface should now be available on port 5601 on the server. To access preconfigured 
-index patterns you have to switch to Global tenant.  
-The Keycloak IdP interface should now be available on port 12443 on the server.
+`ansible-playbook -i inventories soctools.yml -t stop`
+
+Web interfaces are available on the following ports:
+ * 9443 - NiFi
+ * 5601 - Kibana
+ * 6443 - Misp : Default user/password: admin@admin.test/test
+ * 9000 - The Hive : Default user/password: admin@thehive.local/secret
+ * 9001 - Cortex
+ * 12443 - Keycloak : Default user/password: admin/Pass005
 
 License
 -------

buildca.yml

 ---
 
 - name: Build certification authority
-  hosts: dsldev
+  hosts: soctoolsmain
   roles:
     - ca
 

buildimages.yml

 ---
 
 - name: Build docker images
-  hosts: dsldev
+  hosts: soctoolsmain
   roles:
     - build
 

group_vars/all/main.yml

 ---
 
-dslproxy: "dsoclab.gn4-3-wp8-soc.sunet.se"
+soctoolsproxy: "<CHANGE_ME:hostname>"
 
-# TheHive Button plugin
-THEHIVE_URL: "https://hive.gn4-3-wp8-soc.sunet.se/"
-# here enter API key for default admin user
-THEHIVE_API_KEY: "bs2Jc3tGJqhVv0AYyX2NYlhMlorPz7mX"
-# ID of the default admin user
-THEHIVE_OWNER: "admin@thehive.local"
-
-# TheHive Create Organisation and Users
-# Login as default admin user and create API key, populate it here
-# thehive_admin_api: "KoHrKbIJm8XMsJxA9nZLs6YemCu76o3u"
-# thehive_writer: "[write]"
-
-#THEHIVE_API_KEY: "1gFdNhmUSxO3BRe1SBB5JYEvkW9UOo6s"
-THEHIVE_USERS:
-  - kiril:
-    username: "kiril"
-    name: "Kiril"
-    surname: "Kiroski"
-    roles: '["read", "write", "admin"]'
-    organization: "uninett.no"
-  - temur:
-    username: "temur"
-    name: "Temur"
-    surname: "Maisuradze"
-    roles: '["read", "write", "admin"]'
-    organization: "uninett.no"
+maxmind_key: ""
 
+docker_build_dir: "{{playbook_dir}}/build"
 
+# TheHive Button plugin
+THEHIVE_URL: "https://hive.gn4-3-wp8-soc.sunet.se/"
+THEHIVE_API_KEY: "5LymseWiurZBrQN8Kqp8O+9KniTL5cE0"
+THEHIVE_OWNER: "admin"
 
 soctools_netname: "soctoolsnet"
 soctools_network: "172.22.0.0/16"
 
-repo: gn43-dsl
+repo: soctools
 version: 7
 suffix: a20201004
 
-haproxy_name: "dsoclab-haproxy"
+haproxy_name: "soctools-haproxy"
 haproxy_version: "2.2"
 haproxy_img: "{{repo}}/haproxy:{{version}}{{suffix}}"
 HAPROXY_PROCESSES: "2"
-HAPROXY_STATS_PASS: "eiph2Eepaizicheelah3tei+bae3ohgh"
+
+FILEBEAT_VERSION: "7.9.3"
+FILEBEAT_OUTPUT_HOST: "{{soctoolsproxy}}"
+FILEBEAT_OUTPUT_PORT: "6000"
+FILEBEAT_CERT: "/opt/filebeat/filebeat.crt"
+FILEBEAT_KEY: "/opt/filebeat/filebeat.key"
 
 temp_root: "/tmp/centosbuild"
 
 openjdk_img: "{{repo}}/openjdk:{{version}}{{suffix}}"
 
-zookeeper_name: "dsoclab-zookeeper"
+zookeeper_name: "soctools-zookeeper"
 zookeeper_img: "{{repo}}/zookeeper:{{version}}{{suffix}}"
 
-misp_name: "dsoclab-misp"
+misp_name: "soctools-misp"
 misp_img: "{{repo}}/misp:{{version}}{{suffix}}"
+misp_url: "https://{{soctoolsproxy}}:6443"
 
 nifi_img: "{{repo}}/nifi:{{version}}{{suffix}}"
 
-mysql_name: "dsoclab-mysql"
+mysql_name: "soctools-mysql"
 mysql_img: "{{repo}}/mysql:{{version}}{{suffix}}"
-mysql_dbrootpass: "Pass006"
 
-cassandra_name: "dsoclab-cassandra"
+cassandra_name: "soctools-cassandra"
 cassandra_img: "{{repo}}/cassandra:{{version}}{{suffix}}"
 
-thehive_name: "dsoclab-thehive"
+thehive_name: "soctools-thehive"
 thehive_img: "{{repo}}/thehive:{{version}}{{suffix}}"
-# GENERATED WITH cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 64 | head -n 1
-thehive_secret_key: "LcnI9eKLo33711BmCnzf6UM1y05pdmj3dlADL81PxuffWqhobRoiiGFftjNPKpmM"
 
-cortex_name: "dsoclab-cortex"
+cortex_name: "soctools-cortex"
 cortex_img: "{{repo}}/cortex:{{version}}{{suffix}}"
 cortex_elasticsearch_mem: "256m"
-# GENERATED WITH cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 64 | head -n 1
-cortex_secret_key: "9CZ844IcAp5dHjsgU4iuaEssdopLcS6opzhVP3Ys4t4eRpNlHmwZdtfveLEXpM9D"
-cortex_odfe_pass: "Pass009"
-
-kspass: "Testing003"
-tspass: "Testing003"
 
 sysctlconfig:
-  - { key: "net.core.rmem_max", val: "2097152" }
-  - { key: "net.core.wmem_max", val: "2097152" }
+  - { key: "net.core.rmem_max", val: "4194304" }
+  - { key: "net.core.wmem_max", val: "4194304" }
   - { key: "vm.max_map_count" , val:  "524288" }
 
 nifi_javamem: "1g"
 odfe_javamem: "512m"
 
-nifi_version: 1.11.4
+nifi_version: 1.12.1
 nifi_repo: "https://archive.apache.org/dist"
 
 ca_cn: "SOCTOOLS-CA"
 
-soctools_users:
-  - firstname: "Bozidar"
-    lastname: "Proevski"
-    username: "bozidar.proevski"
-    email: "bozidar.proevski@finki.ukim.mk"
-    DN: "CN=Bozidar Proevski"
-    CN: "Bozidar Proevski"
-    password: "Pass001"
-  - firstname: "Arne"
-    lastname: "Oslebo"
-    username: "arne.oslebo"
-    email: "arne.oslebo@uninett.no"
-    DN: "CN=Arne Oslebo"
-    CN: "Arne Oslebo"
-    password: "Pass002"
-  - firstname: "Kiril"
-    lastname: "Kjiroski"
-    username: "kiril.kjiroski"
-    email: "kiril.kjiroski@finki.ukim.mk"
-    DN: "CN=Kiril Kjiroski"
-    CN: "Kiril Kjiroski"
-    password: "Pass003"
-
 odfees_img: "{{repo}}/odfees:{{version}}{{suffix}}"
 odfekibana_img: "{{repo}}/odfekibana:{{version}}{{suffix}}"
-# GENERATE 32-bit secure value
-odfekibana_cookie: "iroAm0ueIV7w6CS1WcJTwIV6R4d5RIAt"
-odfees_adminpass: "Pass004"
 #elk_version: "oss-7.6.1"
 elk_version: "oss-7.4.2"
 #odfeplugin_version: "1.7.0.0"
@@ -129,16 +80,25 @@ openid_scope: profile
 openid_subjkey: preferred_username
 
 keycloak_img: "{{repo}}/keycloak:{{version}}{{suffix}}"
-keycloak_adminpass: "Pass005"
 
 elastic_username: "admin"
-misp_token: ""
-misp_url: ""
-maxmind_key: ""
 
 misp_dbname: "mispdb"
 misp_dbuser: "misp"
-misp_dbpass: "Pass007"
-# misp_salt generated with: openssl rand -base64 32
-misp_salt:   "wa2fJA2mGIn32IDl+uKrCJ069Mg3khDdGzFNv8DOwM0="
+
+services:
+  - mysql
+  - haproxy
+  - openjdk
+  - zookeeper
+  - nifi
+  - elasticsearch
+  - kibana
+  - odfees
+  - odfekibana
+  - keycloak
+  - misp
+  - cassandra
+  - thehive
+  - cortex
 

group_vars/all/users.yml

+---
+
+soctools_users:
+  - firstname: "User1"
+    lastname: "SOC"
+    username: "user1"
+    email: "user1@soctools.test"
+    DN: "CN=User1Soctools"
+    CN: "User1Soctools"
+  - firstname: "User2"
+    lastname: "SOC"
+    username: "user2"
+    email: "user2@soctools.test"
+    DN: "CN=User2Soctools"
+    CN: "User2Soctools"
+
+# Minimum one user is required
+ODFE_ADMIN_USERS:
+  - user1
+
+

inventories/build/group_vars/all.yml

----
-
-docker_image_path: images
-base_image: python:2.7-stretch

inventories/build/hosts.yml

-all:
-  hosts:
-    nifi-image:
-      ansible_connection: docker
-      ansible_python_interpreter: /usr/bin/python
-    localhost:
-      ansible_python_interpreter: /usr/bin/python
-      ansible_connection: local
-  children:
-    nifi:
-      hosts:
-        localhost:

inventories/cassandra

+[cassandra]
+soctools-cassandra ansible_connection=docker

inventories/cortex

+[cortex]
+soctools-cortex ansible_connection=docker

inventories/deploy/group_vars/haproxy.yml

----
-index: haproxy
-scale: "{{ haproxy_scale | default('1')}}"
-docker:
-  haproxy:
-    image: haproxy:latest
-    volumes:
-      - /usr/local/etc/haproxy/:/usr/local/etc/haproxy:ro
-    ports:
-      - "80:80"
-    source: pull
\ No newline at end of file

inventories/deploy/group_vars/nifi.yml

----
-index: nifi
-scale: "{{ nifi_scale | default('1')}}"
-docker:
-  nifi:
-#    image: nifi-soctools #For nifi image built by soctools
-#    source: load
-    image: apache/nifi:latest
-    source: pull
-    command: /opt/nifi/nifi-current/scripts/start.sh
-    env: 
-      NIFI_HOME: "/opt/nifi/nifi-current"
-      NIFI_LOG_DIR: "/opt/nifi/nifi-current/logs"
-      NIFI_PID_DIR: "/opt/nifi/nifi-current/run"
-      NIFI_CLUSTER_IS_NODE: "true"
-      NIFI_ZK_CONNECT_STRING: "zookeeper_1:2181"
-      NIFI_CLUSTER_NODE_PROTOCOL_PORT: "8082"
-      NIFI_ELECTION_MAX_WAIT: "1 min"
-    load_path: "{{ image_location }}/nifi-soctools.tar" 

inventories/deploy/group_vars/zookeeper.yml

----
-index: zookeeper
-scale: "{{ zookeeper_scale | default('1')}}"
-docker:
-  zookeeper:
-    image: zookeeper:latest
-    source: pull
\ No newline at end of file

inventories/deploy/hosts.yml.example

-all:
-  hosts:
-    host1:
-      ansible_ssh_user: debian
-      ansible_python_interpreter: /usr/bin/python
-      become: yes
-  children:
-    soctools_server:
-      hosts:
-        host1:
-    nifi:
-      hosts:
-        host1:
-          nifi_scale: 3
-    haproxy:
-      hosts:
-        host1:
-    zookeeper:
-      hosts:
-        host1:
-          zookeeper_scale: 3
\ No newline at end of file

inventories/elasticsearch

+[odfeescontainers]
+soctools-odfe-1 ansible_connection=docker
+soctools-odfe-2 ansible_connection=docker

inventories/filebeat

+[filebeat]
+soctools-nifi-1 ansible_connection=docker FILEBEAT_FILES='["/opt/nifi/nifi-current/logs/nifi-app.log","/opt/nifi/nifi-current/logs/nifi-bootstrap.log","/opt/nifi/nifi-current/logs/nifi-user.log","/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="nifi" FILEBEAT_LOG_FORMAT="text"
+soctools-nifi-2 ansible_connection=docker FILEBEAT_FILES='["/opt/nifi/nifi-current/logs/nifi-app.log","/opt/nifi/nifi-current/logs/nifi-bootstrap.log","/opt/nifi/nifi-current/logs/nifi-user.log","/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="nifi" FILEBEAT_LOG_FORMAT="text"
+soctools-nifi-3 ansible_connection=docker FILEBEAT_FILES='["/opt/nifi/nifi-current/logs/nifi-app.log","/opt/nifi/nifi-current/logs/nifi-bootstrap.log","/opt/nifi/nifi-current/logs/nifi-user.log","/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="nifi" FILEBEAT_LOG_FORMAT="text"
+soctools-misp ansible_connection=docker FILEBEAT_FILES='["/var/opt/rh/rh-php72/log/php-fpm/*.log","/var/opt/rh/rh-redis32/log/redis/redis.log","/var/log/httpd/*log","/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="misp" FILEBEAT_LOG_FORMAT="text"
+soctools-odfe-1 ansible_connection=docker FILEBEAT_FILES='["/usr/share/elasticsearch/logs/soctools-cluster_server.json"]' FILEBEAT_LOG_TYPE="elasticsearch" FILEBEAT_LOG_FORMAT="json"
+soctools-odfe-2 ansible_connection=docker FILEBEAT_FILES='["/usr/share/elasticsearch/logs/soctools-cluster_server.json"]' FILEBEAT_LOG_TYPE="elasticsearch" FILEBEAT_LOG_FORMAT="json"
+soctools-kibana ansible_connection=docker FILEBEAT_FILES='["/var/log/supervisor/kibana_stdout.log"]' FILEBEAT_LOG_TYPE="kibana" FILEBEAT_LOG_FORMAT="json"
+soctools-keycloak ansible_connection=docker FILEBEAT_FILES='["/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="keycloak" FILEBEAT_LOG_FORMAT="json"
+soctools-mysql ansible_connection=docker FILEBEAT_FILES='["/var/opt/rh/rh-mariadb103/log/mariadb/mariadb.log","/var/opt/rh/rh-mariadb103/lib/mysql/server_audit.log"]' FILEBEAT_LOG_TYPE="mysql" FILEBEAT_LOG_FORMAT="text"
+soctools-haproxy ansible_connection=docker FILEBEAT_SYSLOG_PORT=9000 FILEBEAT_LOG_TYPE="haproxy" FILEBEAT_LOG_FORMAT="text"
+soctools-zookeeper ansible_connection=docker FILEBEAT_FILES='["/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="zookeeper" FILEBEAT_LOG_FORMAT="text"
+soctools-cortex ansible_connection=docker FILEBEAT_FILES='["/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="cortex" FILEBEAT_LOG_FORMAT="text"
+soctools-thehive ansible_connection=docker FILEBEAT_FILES='["/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="thehive" FILEBEAT_LOG_FORMAT="text"
+soctools-cassandra ansible_connection=docker FILEBEAT_FILES='["/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="cassandra" FILEBEAT_LOG_FORMAT="text"
+

inventories/haproxy

+[haproxy]
+soctools-haproxy ansible_connection=docker
\ No newline at end of file

inventories/keycloak

+[keycloakcontainers]
+soctools-keycloak ansible_connection=docker

inventories/kibana

+[odfekibanacontainers]
+soctools-kibana ansible_connection=docker

inventories/misp

+[mispcontainers]
+soctools-misp ansible_connection=docker