diff --git a/ChangeLog b/ChangeLog index 6a30768208a55c77fd40fc4b571fa4c889171433..f2c6c3b0a02ea7a3a547809b34fa48ad800801da 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,29 @@ +======= +1.8.0 RELEASE (2023/05) +Improvements and enhancements + - Upgraded to newer Django + requires adaption of flowspy/settings.py regarding staticfiles handling + (check 'git log -p flowspy/settings.py.dist') + + - Fixed unwanted possibility to mix IPv4 and IPv6 address prefixes in rules + - Fixed broken check_sync for IPv6 rules + - Support for implicit icmp6 in IPv6 rules + - Fixed improper UTF-8 encoding of Shibboleth attribute values + - Rule id shown on rule detail pages + + - Installation support for DEBIAN/UBUNTU, including Systemd support + - Cleanup of log file handling + - Update of documentation + - Enhancement of the Dockerfiles for testing and reference-installation + - DEBIAN as base Docker image is now default: ./Dockerfile + (being linked to ./Dockerfile.fod.debian) + - CENTOS7 keeps to be supported: ./Dockerfile.fod.centos.new + + - New experimental setting variable RULE_NAME_PREFIX: + possibility to prefix every FlowSpec rule + injected via JUNOS_specific NETCONF on the router with a given String, + to distinguish such FlowSpec rules from + FlowSpec rules injected via NETCONF by other tools or means ======= 2022/08 @@ -35,42 +61,42 @@ Updates and Enhancements ======= 1.2 RELEASE Updates and enhancements. Check documentation for updating from previous versions - - Code Cleanup - - Branding - - Rest Api - - Easier Configuration + - Code Cleanup + - Branding + - Rest Api + - Easier Configuration ======= 1.1.1 RELEASE Minor changes release - - Change license to GPLv3 + - Change license to GPLv3 - - Minor documentation updates + - Minor documentation updates =========== 1.1.0 RELEASE Updates and enhancements. Check documentation for updating from previous versions - - Minor UI enhancements - - Patch user model to include user peer in __unicode__ - - Include celery task exceptions in proxy - - Cleanup js files - - Resolve a major issue where the platform would start on 2nd refresh - - Cleanup poller urls - - Enhancements to json routes export - - Include timeout handling in tasks - - Fix issue with ports not updating - - Add longerusername plugin to replace user monkey patching - - Handle a bug in South that causes the Peer table to become unusable after adding autoincrement + - Minor UI enhancements + - Patch user model to include user peer in __unicode__ + - Include celery task exceptions in proxy + - Cleanup js files + - Resolve a major issue where the platform would start on 2nd refresh + - Cleanup poller urls + - Enhancements to json routes export + - Include timeout handling in tasks + - Fix issue with ports not updating + - Add longerusername plugin to replace user monkey patching + - Handle a bug in South that causes the Peer table to become unusable after adding autoincrement =========== 1.0.3 RELEASE Minor fixes. Check documentation for peers table handling - - Fix issue with altlogin redirection - - Switch peers primary key to AutoField - - Make peers tables management configurable + - Fix issue with altlogin redirection + - Switch peers primary key to AutoField + - Make peers tables management configurable =========== 1.0.2 RELEASE @@ -81,214 +107,214 @@ Documentation enhancements Minor fixes Fixes: - -Overview login theme - -Add missing urls + -Overview login theme + -Add missing urls =========== 1.0.0 RELEASE Major UI redesign, Debian Wheezy version, Django 1.4 Improvements: - -New UI based on Bootstrap3 theme - -Minor fixes in long polling init - -Debian Wheezy - Django 1.4 ready + -New UI based on Bootstrap3 theme + -Minor fixes in long polling init + -Debian Wheezy - Django 1.4 ready =========== 0.9.9 RELEASE Major documentation improvements. Minor app enhancements. Minor bug fixes Improvements: - -Wrote documentation in rst format (http://flowspy.readthedocs.org/) - -Update initial data with fragment types - -Add current version in footer via context - -Add beanstalk client, as installation from package is fuzzy - -Comment the helptext line in patched user model (django 1.3 issues) + -Wrote documentation in rst format (http://flowspy.readthedocs.org/) + -Update initial data with fragment types + -Add current version in footer via context + -Add beanstalk client, as installation from package is fuzzy + -Comment the helptext line in patched user model (django 1.3 issues) =========== 0.9.8 RELEASE Minor functionality improvements (check Requirements) Requirements: - -south migration to include database changes if you are at - <=0.9.5 + -south migration to include database changes if you are at + <=0.9.5 Improvements: - -Added more port validation checks - -Added more IP address validation checks + -Added more port validation checks + -Added more IP address validation checks =========== 0.9.7 RELEASE Minor UI improvements (check Requirements) Requirements: - -south migration to include database changes if you are at - <=0.9.5 + -south migration to include database changes if you are at + <=0.9.5 UI Improvements: - -Added badges in rule status + -Added badges in rule status =========== 0.9.6 RELEASE New Feature and minor UI improvements (check Requirements) Requirements: - -south migration to include database changes + -south migration to include database changes Features: - -Added fragment type as an option in rule match statements + -Added fragment type as an option in rule match statements UI Improvements: - -Changed wording;from 'Suspend' to 'Deactivate' - -Increased the size of Console and Add Rule buttons. Made Add Rule button - stand out with different color. + -Changed wording;from 'Suspend' to 'Deactivate' + -Increased the size of Console and Add Rule buttons. Made Add Rule button + stand out with different color. =========== 0.9.5 RELEASE Fixes Fixes: - -Fixed issue with page logo - -Changed Shibboleth attributes from HTML to Shibboleth naming in error.html - -Minor change in the user activation procedure. Activation mail goes only to admins not TechCs + -Fixed issue with page logo + -Changed Shibboleth attributes from HTML to Shibboleth naming in error.html + -Minor change in the user activation procedure. Activation mail goes only to admins not TechCs =========== 0.9.4 RELEASE Minor fixes Fixes: - -Change the name of the released file (Makefile) - -Added copyright info plus updated the README file - -Added missing files in images + -Change the name of the released file (Makefile) + -Added copyright info plus updated the README file + -Added missing files in images =========== 0.9.3 RELEASE Minor fix Fixes: - -Fixed the population of "Any" in source field + -Fixed the population of "Any" in source field =========== 0.9.2 RELEASE Major enhancement and a minor fixes Enhancements: - -Added alternative view for helpdesk + -Added alternative view for helpdesk Fixes: - -Fixed the static url for tinymce in settings - -Fixed an issue caused by multiple Shibboleth attributes + -Fixed the static url for tinymce in settings + -Fixed an issue caused by multiple Shibboleth attributes =========== 0.9.1 RELEASE Major UI enhancements Enhancements: - -Added bootstrap UI framework - -Added TinyMCE in flatpages - -Brought back flatpages with JS magic for translation switching - -HomeOrganization is no longer required-user selects from dropdown - -Added Shibboleth mapping in settings - -Added an Any button in source address + -Added bootstrap UI framework + -Added TinyMCE in flatpages + -Brought back flatpages with JS magic for translation switching + -HomeOrganization is no longer required-user selects from dropdown + -Added Shibboleth mapping in settings + -Added an Any button in source address =========== 0.9 RELEASE Major enhancements Enhancements: - -Added internationalization support - -Added Greek translation + -Added internationalization support + -Added Greek translation =========== 0.8.7 RELEASE Minor enhancements Enhancements: - - Merged all mail txt files into one - - Added all routes in form cleaning (initially, EXPIRED, ADMININACTIVE and ERROR were excluded) + - Merged all mail txt files into one + - Added all routes in form cleaning (initially, EXPIRED, ADMININACTIVE and ERROR were excluded) =========== 0.8.6 RELEASE Minor UI enhancements/Bug fix Fixes: - - Fixed issue where rules in ERROR state would cause check_sync to check them + - Fixed issue where rules in ERROR state would cause check_sync to check them Enhancements: - - Added small dots to ongoing response field to indicate activity + - Added small dots to ongoing response field to indicate activity =========== 0.8.5 RELEASE Feature enhancement release/Minor UI fixes/Cleanup Fixes: - - Changed javascript order to prevent unformated content in datatables - - Un-needed files cleanup - - Error template is now based on base.html template + - Changed javascript order to prevent unformated content in datatables + - Un-needed files cleanup + - Error template is now based on base.html template Enhancements: - - Administrator privileges apply on UI as well - - Enhanced application security + - Administrator privileges apply on UI as well + - Enhanced application security =========== 0.8.4 RELEASE Vulnerability prevention/bug fixes release Fixes: - - Fixed a bug where the shib auth backend erased non-shibboleth users info - - Added an authsource variable to prevent authentication backend overlapping - - Added exception handling for non-Shibboleth users that do not belong to a peer + - Fixed a bug where the shib auth backend erased non-shibboleth users info + - Added an authsource variable to prevent authentication backend overlapping + - Added exception handling for non-Shibboleth users that do not belong to a peer =========== 0.8.3 RELEASE Feature enhancement release Fixes: - - User/username length monkey patching now works with admin forms as well + - User/username length monkey patching now works with admin forms as well =========== 0.8.2 RELEASE Bug Fix release Fixes: - - Fixed bug with csrf cookie not being set while logged in for the first time + - Fixed bug with csrf cookie not being set while logged in for the first time =========== 0.8.1 RELEASE This is the latest alpha release operating on production network Changes: - - Fixed bug with protected networks form cleaning + - Fixed bug with protected networks form cleaning =========== v0.8.0 RELEASE New features Changes: - - DB migration to protocol addition - - Added protocol to match conditions plus check mechanism to form cleaning + - DB migration to protocol addition + - Added protocol to match conditions plus check mechanism to form cleaning =========== 0.7.11 RELEASE Bux fixes Changes: - - Prevented a bug that would cause the rule application to throw exception + - Prevented a bug that would cause the rule application to throw exception =========== 0.7.10 RELEASE Got rid of another cronjob Changes: - - Turned expiration notification cron job into celery job - - Added a preliminary draft for a Makefile facilitating various jobs + - Turned expiration notification cron job into celery job + - Added a preliminary draft for a Makefile facilitating various jobs =========== 0.7.9.7 RELEASE Some minor changes mainly to reinforce security Changes: - - Added FQDN resolving in mail notification templates to denote the host that an action originated + - Added FQDN resolving in mail notification templates to denote the host that an action originated =========== 0.7.9.5 RELEASE Oops! Something was missing from form validation Changes: - - Added source address to required fields + - Added source address to required fields =========== 0.7.9.2 RELEASE Major changes (maybe version tag does not indicate that) Changes: - - Added a custom command to fetch networks for each peer. Got rid of cronjob - - Major change with db engine. Altered database storage engine to MYISAM to allow for software relations between tables and views + - Added a custom command to fetch networks for each peer. Got rid of cronjob + - Major change with db engine. Altered database storage engine to MYISAM to allow for software relations between tables and views =========== 0.7.9.1 RELEASE @@ -300,33 +326,33 @@ Changes: 0.7.9 RELEASE Bug fixes Changes: - - Added a custom uknown_host_cb function to overcome ssh key errors + - Added a custom uknown_host_cb function to overcome ssh key errors =========== 0.7.7 RELEASE Modules cleanup Changes: - - Removed utils/beanstalkc as it is now a deb package + - Removed utils/beanstalkc as it is now a deb package =========== 0.7.1 RELEASE Code improvements Changes: - - Modified peer network range update mechanism + - Modified peer network range update mechanism =========== 0.7 RELEASE Major release/changes Features: - - Added registration to installed apps - - Removed user activation from shibboleth backend. Moved it to login view + - Added registration to installed apps + - Removed user activation from shibboleth backend. Moved it to login view =========== Application features up to now: - - Rule creation and application to device via netconf, nxpy - - Match statements include source, destination addrs, src, dst ports - - Then statements include discard and rate limit for plain users - - User authentication via Shibboleth - - Whois client determines user peer networks and user authority + - Rule creation and application to device via netconf, nxpy + - Match statements include source, destination addrs, src, dst ports + - Then statements include discard and rate limit for plain users + - User authentication via Shibboleth + - Whois client determines user peer networks and user authority diff --git a/_version.py b/_version.py index ae02af8d4e1ea847aea2df79bec9b18e91ef4eec..9632522b2284b3aa25e54cdcd532700902ec81ae 100644 --- a/_version.py +++ b/_version.py @@ -1,5 +1,5 @@ #VERSION = '1.5' -VERSION = '1.7' +VERSION = '1.8.0' if __name__ == "__main__": print(VERSION) diff --git a/doc/administration_and_usage/basic_administration_and_usage-v1.8.md b/doc/administration_and_usage/basic_administration_and_usage-v1.8.md new file mode 100644 index 0000000000000000000000000000000000000000..0b84d7d2eb62615024d7f0da1957b75f199b0cc9 --- /dev/null +++ b/doc/administration_and_usage/basic_administration_and_usage-v1.8.md @@ -0,0 +1,180 @@ +# Basic Administration and Usage + +Early steps for administration and usage that can be done after FoD has been installed. + +## 1: Setup NETCONF + FoD admin user + +### 1 (local): Setup NETCONF + FoD admin user (FoD running locally) + +admin user password and NETCONF connection has to be setup, +either by + +A) via the setup page of FoD in container +(needs ENABLE_SETUP_VIEW=True in ./flowspy/settings.py; +only applicable once, i.e., as long as no admin user has been setup, for security reasons): + +http://SERVERNAME:SERVERPORT/setup/ + +(/setup not only covers NETCONF connectivity and initial admin user/password, +but also the adding of a single peer range, i.e., an IP address prefix, (via a single peer) +to be assigned to the initial admin user, +in order to allow the creation/changing FlowSpec rules with a destination IP address prefix +falling within that single peer range) + +or alternatively + +B) manually + +edit /srv/flowspy/flowspy/settings.py : settings NETCONF_DEVICE, NETCONF_PORT, NETCONF_USER, NETCONF_PASS + +make sure IP connectivity to NETCONF_DEVICE is available + +in FoD installation dir (default: /srv/flowspy): ./pythonenv ./manage.py createsuperuser ... +in FoD installation dir (default: /srv/flowspy): ./pythonenv ./manage.py changepassword ... + +restart FoD: 'systemctl restart fod-gunicorn; systemctl restart fod-celeryd' (if installed and running with Systemd support) + +or alternatively + +C) already having been set by install-\*.sh parameters (check [Debian/Ubuntu Installation](../installation/v1.7/debian_ubuntu.md) or [CENTOS 7 Installation](../installation/v1.7/centos.md) ) + +### 1 (Docker): Setup NETCONF + FoD admin user (FoD running in a Docker container) + +admin user password and NETCONF connection has to be setup, +either by + +A) via the setup page of FoD in container +(only applicable once, i.e., as long as no admin user has been setup, for security reasons): + +http://127.0.0.1:8001/setup/ + +(/setup not only covers NETCONF connectivity and initial admin user/password, +but also the adding of a single peer range, i.e., an IP address prefix, (via a single peer) +to be assigned to the initial admin user, +in order to allow the creation/change of FlowSpec rules with a destination IP address prefix +falling within that single peer range) + +or alternatively + +B) manually + +in by entering the running container and editing +docker exec -ti "$DOCKERID" bash # find out DOCKERID of running container with "docker ps" + +in docker: vi /srv/flowspy/flowspy/settings.py : settings NETCONF_DEVICE, NETCONF_PORT, NETCONF_USER, NETCONF_PASS + +make sure docker container has IP connectivity to NETCONF_DEVICE + +in docker: cd /srv/flowspy; ./pythonenv ./manage.py createsuperuser ... + +in docker: cd /srv/flowspy; ./pythonenv ./manage.py changepassword ... + +## 2: Accessing the FoD UI running in container after setup of admin user and password + +http(s)://SERVERNAME:SERVERPORT/altlogin +(for use with Docker: http(s)://SERVERNAME:8000/altlogin) + +(do not try to use the Shibboleth login, i.e., via /login, as it is not working without a set-up Shibboleth SP, see 2.1.2) + +### 2.1 administration + +via http(s)://SERVERNAME:SERVERPORT/admin (only accessible by admin users, e.g., the initial admin, see 1.) + +#### 2.1.1 administration of peers, peer ranges, and users (via /admin) + +Peer ranges and Peers: +(http(s)://SERVERNAME:SERVERPORT/admin/peers/peerrange/ and http(s)://SERVERNAME:SERVERPORT/admin/peers/peer/) + +The 'peer' is a concept in FoD to support multi-tenancy. +Each peer has assigned a set of allowed destination IP address prefixes ('peer ranges') +for which the peer is allowed to deploy BGP FlowSpec rules. +It typically corresponds to a customer organization of the network operator organization running FoD +to provided to users of the different customer organizations. + +For managing users (beyond the initial setup of the first admin user, under 1.) the /admin web UI interface can be used +as well, specifically http(s)://SERVERNAME:SERVERPORT/admin/auth/user/. +E.g., it allows adding/removing users, changing first/last name of a user, define whether it is an admin or not. + +A 'user' in FoD (including every admin user) has assigned a set of peers (typically, often only 1). +Only for destination IP address prefixes of assigned peers the user has the right to +deploy or change BGP FlowSpec rules. + +This restriction also applies for the initial admin user created initially (see 1.). +So before that admin user can deploy FlowSpec rules via FoD, a peer has to be created (maybe by this admin user). +and appropriate peer ranges have to assigned to the peer +and this peer has to assigned to the user. + +Simple example: a single peer with peer ranges '0.0.0.0/0' and '0::0/0' assigned to an admin user, +allows to add/edit BGP FlowSpec rules with arbitrary destination IP prefixes. +For production deployments this is not generally recommended, +in order to avoid severe mitigation mistakes by admin users. + +The basic setup via http(s)://SERVERNAME:SERVERPORT/setup (see 1.) +not only covers NETCONF connectivity and initial admin user, but also the adding of a single peer range for a single peer +to be assigned to the initial admin user. + +#### 2.1.2 administration of user and their peer mapping (via Shibboleth + enrollment working for users) + +TODO: User management via Shibboleth + +TODO: enrollment workflow for users + +TODO: auto update of enrolled users on login + +### 2.2 usage + +#### 2.2.1 usage via web UI + +When logged into FoD UI via +http(s)://SERVERNAME:SERVERPORT/altlogin +(or via https://SERVERNAME:SERVERPORT/login for use with Shibboleth enrolled/registered users, see 2.1.2) +with a FoD user account which has assigned 1 or more peers with appropriate peer ranges (see 2.1.1) +the normal usage can start. + +##### Rules list/table page + +Provides a list/table of of all rule for all peers of the user. +BGP FlowSpec rule can have either status inactive (stored only in FoD database), +or active (stored in FoD database + installed on the router via NETCONF) + +##### Rules dashboard + +Provides a history of rule changes for all peers of the user. + +##### Add New Rule + +Allows to add a new rule, i.e., one not yet stored in the FoD database, +to FoD database and transfer it via NETCONF to the router(s). + +##### Edit Existing Rule + +Reachable from rules list page or dashboard page for all existing (active or inactive) displayed rules +for all peers of the user. +An edited route is changed in the FoD data base as well as updated on the router, +i.e., will be in active status after the edit operation. + +##### Overview (for admin users) + +TODO + +##### Admin (only for admin users, see 2.1) + +For admin users only, allows to perform (Django) admin actions (see 2.1) + +##### User Profile + +overview of the own user account, showing first/last name etc. + +create REST API token (see [API v1.7](../api/api-v1.7.md)) + +#### 2.2.2 usage via REST API + +see [API v1.7](../api/api-v1.7.md) + +#### 3. Further/Regular Administration + +### FoD run-time status + +There is ./systemd/fod-status.sh, a generic script (not limited to Systemd) for determining the process status of FoD along with some further aspects, e.g., Database connection, NETCONF configuration and reachability + + diff --git a/doc/api/api-v1.8.md b/doc/api/api-v1.8.md new file mode 100644 index 0000000000000000000000000000000000000000..5775381ce1a724b421c42558acdd099636075daa --- /dev/null +++ b/doc/api/api-v1.8.md @@ -0,0 +1,318 @@ +# REST API v1.8 + +(same as v1.7) + +# Description + +Current version of FoD officially has a REST API. +The API needs authentication. Out of the box the supported authentication +type is Token Authentication. + +TO UPDATE + +## Generating Tokens + +A user can generate an API token using the FoD UI. Select "My Profile" from the +top right menu and on the "Api Token" section click "Generate One". + +## Accessing the API + +The API is available at `/api/`. One can see the available API endpoints for +each model by making a GET request there. An authentication token must be added +in the request: + +* Using `cURL`, add the `-H "Authorization: Token <your-token>"` +parameter +* Using Postman, under the "Headers" add a header with name +"Authorization" and value "Token <your-token>". + +# Endpoints + +REST API provides the following endpoints that will be described in more detail in the next sections: + +* `/api/matchprotocol/` +* `/api/fragmenttypes/` +* `/api/matchdscp/` not supported yet +* `/api/thenactions/` +* `/api/routes/` +* `/api/stats/routes/` + +# Usage Examples + +Some basic usage examples will be provided including available +actions. Examples will be provided in `cURL` form. + +An example will be provided for `ThenAction`. This example applies to most other +models (`FragmentType`, `MatchProtocol`, `MatchDscp`) except +`Route` which is more complex and will be treated separately. + +## ThenAction + +### GET + +#### All items + +URL: `/api/thenactions/` + +Example: +``` +curl -X GET https://fod.example.com/api/thenactions/ -H "Authorization: Token <your-token>" +# or on the FoD host locally: +curl -X GET http://localhost:8000/api/thenactions/ -H "Authorization: Token <your-token>" + +RESPONSE: +[ + "discard", + "rate-limit:10000k", + "rate-limit:1000k", + "rate-limit:100k" +] +``` + +## Route + +### GET + +#### All items + +URL: `/api/routes/` + +Example: +``` +curl -X GET https://fod.example.com/api/routes/ -H "Authorization: Token <your-token>" + +RESPONSE: +[ + { + "applier": "admin", + "comments": "test comment", + "destination": "1.0.0.4/32", + "destinationport": "124", + "dscp": [], + "expires": "2021-04-15", + "filed": "2021-03-17T13:39:04.244120Z", + "fragmenttype": [], + "icmpcode": null, + "id": 62, + "last_updated": "2021-03-17T13:39:04.244151Z", + "name": "test_66ML9G", + "packetlength": null, + "port": null, + "protocol": [ + "udp" + ], + "requesters_address": null, + "response": null, + "source": "0.0.0.0/0", + "sourceport": "123", + "status": "PENDING", + "tcpflag": null, + "then": [ + "discard" + ] + } + ... +] +``` + +#### A specific item + +One can also GET a specific `Route`, by using the `id` in the GET url + +URL: `/api/routes/<route-id>/` + +Example: +``` +curl -X GET https://fod.example.com/api/routes/1/ -H "Authorization: Token <your-token>" + +RESPONSE: +{ + "applier": "admin", + "comments": "test comment", + "destination": "1.0.0.4/32", + "destinationport": "124", + "dscp": [], + "expires": "2021-04-15", + "filed": "2021-03-17T13:39:04.244120Z", + "fragmenttype": [], + "icmpcode": null, + "id": 62, + "last_updated": "2021-03-17T13:39:04.244151Z", + "name": "test_66ML9G", + "packetlength": null, + "port": null, + "protocol": [ + "udp" + ], + "requesters_address": null, + "response": null, + "source": "0.0.0.0/0", + "sourceport": "123", + "status": "PENDING", + "tcpflag": null, + "then": [ + "discard" + ] +} +``` + +### POST + +Starting from FoD v1.7 the REST API accepts parameters for POST, PUT and PATCH only via JSON documents, +old method used in REST API of FoD v1.3 to specify parameters as single form values is not supported any more. + +Required fields (to be specified via JSON document): + +* `name`: a name for the route +* `source`: a source subnet in CIDR formation +* `destination`: a destination subnet in CIDR formation +* `comments`: a small comment on what this route is about + +The response will contain all the additional fields + +URL: `/api/routes/` + +Example input data file `newroute.json`: + +``` +{ + "name": "test1", + "comments": "test1 comment", + "source": "0.0.0.0/0", + "destination": "1.0.0.4/32", + "protocol": [ + "udp", "tcp" + ], + "sourceport": "123,126,250-270", + "destinationport": "27,124,300-400,500-600", + "then": ["discard"], + "expires": "2022-10-20" +} +``` + +``` +# protocol: "icmp" | "udp" | "tcp" # actually choices returned by 'curl -X GET https://fod.example.com/api/matchprotocol/ ...) +# fragmenttype: "is-fragment" | "dont-fragment" | "first-fragment" | "last-fragment" | "not-a-fragment" # actually choices returned by 'curl -X GET https://fod.example.com/api/fragmenttype/ ...) +# dscp: not supported yet +# packetlength: not supported yet +# tcpflag: not supported yet +# icmptype: not supported yet +# icmpcode: not supported yet +# port, sourceport, destinationport +# then: "discard" | "rate-limit:10000k" | "rate-limit:1000k" | "rate-limit:100k" # actually choices returned by 'curl -X GET https://fod.example.com/api/thenactions/ ...) +``` + +``` +curl -X POST https://fod.example.com/api/routes/ -H "Content-Type: application/json" -d@newroute.json -H "Authorization: Token <your-token>" + +{ + "name":"testroute_9Q5Y90", + "id":5, + "comments":"Route for testing", + "applier":"admin", + "source":"62.217.45.75/32", + "sourceport":[], + "destination":"62.217.45.94/32", + "destinationport":[], + "port":[], + "dscp":[], + "fragmenttype":[], + "icmpcode":null, + "packetlength":null, + "protocol":[], + "tcpflag":null, + "then":[ + "https://fod.example.com/api/thenactions/4/" + ], + "filed":"2017-03-29T14:21:03.261Z", + "last_updated":"2017-03-29T14:21:03.261Z", + "status":"PENDING", + "expires":"2017-04-05", + "response":null, + "requesters_address":null +} +``` + +Notice that the `Route` has a `PENDING` status. This happens because the `Route` +is applied asynchronously to the Flowspec device (the API does not wait for the +operation). After a while the `Route` application will be finished and the +`status` field will contain the updated status (`ACTIVE`, `ERROR` etc). +You can check this `Route`s status by issuing a `GET` request with the `id` +the API returned. + + + +### PUT, PATCH + +Starting from FoD v1.7 the REST API accepts parameters for POST, PUT and PATCH only via JSON documents, +old method used in REST API of FoD v1.3 to specify parameters as single form values is not supported any more. + +`Route` objects can be modified using the `PUT` / `PATCH` HTTP methods. + +``` +# e.g., for rule id '10100' +curl -X PUT https://fod.example.com/api/routes/10100/ -d@data.json -H "Authorization: Token <your-token>" + +curl -X PATCH https://fod.example.com/api/routes/10100/ -d@data.json -H "Authorization: Token <your-token>" +``` + +When using `PUT` all fields need to be specified (compare `POST` section). +However, when using `PATCH` one can specify single fields, too. +This is especially useful for changing the `status` of an `INACTIVE` `Route` to `ACTIVE` +or vice versa. + +A PATCH or PUT of an active rule towards status INACTIVE will trigger removal of that active +rule from the configured NETCONF-linked router. +A PATCH or PUT of an inactive rule towards status ACTIVE will trigger commiting of that inactive +rule to the configured NETCONF-linked router. +A PATCH or PUT of an inactive rule with no status change will not trigger any change towards +the configured NETCONF-linked router. + +The result of PUT and PATCH are different in the following case: +A PATCH without change of status value and change only of non-FlowSpec-specific +match and action fields will not trigger a recommiting of even an active FlowSpec rule +on the configured NETCONF-linked router. +So, e.g., the comments or the rule expiration period can be changed without interrupting +and active route. + +In contrast, a PUT of an active rule will always trigger a recommiting of that route +on the configured NETCONF-linked router, independent of the attribute values changed. +So, this is useful +for ensuring that a rule is really still actively installed on the configured NETCONF-linked router +with all match and action parameters as is was last deployed by FoD, +and not changed by some other means without FoD's notice, e.g., other tools or directly by the CLI of the router. + +### DELETE + +``` +# e.g., for rule id '10100' +curl -X DELETE https://fod.example.com/api/routes/10100/ -H "Authorization: Token <your-token>" + +``` + +A DELETE method applied to a Flowspec rule has now always the semantics of fully removing the rule +also from data base (and so its history), not only deactivating it via NETCONF on the routers. +Furthermore, the application of this method can be restricted depending on the settings. +3 setting variables (ALLOW_DELETE_FULL_FOR_ADMIN, ALLOW_DELETE_FULL_FOR_USER_ALL, ALLOW_DELETE_FULL_FOR_USER_LIST) +control whether admin users, normal users in general, or a specificly selected +set of normal users is allowed to use DELETE method. +The default values of the settings (flowspy/settings.py.dist) by this means +allow DELETE only for admins, only for a specificly defined set of normal users. + +### General notes on `Route` models (differences to REST API of FoD v1.7): + +* Starting from FoD v1.7 the REST API accepts parameters for POST, PUT and PATCH only via JSON documents, +old method used in REST API of FoD v1.3 to specify parameters as single form values is not supported any more. + +* In contrast to REST API of FoD v1.3, REST API of current version v1.7 will honor the status +value set in POST method calls. +So, it is possible to create a Flowspec rule with status INACTIVE, +i.e., a rule which will not be automatically commited via NETCONF as result of the POST method call. + +* In contrast to REST API of FoD v1.3, REST API of current version v1.7, +the reuslt of PUT and PATCH method differs in some occasions (see above). + +* In contrast to REST API of FoD v1.3, REST API of current version v1.7 +will behave differently regarding the DELETE method (see above). + + diff --git a/doc/configuration/configuration-v1.8.md b/doc/configuration/configuration-v1.8.md new file mode 100644 index 0000000000000000000000000000000000000000..230015502f246d74aa319bd8201c5ff5c225f310 --- /dev/null +++ b/doc/configuration/configuration-v1.8.md @@ -0,0 +1,342 @@ +# Configuration v1.8 + +Time to configure flowspy. + +First of all you have to copy the dist files to their proper position: + + # cd /srv/flowspy/flowspy + # cp settings.py.dist settings.py + # cp urls.py.dist urls.py + +Then, you have to edit the settings.py file to correspond to your needs. The settings one has to focus on are: + +## Settings.py +Its time to configure `settings.py` in order to connect flowspy with a database, a network device and a broker. + +So lets edit settings.py file. + +It is strongly advised that you do not change the following to False +values unless, you want to integrate FoD with you CRM or members +database. This implies that you are able/have the rights to create +database views between the two databases: + + PEER_MANAGED_TABLE = True + PEER_RANGE_MANAGED_TABLE = True + PEER_TECHC_MANAGED_TABLE = True + +By doing that the corresponding tables as defined in peers/models will +not be created. As noted above, you have to create the views that the +tables will rely on. + +### Administrators + + ADMINS: set your admin name and email (assuming that your server can send notifications) + +### Secret Key +Please put a random string in `SECRET_KEY` setting. +Make this *unique*, and don't share it with anybody. It's the unique identifier of this instance of the application. + +### Allowed hosts +A list of strings representing the host/domain names that this Django site can serve. This is a security measure to prevent an attacker from poisoning caches and password reset emails with links to malicious hosts by submitting requests with a fake HTTP Host header, which is possible even under many seemingly-safe webserver configurations. + +For example: + + ALLOWED_HOSTS = ['*'] + +### Protected subnets +Subnets for which source or destination address will prevent rule creation and notify the `NOTIFY_ADMIN_MAILS`. + + PROTECTED_SUBNETS = ['10.10.0.0/16'] + +### Minimum Prefix Length for submitted FlowSpec rules + +IPv4 rules: + + PREFIX_LENGTH = 29 + +IPv6 rules: + + PREFIX_LENGTH_IPV6 = 64 - (32 - 29) + +### Database +`DATABASES` should contain the database credentials: + + DATABASES = { + 'default': { + 'ENGINE': 'django.db.backends.mysql', + 'NAME': 'flowspy', + 'USER': '<db user>', + 'PASSWORD': '<db password>', + 'HOST': '<db host>', + 'PORT': '', + } + } + +### Localization +By default Flowspy has translations for English and Greek. In case you want to add +another language, or remove one of the existing, you can change the `LANGUAGES` +variable and follow [django's localization documentation](https://docs.djangoproject.com/en/1.4/topics/i18n/translation/#localization-how-to-create-language-files) + +You might want to change `TIME_ZONE` setting too. Here is a [list](http://en.wikipedia.org/wiki/List_of_tz_database_time_zones) + + +### Cache +Flowspy uses cache in order to be fast. We recomend the usage of memcached, but +any cache backend supported by django should work fine. + + CACHES = { + 'default': { + 'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache', + 'LOCATION': '127.0.0.1:11211', + } + } + +### Network device access for rule changes (NETCONF) +We have to inform django about the device we set up earlier. + + NETCONF_DEVICE = "device.example.com" + NETCONF_USER = "<netconf user>" + NETCONF_PASS = "<netconf password>" + NETCONF_PORT = 830 + +### Network device access for mitigation counters (SNMP) +We have to inform django about the device we set up earlier. + + SNMP_COMMUNITY = "abcd" + + SNMP_IP = [ + {"ip": "192.168.0.1", "port": 1000}, + {"ip": "192.168.0.2", "port": 1001, "community": "abcdef"}, + {"ip": "192.168.0.3", "port": 1002}, + {"ip": "192.168.0.4", "port": 1002} +] + + SNMP_CNTBYTES = "1.3.6.1.4.1.2636.3.5.2.1.5" # OID of bytes counter (currently unused) + SNMP_CNTPACKETS = "1.3.6.1.4.1.2636.3.5.2.1.4" # OID of packet counter + SNMP_RULESFILTER = ["__flowspec_default_inet__", "__flowspec_IAS_inet__", "__flowspec_default_inet6__", "__flowspec_IAS_inet6__"] # get only statistics of specified tables$ + SNMP_POLL_INTERVAL = 8 #seconds # load new data into cache if it is older that a specified number of seconds + SNMP_TEMP_FILE = "/srv/flowspy/snmp_temp_data" + SNMP_POLL_LOCK = "/var/run/fod/snmppoll.lock" + SNMP_MAX_SAMPLECOUNT = 2016 # one month + SNMP_REMOVE_RULES_AFTER = 604800 # one month + +### statistics calc based on the SNMP counters + + GRAPHS_API_URL = 'http://127.0.0.1:8080/api/routes/' + + STATISTICS_PER_RULE = True + STATISTICS_PER_RULE__ADD_INITIAL_ZERO = True + +### redis +Redis connection configuration (as a broker for celery) + + CELERY_BROKER_URL = "redis://localhost//" + POLLS_TUBE = 'polls' + BROKER_VHOST = "/" + CELERY_CONCURRENCY = 1 + +### Notifications +Outgoing mail address and prefix. + + DISABLE_EMAIL_NOTIFICATION = False # only disable for testing + + SERVER_EMAIL = "Example FoD Service <noreply@example.com>" + EMAIL_SUBJECT_PREFIX = "[FoD] " + NOTIFY_ADMIN_MAILS = ["admin@example.com"] + + +If you have not installed an outgoing mail server you can always use your own account (either corporate or gmail, hotmail ,etc) by adding the +following lines in settings.py: + + EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend" + EMAIL_USE_TLS = True #(or False) + EMAIL_HOST = 'smtp.example.com' + EMAIL_HOST_USER = 'username' + EMAIL_HOST_PASSWORD = 'yourpassword' + EMAIL_PORT = 587 #(outgoing) + +### Whois servers +Add two whois servers in order to be able to get all the subnets for an AS. + + PRIMARY_WHOIS = 'whois.example.com' + ALTERNATE_WHOIS = 'whois.example.net' + +### Branding +Fill your company's information in order to show it in flowspy. + + BRANDING = { + 'name': 'Example.com', + 'url': 'https://example.com', + 'footer_iframe': 'https://example.com/iframes/footer/', + 'facebook': '//facebook.com/example.com', + 'twitter': '//twitter.com/examplecom', + 'phone': '800-12-12345', + 'email': 'helpdesk@example.com', + 'logo': 'logo.png', + 'favicon': 'favicon.ico', + } + + +### Shibboleth +Flowspy supports shibboleth authentication. + + SHIB_ADMIN_DOMAIN = 'example.com' + SHIB_LOGOUT_URL = 'https://example.com/Shibboleth.sso/Logout' + + SHIB_AUTH_ENTITLEMENT = 'urn:mace' # can also be set to '', if no filtering of users by entitlement is not needed + + SHIB_SLUGIFY_USERNAME = False + +attribute configuration: + + SHIB_USERNAME = ['HTTP_EPPN'] # essential attribute, needed for identification of the user + SHIB_USERNAME_DISPLAY_NAME = "eduPersonPrincipalName" # only needed for displaying the error message of missing attribute + SHIB_USERNAME_DISPLAY_ADDINFO = "urn:mace:dir:attribute-def:eduPersonPrincipalName or urn:oid:1.3.6.1.4.1.5923.1.1.1.6" # only needed for displaying the error message of missing attribute + + SHIB_MAIL = ['mail', 'HTTP_MAIL', 'HTTP_SHIB_INETORGPERSON_MAIL'] # essential attribute, because mail is needed for registration and notifications + SHIB_MAIL_DISPLAY_NAME = "MAIL" # only needed for displaying the error message of missing attribute + SHIB_MAIL_DISPLAY_ADDINFO = "urn:mace:dir:attribute-def:mail or urn:oid:0.9.2342.19200300.100.1.3 or SHIB_INETORGPERSON_MAIL" # only needed for displaying the error message of missing attribute + + SHIB_FIRSTNAME = ['HTTP_SHIB_INETORGPERSON_GIVENNAME'] # not essential attribute + SHIB_FIRSTNAME_DISPLAY_NAME = "GIVENNAME" # only needed for displaying the error message of missing attribute + SHIB_FIRSTNAME_DISPLAY_ADDINFO = "urn:mace:dir:attribute-def:givenName or urn:oid:2.5.4.42" # only needed for displaying the error message of missing attribute + + SHIB_LASTNAME = ['HTTP_SHIB_PERSON_SURNAME'] # not essential attribute + SHIB_LASTNAME_DISPLAY_NAME = "SURNAME" # only needed for displaying the error message of missing attribute + SHIB_LASTNAME_DISPLAY_ADDINFO = "urn:mace:dir:attribute-def:sn or urn:oid:2.5.4.4" # only needed for displaying the error message of missing attribute + + SHIB_ENTITLEMENT = ['HTTP_SHIB_EP_ENTITLEMENT'] # not essential, if SHIB_AUTH_ENTITLEMENT='', otherwise essential + SHIB_ENTITLEMENT_DISPLAY_NAME = "eduPersonEntitlement" # only needed for displaying the error message of missing attribute + SHIB_ENTITLEMENT_DISPLAY_ADDINFO = "urn:oid:1.3.6.1.4.1.5923.1.1.1.7; the value of this attribute also has to include 'urn:mace:example.com:pki:user'" # only needed for displaying the error message of missing attribute + +### Various Settings + + TIME_ZONE = 'UTC' + LANGUAGE_CODE = 'en' + LANGUAGES = ( + ('el', _('Greek')), + ('en', _('English')), + ) + + +### Syncing the database +To create all the tables and fill with basic data needed by FoD we have to run the following commands: + + cd /srv/flowspy + ./manage.py migrate + ./manage.py initial_data + +## Create a superuser +A superuser can be added by using the following command from `/srv/flowspy/`: + + ./manage.py createsuperuser + + +## Propagate the flatpages +Inside the initial\_data/fixtures\_manual.xml file we have placed 4 +flatpages (2 for Greek, 2 for English) with Information and Terms of +Service about the service. To import the flatpages, run from root +folder: + + python manage.py loaddata initial_data/fixtures_manual.xml + +### Celery +Celery is a distributed task queue, which helps FoD run some async tasks, like applying a flowspec rule to a router. + +`Note` In order to check if celery runs or even debug it, you can run: + + ./manage.py celeryd --loglevel=debug + + +### Testing/Debugging +In order to see what went wrong you can check the following things. + +#### Django +You can start the server manually by running: + + ./manage.py runserver 127.0.0.1:8081 + +By doing so, you can serve your application like gunicord does just to test that its starting properly. This command should not be used in production! + +Of course you have to stop gunicorn and make sure that port 8081 is free. + +#### Gunicorn +Just curl from the server http://localhost:8081 + +#### Celery +In order to check if celery is working properly one can start celery by typing: + + ./manage.py celeryd --loglevel=debug + +Again this is for debug purposes. + + +#### Connectivity to flowspec device +Just try to connect with the credentials you entered in settings.py from the host that will be serving flowspy. + + +#### General Test +Log in to the admin interface via https://<hostname>/admin. Go to Peer ranges and add a new range (part of/or a complete subnet), eg. 10.20.0.0/19 Go to Peers and add a new peer, eg. id: 1, name: Test, AS: 16503, tag: TEST and move the network you have created from Available to Chosen. From the admin front, go to User, and edit your user. From the bottom of the page, select the TEST peer and save. Last but not least, modify as required the existing (example.com) Site instance (admin home->Sites). You are done. As you are logged-in via the admin, there is no need to go through Shibboleth at this time. Go to https://<hostname>/ and create a new rule. Your rule should be applied on the flowspec capable device after aprox. 10 seconds. + +## Footer +Under the templates folder (templates), you can alter the footer.html +file to include your own footer messages, badges, etc. + +## Welcome Page +Under the templates folder (templates), you can alter the welcome page - +welcome.html with your own images, carousel, videos, etc. + +## Usage + +### Web interface +FoD comes with a web interface, in which one can edit and apply new routes. + +### Rest Api +FoD provides a rest api. It uses token as authentication method. + +### Generating Tokens +A user can generate a token for his account on "my profile" page from FoD's +UI. Then by using this token in the header of the request he can list, retrieve, +modify and create rules. + +### Example Usage +Here are some examples: + +#### GET items +- List all the rules your user has created (admin users can see all the rules) + + curl -X GET https://fod.example.com/api/routes/ -H 'Authorization: Token <Your users token>' + +- Retrieve a specific rule: + + curl -X GET https://fod.example.com/api/routes/<rule_id>/ -H 'Authorization: Token <Your users token>' + +- In order to create or modify a rule you have to use POST/PUT methods. + +#### POST/PUT rules +In order to update or create rules you can follow this example: + +##### Foreign Keys +In order to create/modify a rule you have to connect the rule with some foreign keys: + +###### Ports, Fragmentypes, protocols, thenactions +When creating a rule, one can specify: + +- source port +- destination port +- port (if source = destination) + +That can be done by getting the url of the desired port instance from `/api/ports/<port_id>/` + +Same with Fragmentypes in `/api/fragmenttypes/<fragmenttype_id>/`, protocols in `/api/matchprotocol/<protocol_id>/` and then actions in `/api/thenactions/<action_id>/`. + +Since we have the urls we want to connect with the rule we want to create, we can make a POST request like the following: + + + curl -X POST -H 'Authorization: Token <Your users token>' -F "name=Example" -F "comments=Description" -F "source=0.0.0.0/0" -F "sourceport=https://fod.example.com/api/ports/7/" -F "destination=203.0.113.12" https://fod.example.com/api/routes/ + +And here is a PUT request example: + + curl -X PUT -F "name=Example" -F "comments=Description" -F "source=0.0.0.0/0" -F "sourceport=https://fod.example.com/api/ports/7/" -F "destination=83.212.9.93" https://fod.example.com/api/routes/12/ -H 'Authorization: Token <Your users token>' + + diff --git a/doc/installation/v1.8/centos.md b/doc/installation/v1.8/centos.md new file mode 100644 index 0000000000000000000000000000000000000000..175626a94c88cb8cb86d6c49bbb3e72e96de5fec --- /dev/null +++ b/doc/installation/v1.8/centos.md @@ -0,0 +1,52 @@ +# Installation on CENTOS + +For the installation on CENTOS, +there is ./install-centos.sh, currently mainly used for installation inside a Docker container. +Currently, it understands several command line switches: + +- --fod_dir DIRNAME : specify FoD run-time directory, where FoD files are to be copied and to be run from +- --venv_dir DIRNAME : specify where Python virtualenv used by FoD will be located +- --base-dir DIRNAME : specify base directory where FoD run-time directory and virtualenv directory will be sub directories +- --here : do not copy FoD files into a separate FoD run-time directory, but instead install and make to run from current directory (probably not so useful for production) + +- --systemd : enable Systemd support explicitly, by default will be enabled if Systemd is detected +- --no_systemd : disable Systemd support explicitly, even if Systemd is detected +- --systemd_only_install : when enabling Systemd support, but do not try to contact a running Systemd to reload, only install Systemd service files (useful during Docker build) +- --systemd_check_start : when enabling Systemd support, do try to contact a running Systemd to reload and show status (default) + +- --supversiord : enable Supversiord instead of Systemd, inside Docker Supversiord is now enabled by default in case no Systemd running s detected +- --no_supversiord : disable Supversiord explicitly + +- --basesw : only install FoD dependencies (OS packages + Python Pip packages in virtualenv directory), do not install and setup FoD run-time directory (combines --basesw_os and --basesw_python) +- --basesw_os : only install FoD OS dependencies +- --basesw_python : only install FoD Python/Pip dependencies in virtualenv directory +- --fodproper : skip installing FoD dependencies, only install and setup FoD run-time directory + +- --setup_admin_user : setup default admin +- --setup_admin_user5 : setup admin specified by user name, password, email address, peer name, first IP prefix + +- --netconf : setup NETCONF specified by device address, TCP port, user name, password +- --exabgp : setup exabgp specified by BGP nodeid, IP address, AS number, peer IP address, peer AS number (only for the new FoD version, currently available in git branch "feature/exabgp_support2") + +## Systemd support / Starting FoD + +FoD installation currently supports Systemd, i.e., it installs Systemd startup files (./systemd/fod-\*.service) when a running Systemd is detected. Otherwise, in case of running inside docker Supversiord is enabled. +Also, Supversiord can be enabled instead of Systemd explicitly (check options above) + +When not running under systemd, +alternatives for startup of FoD +are either +./runfod-fg.sh for directly starting FoD processes together with Redis task broker +or +./runfod-supervisord.sh for starting FoD processes and Redis task broker via Supervisord +or +the overall wrapper script ./runfod.sh which uses either of the startup method depending +on the choice selected by options for install-centos.sh. + +The Systemd support also includes an experimental e-mail notification for startup-failures of the FoD processes + +## FoD run-time status + +There is ./systemd/fod-status.sh, a generic script (not limited to Systemd) for determining the process status of FoD along with some further aspects, e.g., Database connection, NETCONF configuration and reachability + + diff --git a/doc/installation/v1.8/debian_ubuntu.md b/doc/installation/v1.8/debian_ubuntu.md new file mode 100644 index 0000000000000000000000000000000000000000..789ec113e0b4278af2506f00ce1c63b31c4a3603 --- /dev/null +++ b/doc/installation/v1.8/debian_ubuntu.md @@ -0,0 +1,60 @@ +# Installation on DEBIAN/UBUNTU + +For the installation on recent DEBIAN or UBUNTU, e.g., UBUNTU focal (20.04), +there is ./install-debian.sh . + +It understands several command line switches: + +- --fod_dir DIRNAME : specify FoD run-time directory, where FoD files are to be copied and to be run from +- --venv_dir DIRNAME : specify where Python virtualenv used by FoD will be located +- --base-dir DIRNAME : specify base directory where FoD run-time directory and virtualenv directory will be sub directories +- --here : do not copy FoD files into a separate FoD run-time directory, but instead install and make to run from current directory (probably not so useful for production) + +- --systemd : enable Systemd support explicitly, by default will be enabled if Systemd is detected +- --no_systemd : disable Systemd support explicitly, even if Systemd is detected +- --systemd_only_install : when enabling Systemd support, but do not try to contact a running Systemd to reload, only install Systemd service files (useful during Docker build) +- --systemd_check_start : when enabling Systemd support, do try to contact a running Systemd to reload and show status (default) + +- --supversiord : enable Supversiord instead of Systemd, inside Docker Supversiord is now enabled by default in case no Systemd running s detected +- --no_supversiord : disable Supversiord explicitly + +- --basesw : only install FoD dependencies (OS packages + Python Pip packages in virtualenv directory), do not install and setup FoD run-time directory (combines --basesw_os and --basesw_python) +- --basesw_os : only install FoD OS dependencies +- --basesw_python : only install FoD Python/Pip dependencies in virtualenv directory +- --fodproper : skip installing FoD dependencies, only install and setup FoD run-time directory + +- --setup_admin_user : setup default admin +- --setup_admin_user5 : setup admin specified by user name, password, email address, peer name, first IP prefix + +- --netconf : setup NETCONF specified by device address, TCP port, user name, password +- --exabgp : setup exabgp specified by BGP nodeid, IP address, AS number, peer IP address, peer AS number (only for the new FoD version, currently available in git branch "feature/exabgp_support2") + +- --with-mta-postfix : if no MTA is installed, will try to install postfix (FoD operation is dependant on e-mail sending being available) + +- --with-db-sqlite : use a SQLite DB for FoD DB (not recommended for production, but easy for testing) +- --with-db-mysql : will try to install MySQL and to setup FoD database in MySQL +- --with-db-mariadb : will try to install MariaDB and to setup FoD database in MariaDB +- --with-db-mysqllike : assumes that MySQL or MariaDB is installed and will try to setup FoD database based on this + +## Systemd support / Starting FoD + +FoD installation currently supports Systemd, i.e., it installs Systemd startup files (./systemd/fod-\*.service) when a running Systemd is detected. Otherwise, in case of running inside docker Supversiord is enabled. +Also, Supversiord can be enabled instead of Systemd explicitly (check options above) + +When not running under systemd, +alternatives for startup of FoD +are either +./runfod-fg.sh for directly starting FoD processes together with Redis task broker +or +./runfod-supervisord.sh for starting FoD processes and Redis task broker via Supervisord +or +the overall wrapper script ./runfod.sh which uses either of the startup method depending +on the choice selected by options for install-debian.sh. + +The Systemd support also includes an experimental e-mail notification for startup-failures of the FoD processes + +## FoD run-time status + +There is ./systemd/fod-status.sh, a generic script (not limited to Systemd) for determining the process status of FoD along with some further aspects, e.g., Database connection, NETCONF configuration and reachability + + diff --git a/doc/installation/v1.8/docker.md b/doc/installation/v1.8/docker.md new file mode 100644 index 0000000000000000000000000000000000000000..4f7bdfc5de5b58338d2a57bd86688b8a86ab47ce --- /dev/null +++ b/doc/installation/v1.8/docker.md @@ -0,0 +1,183 @@ +# Installing Flowspy v1.8 with Docker + +Flowspy users may wish to test Flowspy in a container rather than directly onto a server or virtual host; this document shows how to build and deploy Flowspy in a Docker container. + +Flowspy installs into `/srv/flowspy` by default; if you wish to use a different directory then you will need to update several configuration files. It is also assumed that the `root` user will perform every action. + +# Requirements + +Docker must be installed on your target OS; the installation of Docker is outside the scope of this document. + +Docker images require disk space on the host OS; approximately 2 GB for Flowspy and approximately 200 MB for CentOS 7 which is the default OS used. + +Although the default `Dockerfile` = `Dockerfile.fod.debian` is for Debian, `Dockerfile.fod.centos.new` for CENTOS 7, and `Dockerfile.fod.centos.old` left for compatibility, +several older Dockerfiles for both Debian and CentOS and can be found in the `Dockerfiles.d` directory. These are meant **for testing only** and are _not_ production-ready. + +The containers run `gunicorn`, `celeryd` and Redis - and use a SQLite database (`/srv/flowspy/example-data`) to hold resources. They also make a web server available on port 8000. + +# Building the Flowspy container + +First build your Flowspy container. The default (`Dockerfile` = `Dockerfile.fod.debian`) is a container which uses Debian and `supervisord`, but there are other options; `Dockerfile.fod.centos.\*` uses CENTOS 7, `Dockerfile.fod.centos.new` with `supervisord`, `Dockerfile.fod.centos.old` is left for compatibility without `supervisord`. +`Dockerfile.fod.debian` and `Dockerfile.fod.centos.new` by default will split the installation of FoD during container build into 3 phases (OS dependencies, python/pip dependencies, FoD proper installation) to exploit Docker build cache for faster rebuilding. +Moreover, `Dockerfile.fod.debian` and `Dockerfile.fod.centos.new` can be edited for uncommenting/commenting to switch more options: e.g., use `systemd` instead of `supervisord` or not splitting the FoD installation into 3 phases +(first docker build will be a bit faster and the docker container building is more efficient, as the number of docker image layers is reduced, but subsequent build after some changes in code or settings will require nearly the same amount of time as the first build) + +> Although the examples below use CentOS, just replace `Dockerfile.centos.new` with `Dockerfile.debian` if you wish to use Debian instead. + +## CentOS with `supervisord` + +``` +docker build -f Dockerfile.centos.new -t fod-centos . +``` + +# Running Flowspy + +You should now be able to see relevant images for Flowspy in your Docker environment; + +``` +[user@host-os ~]$ docker images +REPOSITORY TAG IMAGE ID CREATED SIZE +fod-centos latest 42da1eb2b4a5 3 minutes ago 1.73 GB +``` + +Although the web server listens on port 8000 by default you can map this to a different port on the Docker command line. Our examples map port 8000 on the docker container to port 80 on the host OS. + +``` +docker run -p 80:8000 fod-centos # run in foreground +or +docker run -d -p 80:8000 fod-centos # run in background +``` + +If you are using the Debian container then replace `fod-centos` with `fod-debian`. + +You can confirm that the docker container is running with `docker ps` - this will also give you the autogenerated name of the container and show you port mappings if any; + +``` +[user@host-os Dockerfiles.d]$ docker ps +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +5b125f76470d fod-centos "/srv/flowspy/runfod.sh" 1 month ago Up 2 days 0.0.0.0:80->8000/tcp lucid_mcnulty +``` + +You can connect the container directly and start a shell using either the `CONTAINER ID` or the `NAME` and the `docker exec -it` command; + +``` +[user@host-os Dockerfiles.d]$ docker exec -ti lucid_mcnulty bash # replace with name of whose of your container +[root@5b125f76470d /]# +``` + +# Initial setup of Flowspy + +You need to set a couple of options once the container is running; the password for the `admin` user and the details of the NETCONF-enabled router which Flowspy will use to inject FLOWSPEC routes. + +- NETCONF_DEVICE +- NETCONF_PORT +- NETCONF_USER +- NETCONF_PASS + +## via Web GUI + +Access the Flowspy web server at `http://127.0.0.1:8000/setup/` (if you are connecting from _inside_ the container) or `http://your.host-os.ipaddress/` if you are connecting from _outside_ the container (and have mapped port 8000 to port 80). + +Once you hit "Save", the config will be saved **in the container** as `/srv/flowspy/flowspy/settings_local.py`. + +You will need to restart the container for Flowspy to pick up this config; from the Host OS + +``` +docker restart lucid_mcnulty # replace name with whose of your container +``` + +You can now access the Flowspy web server using your new credentials. + +## via manual configuration + +> This needs to be done from **inside** the container. + +The relevant file is `/srv/flowspy/flowspy/settings.py`; the relevant variables are + +- NETCONF_DEVICE +- NETCONF_PORT +- NETCONF_USER +- NETCONF_PASS + +You will then need to use the `manage.py` script to change the admin password; + +``` +cd /srv/flowspy; ./pythonenv ./manage.py changepassword admin +``` + +You will need to restart the container for Flowspy to pick up this config; from the Host OS + +``` +docker restart lucid_mcnulty # replace name with whose of your container +``` + +You can now access the Flowspy web server using your new credentials. + +## As fixed parameters in the Dockerfile + +As a new alternative, admin user name/password and NETCONF parameters can be specified in the Dockerfile.fod.debian or Dockerfile.fod.centos.new: + +Comment the line + +RUN ./install-centos.sh --both --here --supervisord + +and as replacement uncomment the line + +\#RUN ./install-centos.sh --both --here --supervisord --setup_admin_user --setup_admin_user5 admin adminpwd admin@localhost.local testpeer 0.0.0.0/0 --netconf 172.17.0.3 830 netconf netconf +(In case of Dockerfile.fod.debian ./install-centos.sh is replaced by ./install-debian.sh) + +Replace values as it suits you, the 2 parameters following "--setup_admin_user5" are admin username and password, the 4 parameters following "--netconf" are NETCONF_DEVICE, NETCONF_PORT, NETCONF_USER, NETCONF_PASS . + +The docker container built with this changed Dockerfile will have the respective values setup and /setup URL will not be usable any more. + +## Testing NETCONF connectivity + +**Please** make sure that the docker container has IP connectivity to your NETCONF device! + +If you wish to confirm this, then use `yum install telnet` to install telnet in your container and telnet to whichever port on which your NETCONF device is listening (`netconf-ssh`/830 by default); + +``` +[root@5b125f76470d flowspy]# telnet 192.0.2.1 netconf-ssh +Trying 192.0.2.1... +Connected to 192.0.2.1. +Escape character is '^]'. +SSH-2.0-OpenSSH_7.5 +^] +telnet> close +Connection closed. +[root@5b125f76470d flowspy]# +``` + +# Misc + +You may access the containerised Web GUI after the password of the admin user has been reset via `http://127.0.0.1:8000/altlogin`. + +> Do not try to use the Shibboleth login via `/altlogin` as it will not work without a fully configured Shibboleth SP + +# Persistent storage + +The configuration will only last as long as the docker container itself; if the container is stopped or the host OS rebooted then the configuration will be lost. + +You can use [Docker volumes](https://docs.docker.com/storage/volumes/) for persistent storage; + +``` +[user@host-os]$ docker volume create Flowspy-srv +Flowspy-srv +[user@host-os]$ docker volume inspect Flowspy-srv +[ + { + "Driver": "local", + "Labels": {}, + "Mountpoint": "/var/lib/docker/volumes/Flowspy-srv/_data", + "Name": "Flowspy-srv", + "Options": {}, + "Scope": "local" + } +] +``` + +Then use the Docker CLI argument `-v` (**v**olume) to mount the container at runtime; + +``` +docker run -d -p 80:8000 -v Flowspy-srv:/srv fod-centos +``` diff --git a/doc/installation/v1.8/docker_extra.md b/doc/installation/v1.8/docker_extra.md new file mode 100644 index 0000000000000000000000000000000000000000..f565745d8ff2c43bbe770572336580f84e5d316c --- /dev/null +++ b/doc/installation/v1.8/docker_extra.md @@ -0,0 +1,53 @@ +# Extra Docker containers for testing FoD without router hardware + + +### NETCONF test server docker container + +When no real NETCONF-enabled router supporting BGP FlowSpec is available, just for testing the NETCONF test server Docker container can be used: + +In FoD-cloned installation dir, e.g., residing in /srv/flowspy, go to sub directory `router-container` + +``` +docker build -t juniper . + +docker run -it --name juniper -p 830:830 --rm juniper:latest +``` + +To manually test NETCONF access to the test server, see `/router-container/run.txt`. + +Now, find out IP address of the running test server container, e.g., by +docker inspect "$DOCKERID_NETCONF" | grep IPAddress # find out DOCKERID_NETCONF, e.g. by "docker ps" + +In FoD test container (see above) configure this IP address of the NETCONF test server, +with NETCONF port 830, NETCONF_USER "netconf" and NETCONF_PASS "netconf" + +Now, FoD can submit FlowSpec rules which are actually only stored inside the NETCONF test server +without an actual effect on any network, but FoD functionality of controlling rules can be tested. + +### NETCONF test server docker container based on netconfd instead of netopeer + +similar to router-container/Dockerfile but will use netconfd (DEBIAN package) instead of CESNET's netopeer NETCONF server + +- ./Dockerfiles.d/Dockerfile.vnet_router0a : + +### NETCONF test server docker container extended to virtual DDoS test network + +Based on an instance of the NETCONF test server docker container + +Extending with Mininet/OpenVSwitch, +a basic a script for syncing NETCONF rules stored in the test NETCONF server +to OpenFlow rules in Mininet simulating BGP FlowSpec rules (not all cases supported), +SNMPd and a Perl SNMPd statistic collector script + +Yields a more complete simulation of a router for FoD. + +- ./Dockerfiles.d/Dockerfile.vnet_router1 : based on netopeer2 +- ./Dockerfiles.d/Dockerfile.vnet_router2 : similar to Dockerfile.vnet_router1, but will use netconfd (DEBIAN package) instead of CESNET's netopeer NETCONF server + +- ./Dockerfiles.d/Dockerfile.vnet_router2.ubuntu: like ./Dockerfiles.d/Dockerfile.vnet_router2, but based on UBUNTU +- ./Dockerfiles.d/Dockerfile.vnet_router2.debian.exabgp1: extending ./Dockerfiles.d/Dockerfile.vnet_router2 to install some stuff for bgp testing inside the Mininet: exabgp, quagga, bird1 +- ./Dockerfiles.d/Dockerfile.vnet_router2.debian.exabgp2: extending ./Dockerfiles.d/Dockerfile.vnet_router2 to install some stuff for bgp testing inside the Mininet: exabgp, bird2 +- ./Dockerfiles.d/Dockerfile.vnet_router2.debian.exabgp2.topo2: extending ./Dockerfiles.d/Dockerfile.vnet_router2 to simple BGP testing vnet based on exabgp and bird2 + +(find instructions how to build and run inside the Dockerfiles) + diff --git a/doc/installation/v1.8/generic.md b/doc/installation/v1.8/generic.md new file mode 100644 index 0000000000000000000000000000000000000000..85cba8e876407c3441eec9416de9570674226944 --- /dev/null +++ b/doc/installation/v1.8/generic.md @@ -0,0 +1,151 @@ +# Installing FoD v1.7 Generic + +This guide provides general information about the installation of FoD. In case you use Debian/UBUNTU, we provide detailed instructions for the installation. + +Also it assumes that installation is carried out in `/srv/flowspy` +directory. If other directory is to be used, please change the +corresponding configuration files. It is also assumed that the `root` user +will perform every action. + +TO UPDATE + +## Requirements + +### System Requirements +In order to install FoD properly, make sure the following software is installed on your computer. + +- apache 2 +- libapache2-mod-proxy-html +- gunicorn +- redis +- mysql +- python3 +- python3-dev +- python-virtualenv +- pip +- gcc + +### Download FoD +You can clone FoD from GEANT github repository. + + mkdir -p /srv/ + cd /srv/ + git clone https://github.com/GEANT/FoD flowspy + +### Python-virtualenv + + cd /srv + virtualenv --python=python3 /srv/venv + . /srv/venv/bin/activate + +### Pip +In order to install the required python packages for FoD you can use pip: + + . /srv/venv/bin/activate + cd /srv/flowspy + pip install -r requirements.txt + +### Create a database +If you are using mysql, you should create a database: + + mysql -u root -p -e 'create database fod' + +### Copy dist files + + cd /srv/flowspy/flowspy + cp settings.py.dist settings.py + cp urls.py.dist urls.py + +### Device Configuration +FoD generates and commits flowspec rules to a +device via netconf. You have to create an account +with rw access to flowspec and set these credentials +in settings.py. See Configuration for details. + + +### Adding some default data +Into `/srv/flowspy` you will notice that there is a directory called `initial_data`. In there, there is a file called `fixtures_manual.xml` which contains some default static pages for django's flatpages app. In this file we have placed 4 flatpages (2 for Greek, 2 for English) with Information and Terms of Service about the service. To import the flatpages, run from `/srv/flowspy`: + + . /srv/venv/bin/activate + ./manage.py loaddata initial_data/fixtures_manual.xml + + +### Redis +Just make sure redis is installed and started + + # on DEBIAN/UBUNTU (>= DEBIAN 10/UBUNTU 18) + apt-get install redis-server + systemctl enable redis-server + systemctl start redis-server + +### mkdocs-based internal documentation +Just make sure mkdocs is installed and prepare documentation with it +(the documentation is optional and non-essential for the operation of FoD) + + # on DEBIAN/UBUNTU (>= DEBIAN 10/UBUNTU 18) + apt-get install mkdocs + cd /srv/flowspy + mkdocs build + +### Apache2 +Apache proxies gunicorn. Things are more flexible here as you may follow your own configuration and conventions. + +#### Example config +Here is an example configuration. + + <VirtualHost *:80> + ServerName fod.example.org + DocumentRoot /var/www + ErrorLog /var/log/httpd/fod_error.log + LogLevel debug + CustomLog /var/log/httpd/fod_access.log combined + RewriteEngine On + RewriteRule ^/(.*) https://fod.example.com/$1 [L,R] + </VirtualHost> + + + <VirtualHost *:443> + SSLEngine on + SSLProtocol TLSv1 + + SSLCertificateFile /home/local/GEANT/dante.spatharas/filename.crt + SSLCertificateKeyFile /home/local/GEANT/dante.spatharas/filename.key + SSLCACertificateFile /home/local/GEANT/dante.spatharas/filename.crt + + + Alias /static /srv/flowspy/static + + AddDefaultCharset UTF-8 + IndexOptions +Charset=UTF-8 + + #SSLProxyEngine off + ProxyErrorOverride off + ProxyTimeout 28800 + ProxyPass /static ! + ProxyPass / http://localhost:8080/ retry=0 + ProxyPassReverse / http://localhost:8080/ + + </VirtualHost> + +`Important!` You have to comment out/disable the default `Virtualhost` defined on line 74 until the end of this block at `/etc/httpd/conf.d/ssl.conf `. + + +### Gunicorn +FoD is served via gunicorn and is then proxied by Apache. If the above +directory conventions have been followed so far, then your configuration +should be: + + CONFIG = { + 'mode': 'django', + 'working_dir': '/srv/flowspy', + 'args': ( + '--bind=127.0.0.1:8081', + '--workers=1', + '--worker-class=egg:gunicorn#gevent', + '--timeout=30', + '--debug', + '--log-level=debug', + '--log-file=/var/log/gunicorn/fod.log', + ), + } + diff --git a/mkdocs.yml b/mkdocs.yml index 9b3d359bbf243dc5877cb722a127bab57feda58f..0f99252ae98b413575f67a893cd8e038fab37105 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -2,7 +2,7 @@ site_name: Firewall On Demand #repo_url: https://github.com/grnet/flowspy/ repo_url: https://github.com/GEANT/FOD/ #site_author: Stavros Kroustouris -site_author: GÉANT4-3 WP8 T3 +site_author: GÉANT5-1 WP8-T3 site_dir: static/site docs_dir: doc #theme: readthedocs @@ -21,14 +21,23 @@ nav: - 'CentOS': 'installation/v1.7/centos.md' - 'Docker': 'installation/v1.7/docker.md' - 'Docker Extra for Testing': 'installation/v1.7/docker_extra.md' + - 'v1.8': + - 'Generic': 'installation/v1.8/generic.md' + - 'DEBIAN/UBUNTU': 'installation/v1.8/debian_ubuntu.md' + - 'CentOS': 'installation/v1.8/centos.md' + - 'Docker': 'installation/v1.8/docker.md' + - 'Docker Extra for Testing': 'installation/v1.8/docker_extra.md' - 'Configuration': - 'v1.3': 'configuration/configuration-v1.3.md' - 'v1.7': 'configuration/configuration-v1.7.md' + - 'v1.8': 'configuration/configuration-v1.8.md' - 'Administration and Usage': - 'Basic Administration and Usage (v1.7)': 'administration_and_usage/basic_administration_and_usage-v1.7.md' + - 'Basic Administration and Usage (v1.8)': 'administration_and_usage/basic_administration_and_usage-v1.8.md' - 'API': - 'v1.3': 'api/api-v1.3.md' - 'v1.7': 'api/api-v1.7.md' + - 'v1.8': 'api/api-v1.8.md' - 'Development': - 'Design Overview': 'development/design-overview.md' diff --git a/templates/500.html b/templates/500.html index eeb67248eab1fba214747f03f63d20ce9b4f71e2..c7e72433b8dbb8c462eb97bcae38678daf64694e 100644 --- a/templates/500.html +++ b/templates/500.html @@ -3362,7 +3362,7 @@ n6PVRQV1z4Xb5R2/ig2gdF2X+48iDxWt0GPAb2xj+2RYXbWgbiBKtWpQ/T913YsPOb733o93/rGN <div id="footer"> If you have any questions or need help, contact GÉANT OC at <a href='mailto:support@oc.geant.net'>support@oc.geant.net</a> or +44 1223 733033. <a href="https://github.com/GEANT/FOD">FoD</strong></a> - <p> - Designed and developed in GÉANT4-3 WP8 T3; Originally designed and developed by GRNET NOC + Designed and developed in GÉANT5-1 WP8-T3; Originally designed and developed by GRNET NOC </div> </div> diff --git a/templates/footer.html b/templates/footer.html index 177266e450e132f4c23c08aafa3259ba098ed447..cd07dd95936e99fbac12e1df788998605d48b47d 100644 --- a/templates/footer.html +++ b/templates/footer.html @@ -7,7 +7,7 @@ {% endif %} <!--<div style="padding-top: 10px;"><a href="https://code.grnet.gr/projects/flowspy">Version: <strong>{{ settings.SW_VERSION }}</strong></a> --> <div style="padding-top: 10px;"><a href="https://github.com/GEANT/FOD">Version: <strong>{{ settings.SW_VERSION }}</strong></a> - - {% trans "Designed and developed in GÉANT4-3 WP8 T3; Originally designed and developed by GRNET NOC" %} + {% trans "Designed and developed in GÉANT5-1 WP8-T3; Originally designed and developed by GRNET NOC" %} </a> </div> <div style="padding-top: 10px;">