Skip to content
Snippets Groups Projects
Commit 8ec6776c authored by Stauros Kroustouris's avatar Stauros Kroustouris
Browse files

docs

parent c35eb1fe
Branches
Tags
No related merge requests found
Hide whitespace changes
Inline Side-by-side
README.md
+ 1
− 1
View file @ 8ec6776c
...@@ -92,7 +92,7 @@ And here is a PUT request example: ...@@ -92,7 +92,7 @@ And here is a PUT request example:
You can find more about FoD or raise your issues at GRNET FoD You can find more about FoD or raise your issues at GRNET FoD
repository: [GRNET repo](https://code.grnet.gr/fod) or [Github repo](https://github.com/grnet/flowspy). repository: [GRNET repo](https://code.grnet.gr/fod) or [Github repo](https://github.com/grnet/flowspy).
You can contact us directly at noc{at}noc[dot]grnet(.)gr You can contact us directly at dev{at}noc[dot]grnet(.)gr
## Copyright and license ## Copyright and license
......
doc/configuration.md 0 → 100644
+ 271
− 0
View file @ 8ec6776c
Time to configure flowspy.
First of all you have to copy the dist files to their proper position:
# cd /srv/flowspy/flowspy
# cp settings.py.dist settings.py
# cp urls.py.dist urls.py
Then, you have to edit the settings.py file to correspond to your needs. The settings one has to focus on are:
## Settings.py
Its time to configure `settings.py` in order to connect flowspy with a database, a network device and a broker.
So lets edit settings.py file.
It is strongly advised that you do not change the following to False
values unless, you want to integrate FoD with you CRM or members
database. This implies that you are able/have the rights to create
database views between the two databases:
PEER_MANAGED_TABLE = True
PEER_RANGE_MANAGED_TABLE = True
PEER_TECHC_MANAGED_TABLE = True
By doing that the corresponding tables as defined in peers/models will
not be created. As noted above, you have to create the views that the
tables will rely on.
### Administratos
ADMINS: set your admin name and email (assuming that your server can send notifications)
### Secret Key
Please put a random string in `SECRET_KEY` setting.
Make this *unique*, and don't share it with anybody. It's the unique identifier of this instance of the application.
### Allowed hosts
A list of strings representing the host/domain names that this Django site can serve. This is a security measure to prevent an attacker from poisoning caches and password reset emails with links to malicious hosts by submitting requests with a fake HTTP Host header, which is possible even under many seemingly-safe webserver configurations.
For example:
ALLOWED_HOSTS = ['*']
### Protected subnets
Subnets for which source or destination address will prevent rule creation and notify the `NOTIFY_ADMIN_MAILS`.
PROTECTED_SUBNETS = ['10.10.0.0/16']
### Database
`DATABASES` should contain the database credentials:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'flowspy',
'USER': '<db user>',
'PASSWORD': '<db password>',
'HOST': '<db host>',
'PORT': '',
}
}
### Localization
By default Flowspy has translations for English and Greek. In case you want to add
another language, or remove one of the existing, you can change the `LANGUAGES`
variable and follow [django's localization documentation](https://docs.djangoproject.com/en/1.4/topics/i18n/translation/#localization-how-to-create-language-files)
You might want to change `TIME_ZONE` setting too. Here is a [list](http://en.wikipedia.org/wiki/List_of_tz_database_time_zones)
### Cache
Flowspy uses cache in order to be fast. We recomend the usage of memcached, but
any cache backend supported by django should work fine.
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
'LOCATION': '127.0.0.1:11211',
}
}
### Network device access
We have to inform django about the device we set up earlier.
NETCONF_DEVICE = "device.example.com"
NETCONF_USER = "<netconf user>"
NETCONF_PASS = "<netconf password>"
NETCONF_PORT = 830
### Beanstalkd
Beanstalk configuration (as a broker for celery)
BROKER_HOST = "localhost"
BROKER_PORT = 11300
POLLS_TUBE = 'polls'
BROKER_URL = "beanstalk://localhost:11300//"
### Notifications
Outgoing mail address and prefix.
SERVER_EMAIL = "Example FoD Service <noreply@example.com>"
EMAIL_SUBJECT_PREFIX = "[FoD] "
NOTIFY_ADMIN_MAILS = ["admin@example.com"]
If you have not installed an outgoing mail server you can always use your own account (either corporate or gmail, hotmail ,etc) by adding the
following lines in settings.py:
EMAIL_USE_TLS = True #(or False)
EMAIL_HOST = 'smtp.example.com'
EMAIL_HOST_USER = 'username'
EMAIL_HOST_PASSWORD = 'yourpassword'
EMAIL_PORT = 587 #(outgoing)
### Whois servers
Add two whois servers in order to be able to get all the subnets for an AS.
PRIMARY_WHOIS = 'whois.example.com'
ALTERNATE_WHOIS = 'whois.example.net'
### Branding
Fill your company's information in order to show it in flowspy.
BRANDING = {
'name': 'Example.com',
'url': 'https://example.com',
'footer_iframe': 'https://example.com/iframes/footer/',
'facebook': '//facebook.com/example.com',
'twitter': '//twitter.com/examplecom',
'phone': '800-12-12345',
'email': 'helpdesk@example.com',
'logo': 'logo.png',
'favicon': 'favicon.ico',
}
### Shibboleth
Flowspy supports shibboleth authentication.
SHIB_AUTH_ENTITLEMENT = 'urn:mace'
SHIB_ADMIN_DOMAIN = 'example.com'
SHIB_LOGOUT_URL = 'https://example.com/Shibboleth.sso/Logout'
SHIB_USERNAME = ['HTTP_EPPN']
SHIB_MAIL = ['mail', 'HTTP_MAIL', 'HTTP_SHIB_INETORGPERSON_MAIL']
SHIB_FIRSTNAME = ['HTTP_SHIB_INETORGPERSON_GIVENNAME']
SHIB_LASTNAME = ['HTTP_SHIB_PERSON_SURNAME']
SHIB_ENTITLEMENT = ['HTTP_SHIB_EP_ENTITLEMENT']
### Syncing the database
To create all the tables needed by FoD we have to run the following commands:
cd /srv/flowspy
./manage.py syncdb --noinput
./manage.py migrate
## Create a superuser
A superuser can be added by using the following command from `/srv/flowspy/`:
./manage.py createsuperuser
## Propagate the flatpages
Inside the initial\_data/fixtures\_manual.xml file we have placed 4
flatpages (2 for Greek, 2 for English) with Information and Terms of
Service about the service. To import the flatpages, run from root
folder:
python manage.py loaddata initial_data/fixtures_manual.xml
### Celery
Celery is a distributed task queue, which helps FoD run some async tasks, like applying a flowspec rule to a router.
`Note` In order to check if celery runs or even debug it, you can run:
./manage.py celeryd --loglevel=debug
### Testing/Debugging
In order to see what went wrong you can check the following things.
#### Django
You can start the server manually by running:
./manage.py runserver 127.0.0.1:8081
By doing so, you can serve your application like gunicord does just to test that its starting properly. This command should not be used in production!
Of course you have to stop gunicorn and make sure that port 8081 is free.
#### Gunicorn
Just curl from the server http://localhost:8081
#### Celery
In order to check if celery is working properly one can start celery by typing:
./manage.py celeryd --loglevel=debug
Again this is for debug purposes.
#### Connectivity to flowspec device
Just try to connect with the credentials you entered in settings.py from the host that will be serving flowspy.
#### General Test
Log in to the admin interface via https://<hostname>/admin. Go to Peer ranges and add a new range (part of/or a complete subnet), eg. 10.20.0.0/19 Go to Peers and add a new peer, eg. id: 1, name: Test, AS: 16503, tag: TEST and move the network you have created from Available to Chosen. From the admin front, go to User, and edit your user. From the bottom of the page, select the TEST peer and save. Last but not least, modify as required the existing (example.com) Site instance (admin home->Sites). You are done. As you are logged-in via the admin, there is no need to go through Shibboleth at this time. Go to https://<hostname>/ and create a new rule. Your rule should be applied on the flowspec capable device after aprox. 10 seconds.
## Footer
Under the templates folder (templates), you can alter the footer.html
file to include your own footer messages, badges, etc.
## Welcome Page
Under the templates folder (templates), you can alter the welcome page -
welcome.html with your own images, carousel, videos, etc.
## Usage
### Web interface
FoD comes with a web interface, in which one can edit and apply new routes.
### Rest Api
FoD provides a rest api. It uses token as authentication method.
### Generating Tokens
A user can generate a token for his account on "my profile" page from FoD's
UI. Then by using this token in the header of the request he can list, retrieve,
modify and create rules.
### Example Usage
Here are some examples:
#### GET items
- List all the rules your user has created (admin users can see all the rules)
curl -X GET https://fod.example.com/api/routes/ -H 'Authorization: Token <Your users token>'
- Retrieve a specific rule:
curl -X GET https://fod.example.com/api/routes/<rule_id>/ -H 'Authorization: Token <Your users token>'
- In order to create or modify a rule you have to use POST/PUT methods.
#### POST/PUT rules
In order to update or create rules you can follow this example:
##### Foreign Keys
In order to create/modify a rule you have to connect the rule with some foreign keys:
###### Ports, Fragmentypes, protocols, thenactions
When creating a rule, one can specify:
- source port
- destination port
- port (if source = destination)
That can be done by getting the url of the desired port instance from `/api/ports/<port_id>/`
Same with Fragmentypes in `/api/fragmenttypes/<fragmenttype_id>/`, protocols in `/api/matchprotocol/<protocol_id>/` and then actions in `/api/thenactions/<action_id>/`.
Since we have the urls we want to connect with the rule we want to create, we can make a POST request like the following:
curl -X POST -H 'Authorization: Token <Your users token>' -F "name=Example" -F "comments=Description" -F "source=0.0.0.0/0" -F "sourceport=https://fod.example.com/api/ports/7/" -F "destination=203.0.113.12" https://fod.example.com/api/routes/
And here is a PUT request example:
curl -X PUT -F "name=Example" -F "comments=Description" -F "source=0.0.0.0/0" -F "sourceport=https://fod.example.com/api/ports/7/" -F "destination=83.212.9.93" https://fod.example.com/api/routes/12/ -H 'Authorization: Token <Your users token>'
doc/index.md
+ 21
− 641
View file @ 8ec6776c
Firewall on Demand # Description
==================
Description
-----------
Firewall on Demand applies, via Netconf, flow rules to a network device. Firewall on Demand applies, via Netconf, flow rules to a network device.
These rules are then propagated via e-bgp to peering routers. Each user These rules are then propagated via e-bgp to peering routers. Each user
...@@ -28,655 +24,39 @@ flowspec capable routers. Of course FoD could apply rules directly (via ...@@ -28,655 +24,39 @@ flowspec capable routers. Of course FoD could apply rules directly (via
NETCONF always) to a router and then ibgp would do the rest. In GRNET’s NETCONF always) to a router and then ibgp would do the rest. In GRNET’s
case the flowspec capable device is an EX4200. case the flowspec capable device is an EX4200.
> **attention** > ** Attention **
> >
> Make sure your FoD server has ssh access to your flowspec device. > Make sure your FoD server has ssh access to your flowspec device.
> **attention** # Contact
>
> Installation instructions assume a clean Debian Wheezy with Django 1.4
Contact
-------
You can find more about FoD or raise your issues at [GRNET FoD
repository][].
You can contact us directly at staurosk{at}noc[dot]grnet(.)gr
Repositories
------------
[GRNET FoD repository]: https://code.grnet.gr/projects/flowspy
[Github FoD repository]: https://github.com/grnet/flowspy
Installation
============
Debian Wheezy (x64) - Django 1.4.x
----------------------------------
This guide assumes that installation is carried out in /srv/flowspy
directory. If other directory is to be used, please change the
corresponding configuration files. It is also assumed that the root user
will perform every action.
### Upgrading from v\<1.1.x
> **note**
>
> If PEER\_\*\_TABLE tables are set to FALSE in settings.py, you need to
> perform the south migrations per application:
>
> ./manage.py migrate longerusername
> ./manage.py migrate flowspec
> ./manage.py migrate accounts
If upgrading from flowspy version \<1.1.x pay attention to settings.py
changes. Also, do not forget to run if PEER\_\*\_TABLE tables are set to
TRUE in settings.py:
./manage.py migrate
to catch-up with latest database changes.
### Upgrading from v\<1.0.x
If upgrading from flowspy version \<1.0.x pay attention to settings.py
changes. Also, do not forget to run:
./manage.py migrate
to catch-up with latest database changes.
### Required system packages
Update and install the required packages:
apt-get update
apt-get upgrade
apt-get install mysql-server apache2 memcached libapache2-mod-proxy-html gunicorn beanstalkd python-django python-django-south python-django-tinymce tinymce python-mysqldb python-yaml python-memcache python-django-registration python-ipaddr python-lxml mysql-client git python-django-celery python-paramiko python-gevent vim
Also, django rest framework package is required. In debian Wheezy it is
not available, but one can install it via pip.
> **note**
>
> Set username and password for mysql if used
> **note**
>
> If you wish to deploy an outgoing mail server, now it is time to do
> it. Otherwise you could set FoD to send out mails via a third party
> account
### Create a database
If you are using mysql, you should create a database:
mysql -u root -p -e 'create database fod'
### Required application packages
Get the required packages and their dependencies and install them:
apt-get install libxml2-dev libxslt-dev gcc python-dev
- ncclient: NETCONF python client:
cd ~
git clone https://github.com/leopoul/ncclient.git
cd ncclient
python setup.py install
- nxpy: Python Objects from/to XML proxy:
cd ~
git clone https://code.grnet.gr/git/nxpy
cd nxpy
python setup.py install
- flowspy: core web application. Installation is done at /srv/flowspy::
cd /srv
git clone https://code.grnet.gr/git/flowspy
cd flowspy
Application configuration
=========================
Copy settings.py.dist to settings.py:
cd flowspy
cp settings.py.dist settings.py
Edit settings.py file and set the following according to your
configuration:
ADMINS: set your admin name and email (assuming that your server can send notifications)
DATABASES (to point to your local database). You could use views instead of tables for models: peer, peercontacts, peernetworks. For this to work we suggest MySQL with MyISAM db engine
SECRET_KEY : Make this unique, and don't share it with anybody
STATIC_ROOT: /srv/flowspy/static (or your installation directory)
STATIC_URL (static media directory) . If you have followed the above this should be: /static/
TEMPLATE_DIRS : If you have followed the above this should be: /srv/flowspy/templates
CACHE_BACKEND: Enable Memcached for production or leave to DummyCache for development environments
Alternatively you could go for redis with the corresponding Django client lib.
NETCONF_DEVICE (tested with Juniper EX4200 but any BGP enabled Juniper should work). This is the flowspec capable device
NETCONF_USER (enable ssh and netconf on device)
NETCONF_PASS
If beanstalk is selected the following should be left intact.
BROKER_HOST (beanstalk host)
BROKER_PORT (beanstalk port)
SERVER_EMAIL
EMAIL_SUBJECT_PREFIX
If beanstalk is selected the following should be left intact.
BROKER_URL (beanstalk url)
SHIB_AUTH_ENTITLEMENT (if you go for Shibboleth authentication)
NOTIFY_ADMIN_MAILS (bcc mail addresses)
PROTECTED_SUBNETS (subnets for which source or destination address will prevent rule creation and notify the NOTIFY_ADMIN_MAILS)
The whois client is meant to be used in case you have inserted peers with their ASes in the peers table and wish to get network info for each one in an automated manner.
PRIMARY_WHOIS
ALTERNATE_WHOIS
If you wish to deploy FoD with Shibboleth change the following attributes according to your setup:
SHIB_AUTH_ENTITLEMENT = 'urn:mace'
SHIB_ADMIN_DOMAIN = 'example.com'
SHIB_LOGOUT_URL = 'https://example.com/Shibboleth.sso/Logout'
SHIB_USERNAME = ['HTTP_EPPN']
SHIB_MAIL = ['mail', 'HTTP_MAIL', 'HTTP_SHIB_INETORGPERSON_MAIL']
SHIB_FIRSTNAME = ['HTTP_SHIB_INETORGPERSON_GIVENNAME']
SHIB_LASTNAME = ['HTTP_SHIB_PERSON_SURNAME']
SHIB_ENTITLEMENT = ['HTTP_SHIB_EP_ENTITLEMENT']
If you have not installed an outgoing mail server you can always use
your own account (either corporate or gmail, hotmail ,etc) by adding the
following lines in settings.py:
EMAIL_USE_TLS = True #(or False)
EMAIL_HOST = 'smtp.example.com'
EMAIL_HOST_USER = 'username'
EMAIL_HOST_PASSWORD = 'yourpassword'
EMAIL_PORT = 587 #(outgoing)
It is strongly advised that you do not change the following to False
values unless, you want to integrate FoD with you CRM or members
database. This implies that you are able/have the rights to create
database views between the two databases:
PEER_MANAGED_TABLE = True
PEER_RANGE_MANAGED_TABLE = True
PEER_TECHC_MANAGED_TABLE = True
By doing that the corresponding tables as defined in peers/models will
not be created. As noted above, you have to create the views that the
tables will rely on.
> **note**
>
> Soon we will release a version with django-registration as a means to
> add users and Shibboleth will become an alternative
Let’s move on with some copies and dir creations:
mkdir /var/log/fod
chown www-data.www-data /var/log/fod
cp urls.py.dist urls.py
cd ..
> **note**
>
> LOG\_FILE\_LOCATION in settings.py is set to **/var/log/fod**. Adjust
> the chown command above to your selected dir.
System configuration
====================
Apache operates as a gunicorn Proxy with WSGI and Shibboleth modules
enabled. Depending on the setup the apache configuration may vary:
a2enmod rewrite
a2enmod proxy
a2enmod ssl
a2enmod proxy_http
If shibboleth is to be used:
apt-get install libapache2-mod-shib2
a2enmod shib2
Now it is time to configure beanstalk, gunicorn, celery and apache.
beanstalkd
----------
Enable beanstalk by editting /etc/default/beanstalkd:
vim /etc/default/beanstalkd
Uncomment the line **START=yes** to enable beanstalk
Start beanstalkd:
service beanstalkd start
gunicorn.d
----------
Create and edit /etc/gunicorn.d/fod:
vim /etc/gunicorn.d/fod
FoD is served via gunicorn and is then proxied by Apache. If the above
directory conventions have been followed so far, then your configuration
should be:
CONFIG = {
'mode': 'django',
'working_dir': '/srv/flowspy',
'args': (
'--bind=127.0.0.1:8081',
'--workers=1',
'--worker-class=egg:gunicorn#gevent',
'--timeout=30',
'--debug',
'--log-level=debug',
'--log-file=/var/log/gunicorn/fod.log',
),
}
celeryd
=======
Celery is used over beanstalkd to apply firewall rules in a serial
manner so that locks are avoided on the flowspec capable device. In our
setup celery runs via django. That is why the python-django-celery
package was installed.
Create the celeryd daemon at /etc/init.d/celeryd **if it does not
already exist**:
vim /etc/init.d/celeryd
The configuration should be:
#!/bin/sh -e
# ============================================
# celeryd - Starts the Celery worker daemon.
# ============================================
#
# :Usage: /etc/init.d/celeryd {start|stop|force-reload|restart|try-restart|status}
# :Configuration file: /etc/default/celeryd
#
# See http://docs.celeryq.org/en/latest/cookbook/daemonizing.html#init-script-celeryd
### BEGIN INIT INFO
# Provides: celeryd
# Required-Start: $network $local_fs $remote_fs
# Required-Stop: $network $local_fs $remote_fs
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: celery task worker daemon
# Description: Starts the Celery worker daemon for a single project.
### END INIT INFO
#set -e
DEFAULT_PID_FILE="/var/run/celery/%n.pid"
DEFAULT_LOG_FILE="/var/log/celery/%n.log"
DEFAULT_LOG_LEVEL="INFO"
DEFAULT_NODES="celery"
DEFAULT_CELERYD="-m celery.bin.celeryd_detach"
ENABLED="false"
[ -r "$CELERY_DEFAULTS" ] && . "$CELERY_DEFAULTS"
[ -r /etc/default/celeryd ] && . /etc/default/celeryd
if [ "$ENABLED" != "true" ]; then
echo "celery daemon disabled - see /etc/default/celeryd."
exit 0
fi
CELERYD_PID_FILE=${CELERYD_PID_FILE:-${CELERYD_PIDFILE:-$DEFAULT_PID_FILE}}
CELERYD_LOG_FILE=${CELERYD_LOG_FILE:-${CELERYD_LOGFILE:-$DEFAULT_LOG_FILE}}
CELERYD_LOG_LEVEL=${CELERYD_LOG_LEVEL:-${CELERYD_LOGLEVEL:-$DEFAULT_LOG_LEVEL}}
CELERYD_MULTI=${CELERYD_MULTI:-"celeryd-multi"}
CELERYD=${CELERYD:-$DEFAULT_CELERYD}
CELERYCTL=${CELERYCTL:="celeryctl"}
CELERYD_NODES=${CELERYD_NODES:-$DEFAULT_NODES}
export CELERY_LOADER
if [ -n "$2" ]; then
CELERYD_OPTS="$CELERYD_OPTS $2"
fi
CELERYD_LOG_DIR=`dirname $CELERYD_LOG_FILE`
CELERYD_PID_DIR=`dirname $CELERYD_PID_FILE`
if [ ! -d "$CELERYD_LOG_DIR" ]; then
mkdir -p $CELERYD_LOG_DIR
fi
if [ ! -d "$CELERYD_PID_DIR" ]; then
mkdir -p $CELERYD_PID_DIR
fi
# Extra start-stop-daemon options, like user/group.
if [ -n "$CELERYD_USER" ]; then
DAEMON_OPTS="$DAEMON_OPTS --uid=$CELERYD_USER"
chown "$CELERYD_USER" $CELERYD_LOG_DIR $CELERYD_PID_DIR
fi
if [ -n "$CELERYD_GROUP" ]; then
DAEMON_OPTS="$DAEMON_OPTS --gid=$CELERYD_GROUP"
chgrp "$CELERYD_GROUP" $CELERYD_LOG_DIR $CELERYD_PID_DIR
fi
if [ -n "$CELERYD_CHDIR" ]; then
DAEMON_OPTS="$DAEMON_OPTS --workdir=\"$CELERYD_CHDIR\""
fi
check_dev_null() {
if [ ! -c /dev/null ]; then
echo "/dev/null is not a character device!"
exit 1
fi
}
export PATH="${PATH:+$PATH:}/usr/sbin:/sbin"
stop_workers () {
$CELERYD_MULTI stop $CELERYD_NODES --pidfile="$CELERYD_PID_FILE"
}
start_workers () {
$CELERYD_MULTI start $CELERYD_NODES $DAEMON_OPTS \
--pidfile="$CELERYD_PID_FILE" \
--logfile="$CELERYD_LOG_FILE" \
--loglevel="$CELERYD_LOG_LEVEL" \
--cmd="$CELERYD" \
$CELERYD_OPTS
}
restart_workers () {
$CELERYD_MULTI restart $CELERYD_NODES $DAEMON_OPTS \
--pidfile="$CELERYD_PID_FILE" \
--logfile="$CELERYD_LOG_FILE" \
--loglevel="$CELERYD_LOG_LEVEL" \
--cmd="$CELERYD" \
$CELERYD_OPTS
}
case "$1" in
start)
check_dev_null
start_workers
;;
stop)
check_dev_null
stop_workers
;;
reload|force-reload)
echo "Use restart"
;;
status)
$CELERYCTL status $CELERYCTL_OPTS
;;
restart)
check_dev_null
restart_workers
;;
try-restart)
check_dev_null
restart_workers
;;
*)
echo "Usage: /etc/init.d/celeryd {start|stop|restart|try-restart|kill}"
exit 1
;;
esac
exit 0
celeryd configuration
=====================
celeryd requires a /etc/default/celeryd file to be in place. Thus we are
going to create this file (/etc/default/celeryd):
vim /etc/default/celeryd
Again if the directory conventions have been followed the file is (pay
attention to the CELERYD\_USER, CELERYD\_GROUP and change accordingly) :
# Default: false
ENABLED="true"
# Name of nodes to start, here we have a single node
CELERYD_NODES="w1"
# or we could have three nodes:
#CELERYD_NODES="w1 w2 w3"
# Where to chdir at start.
CELERYD_CHDIR="/srv/flowspy"
# How to call "manage.py celeryd_multi"
CELERYD_MULTI="python $CELERYD_CHDIR/manage.py celeryd_multi"
# How to call "manage.py celeryctl"
CELERYCTL="python $CELERYD_CHDIR/manage.py celeryctl"
# Extra arguments to celeryd
#CELERYD_OPTS="--time-limit=300 --concurrency=8"
CELERYD_OPTS="-E -B --schedule=/var/run/celery/celerybeat-schedule --concurrency=1 --soft-time-limit=180 --time-limit=1800"
# Name of the celery config module.
CELERY_CONFIG_MODULE="celeryconfig"
# %n will be replaced with the nodename.
CELERYD_LOG_FILE="/var/log/celery/fod_%n.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_USER="root"
CELERYD_GROUP="root"
# Name of the projects settings module.
export DJANGO_SETTINGS_MODULE="flowspy.settings"
Apache
======
Apache proxies gunicorn. Things are more flexible here as you may follow
your own configuration and conventions. Create and edit
/etc/apache2/sites-available/fod. You should set \<server\_name\> and
\<admin\_mail\> along with your certificates. If under testing
environment, you can use the provided snakeoil certs. If you do not
intent to use Shibboleth delete or comment the corresponding
configuration parts inside **Shibboleth configuration** :
vim /etc/apache2/sites-available/fod
Again if the directory conventions have been followed the file should
be:
<VirtualHost *:80>
ServerAdmin webmaster@localhost
ServerName fod.example.com
DocumentRoot /var/www
ErrorLog ${APACHE_LOG_DIR}/fod_error.log
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel debug
CustomLog ${APACHE_LOG_DIR}/fod_access.log combined
Alias /static /srv/flowspy/static
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule ^/(.*) https://fod.example.com/$1 [L,R]
</VirtualHost>
<VirtualHost *:443>
ServerName fod.example.com
ServerAdmin webmaster@localhost
ServerSignature On
SSLEngine on
SSLCertificateFile /etc/ssl/certs/fod.example.com.crt
SSLCertificateChainFile /etc/ssl/certs/example-chain.pem
SSLCertificateKeyFile /etc/ssl/private/fod.example.com.key
AddDefaultCharset UTF-8
IndexOptions +Charset=UTF-8
ShibConfig /etc/shibboleth/shibboleth2.xml
Alias /shibboleth-sp /usr/share/shibboleth
<Location /login>
AuthType shibboleth
ShibRequireSession On
ShibUseHeaders On
ShibRequestSetting entityID https://idp.example.com/idp/shibboleth
require valid-user
</Location>
# Shibboleth debugging CGI script
ScriptAlias /shibboleth/test /usr/lib/cgi-bin/shibtest.cgi
<Location /shibboleth/test>
AuthType shibboleth
ShibRequireSession On
ShibUseHeaders On
require valid-user
</Location>
<Location /Shibboleth.sso>
SetHandler shib
</Location>
# Shibboleth SP configuration
#SetEnv proxy-sendchunked
<Proxy *>
Order allow,deny
Allow from all
</Proxy>
SSLProxyEngine off
ProxyErrorOverride off
ProxyTimeout 28800
ProxyPass /static !
ProxyPass /shibboleth !
ProxyPass /Shibboleth.sso !
ProxyPass / http://localhost:8081/ retry=0
ProxyPassReverse / http://localhost:8081/
Alias /static /srv/flowspy/static
LogLevel warn
Now, enable your site. You might want to disable the default site if fod
is the only site you host on your server:
a2dissite default
a2ensite fod
You are not far away from deploying FoD. When asked for a super user,
create one:
cd /srv/flowspy
python manage.py syncdb
python manage.py migrate longerusername
python manage.py migrate flowspec
python manage.py migrate djcelery
python manage.py migrate accounts
python manage.py migrate
If you have not changed the values of the PEER\_\*\_TABLE variables to
False and thus you are going for a default installation (that is
PEER\_\*\_TABLE variables are set to True) , then run:
python manage.py migrate peers
If however you have set the PEER\_\*\_TABLE variables to False and by
accident you have ran the command above, then you have to cleanup you
database manually by dropping the peer\* tables plus the techc\_email
table. For MySQL the command is:
DROP TABLE `peer`, `peer_networks`, `peer_range`, `peer_techc_emails`, techc_email;
Restart, gunicorn and apache:
service gunicorn restart && service apache2 restart
Propagate the flatpages
=======================
Inside the initial\_data/fixtures\_manual.xml file we have placed 4
flatpages (2 for Greek, 2 for English) with Information and Terms of
Service about the service. To import the flatpages, run from root
folder:
python manage.py loaddata initial_data/fixtures_manual.xml
Testing the platform
====================
Log in to the admin interface via [https:\\/\\/][]\<hostname\>/admin. Go You can find more about FoD or raise your issues at [Github FoD
to Peer ranges and add a new range (part of/or a complete subnet), eg. repository].
10.20.0.0/19 Go to Peers and add a new peer, eg. id: 1, name: Test, AS:
16503, tag: TEST and move the network you have created from Avalable to
Chosen. From the admin front, go to User, and edit your user. From the
bottom of the page, select the TEST peer and save. Last but not least,
modify as required the existing (example.com) Site instance (admin
home-\>Sites). You are done. As you are logged-in via the admin, there
is no need to go through Shibboleth at this time. Go to
<hostname\> and create a new rule. Your rule should be
applied on the flowspec capable device after aprox. 10 seconds. If no
Shibboleth authentication is available, a
<hostname\> altlogin is provided.
Branding You can contact us directly at dev{at}noc[dot]grnet(.)gr
========
Via the admin interface you can modify flatpages to suit your needs # Repositories
Footer - [GRNET FoD repository](https://code.grnet.gr/projects/flowspy)
------
Under the templates folder (templates), you can alter the footer.html - [Github FoD repository](https://github.com/grnet/flowspy)
file to include your own footer messages, badges, etc.
Welcome Page
------------
Under the templates folder (templates), you can alter the welcome page - ## Copyright and license
welcome.html with your own images, carousel, videos, etc.
Usage Copyright © 2010-2014 Greek Research and Technology Network (GRNET S.A.)
======
Web interface This program is free software: you can redistribute it and/or modify
------------------------- it under the terms of the GNU General Public License as published by
FoD comes with a web interface, in which one can edit and apply new routes. the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
Rest Api This program is distributed in the hope that it will be useful,
-------------- but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
There is a rest api available in /api/v1/. One can set new rules or see the applied ones by using it. You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
doc/installation/debian_wheezy.md 0 → 100644
+ 483
− 0
View file @ 8ec6776c
# Debian Wheezy (x64) - Django 1.4.x
The following document describes the installation process of Firewall On Demand
on a Debian Wheezy machine with Django 1.4
## Upgrading
### Upgrading from v<1.1.x
> ** Note **
>
> If PEER\_\*\_TABLE tables are set to FALSE in settings.py, you need to
> perform the south migrations per application:
./manage.py migrate longerusername
./manage.py migrate flowspec
./manage.py migrate accounts
Also, pay attention to settings.py
changes. Also, do not forget to run if PEER\_\*\_TABLE tables are set to
TRUE in settings.py:
./manage.py migrate
to catch-up with latest database changes.
### Upgrading from v<1.0.x
If upgrading from flowspy v<1.0.x pay attention to settings.py
changes. Also, do not forget to run:
./manage.py migrate
to catch-up with latest database changes.
## Installing from scratch
### Required system packages (with apt)
Update and install the required packages:
apt-get update
apt-get upgrade
apt-get install mysql-server apache2 memcached libapache2-mod-proxy-html gunicorn beanstalkd python-django python-django-south python-django-tinymce tinymce python-mysqldb python-yaml python-memcache python-django-registration python-ipaddr python-lxml mysql-client git python-django-celery python-paramiko python-gevent
Also, django rest framework package is required. In debian Wheezy it is
not available, but one can install it via pip.
> **note**
>
> Set username and password for mysql if used
> **note**
>
> If you wish to deploy an outgoing mail server, now it is time to do
> it. Otherwise you could set FoD to send out mails via a third party
> account
### Create a database
If you are using mysql, you should create a database:
mysql -u root -p -e 'create database fod'
### Required application packages
Get the required packages and their dependencies and install them:
apt-get install libxml2-dev libxslt-dev gcc python-dev
- ncclient: NETCONF python client:
cd ~
git clone https://github.com/leopoul/ncclient.git
cd ncclient
python setup.py install
- nxpy: Python Objects from/to XML proxy:
cd ~
git clone https://code.grnet.gr/git/nxpy
cd nxpy
python setup.py install
- flowspy: core web application. Installation is done at /srv/flowspy::
cd /srv
git clone https://code.grnet.gr/git/flowspy
cd flowspy
Let’s move on with some copies and dir creations:
mkdir /var/log/fod
chown www-data.www-data /var/log/fod
cp urls.py.dist urls.py
cd ..
## System configuration
Apache operates as a gunicorn Proxy with WSGI and Shibboleth modules
enabled. Depending on the setup the apache configuration may vary:
a2enmod rewrite
a2enmod proxy
a2enmod ssl
a2enmod proxy_http
If shibboleth is to be used:
apt-get install libapache2-mod-shib2
a2enmod shib2
Now it is time to configure beanstalk, gunicorn, celery and apache.
### beanstalkd
Enable beanstalk by editting /etc/default/beanstalkd:
vim /etc/default/beanstalkd
Uncomment the line **START=yes** to enable beanstalk
Start beanstalkd:
service beanstalkd start
### gunicorn.d
Create and edit /etc/gunicorn.d/fod:
vim /etc/gunicorn.d/fod
FoD is served via gunicorn and is then proxied by Apache. If the above
directory conventions have been followed so far, then your configuration
should be:
CONFIG = {
'mode': 'django',
'working_dir': '/srv/flowspy',
'args': (
'--bind=127.0.0.1:8081',
'--workers=1',
'--worker-class=egg:gunicorn#gevent',
'--timeout=30',
'--debug',
'--log-level=debug',
'--log-file=/var/log/gunicorn/fod.log',
),
}
### celeryd
Celery is used over beanstalkd to apply firewall rules in a serial
manner so that locks are avoided on the flowspec capable device. In our
setup celery runs via django. That is why the python-django-celery
package was installed.
Create the celeryd daemon at /etc/init.d/celeryd **if it does not
already exist**:
vim /etc/init.d/celeryd
The configuration should be:
#!/bin/sh -e
# ============================================
# celeryd - Starts the Celery worker daemon.
# ============================================
#
# :Usage: /etc/init.d/celeryd {start|stop|force-reload|restart|try-restart|status}
# :Configuration file: /etc/default/celeryd
#
# See http://docs.celeryq.org/en/latest/cookbook/daemonizing.html#init-script-celeryd
### BEGIN INIT INFO
# Provides: celeryd
# Required-Start: $network $local_fs $remote_fs
# Required-Stop: $network $local_fs $remote_fs
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: celery task worker daemon
# Description: Starts the Celery worker daemon for a single project.
### END INIT INFO
#set -e
DEFAULT_PID_FILE="/var/run/celery/%n.pid"
DEFAULT_LOG_FILE="/var/log/celery/%n.log"
DEFAULT_LOG_LEVEL="INFO"
DEFAULT_NODES="celery"
DEFAULT_CELERYD="-m celery.bin.celeryd_detach"
ENABLED="false"
[ -r "$CELERY_DEFAULTS" ] && . "$CELERY_DEFAULTS"
[ -r /etc/default/celeryd ] && . /etc/default/celeryd
if [ "$ENABLED" != "true" ]; then
echo "celery daemon disabled - see /etc/default/celeryd."
exit 0
fi
CELERYD_PID_FILE=${CELERYD_PID_FILE:-${CELERYD_PIDFILE:-$DEFAULT_PID_FILE}}
CELERYD_LOG_FILE=${CELERYD_LOG_FILE:-${CELERYD_LOGFILE:-$DEFAULT_LOG_FILE}}
CELERYD_LOG_LEVEL=${CELERYD_LOG_LEVEL:-${CELERYD_LOGLEVEL:-$DEFAULT_LOG_LEVEL}}
CELERYD_MULTI=${CELERYD_MULTI:-"celeryd-multi"}
CELERYD=${CELERYD:-$DEFAULT_CELERYD}
CELERYCTL=${CELERYCTL:="celeryctl"}
CELERYD_NODES=${CELERYD_NODES:-$DEFAULT_NODES}
export CELERY_LOADER
if [ -n "$2" ]; then
CELERYD_OPTS="$CELERYD_OPTS $2"
fi
CELERYD_LOG_DIR=`dirname $CELERYD_LOG_FILE`
CELERYD_PID_DIR=`dirname $CELERYD_PID_FILE`
if [ ! -d "$CELERYD_LOG_DIR" ]; then
mkdir -p $CELERYD_LOG_DIR
fi
if [ ! -d "$CELERYD_PID_DIR" ]; then
mkdir -p $CELERYD_PID_DIR
fi
# Extra start-stop-daemon options, like user/group.
if [ -n "$CELERYD_USER" ]; then
DAEMON_OPTS="$DAEMON_OPTS --uid=$CELERYD_USER"
chown "$CELERYD_USER" $CELERYD_LOG_DIR $CELERYD_PID_DIR
fi
if [ -n "$CELERYD_GROUP" ]; then
DAEMON_OPTS="$DAEMON_OPTS --gid=$CELERYD_GROUP"
chgrp "$CELERYD_GROUP" $CELERYD_LOG_DIR $CELERYD_PID_DIR
fi
if [ -n "$CELERYD_CHDIR" ]; then
DAEMON_OPTS="$DAEMON_OPTS --workdir=\"$CELERYD_CHDIR\""
fi
check_dev_null() {
if [ ! -c /dev/null ]; then
echo "/dev/null is not a character device!"
exit 1
fi
}
export PATH="${PATH:+$PATH:}/usr/sbin:/sbin"
stop_workers () {
$CELERYD_MULTI stop $CELERYD_NODES --pidfile="$CELERYD_PID_FILE"
}
start_workers () {
$CELERYD_MULTI start $CELERYD_NODES $DAEMON_OPTS \
--pidfile="$CELERYD_PID_FILE" \
--logfile="$CELERYD_LOG_FILE" \
--loglevel="$CELERYD_LOG_LEVEL" \
--cmd="$CELERYD" \
$CELERYD_OPTS
}
restart_workers () {
$CELERYD_MULTI restart $CELERYD_NODES $DAEMON_OPTS \
--pidfile="$CELERYD_PID_FILE" \
--logfile="$CELERYD_LOG_FILE" \
--loglevel="$CELERYD_LOG_LEVEL" \
--cmd="$CELERYD" \
$CELERYD_OPTS
}
case "$1" in
start)
check_dev_null
start_workers
;;
stop)
check_dev_null
stop_workers
;;
reload|force-reload)
echo "Use restart"
;;
status)
$CELERYCTL status $CELERYCTL_OPTS
;;
restart)
check_dev_null
restart_workers
;;
try-restart)
check_dev_null
restart_workers
;;
*)
echo "Usage: /etc/init.d/celeryd {start|stop|restart|try-restart|kill}"
exit 1
;;
esac
exit 0
#### celeryd configuration
celeryd requires a /etc/default/celeryd file to be in place. Thus we are
going to create this file (/etc/default/celeryd):
vim /etc/default/celeryd
Again if the directory conventions have been followed the file is (pay
attention to the CELERYD\_USER, CELERYD\_GROUP and change accordingly) :
# Default: false
ENABLED="true"
# Name of nodes to start, here we have a single node
CELERYD_NODES="w1"
# or we could have three nodes:
#CELERYD_NODES="w1 w2 w3"
# Where to chdir at start.
CELERYD_CHDIR="/srv/flowspy"
# How to call "manage.py celeryd_multi"
CELERYD_MULTI="python $CELERYD_CHDIR/manage.py celeryd_multi"
# How to call "manage.py celeryctl"
CELERYCTL="python $CELERYD_CHDIR/manage.py celeryctl"
# Extra arguments to celeryd
#CELERYD_OPTS="--time-limit=300 --concurrency=8"
CELERYD_OPTS="-E -B --schedule=/var/run/celery/celerybeat-schedule --concurrency=1 --soft-time-limit=180 --time-limit=1800"
# Name of the celery config module.
CELERY_CONFIG_MODULE="celeryconfig"
# %n will be replaced with the nodename.
CELERYD_LOG_FILE="/var/log/celery/fod_%n.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_USER="root"
CELERYD_GROUP="root"
# Name of the projects settings module.
export DJANGO_SETTINGS_MODULE="flowspy.settings"
### Apache
Apache proxies gunicorn. Things are more flexible here as you may follow
your own configuration and conventions. Create and edit
/etc/apache2/sites-available/fod. You should set \<server\_name\> and
\<admin\_mail\> along with your certificates. If under testing
environment, you can use the provided snakeoil certs. If you do not
intent to use Shibboleth delete or comment the corresponding
configuration parts inside **Shibboleth configuration** :
vim /etc/apache2/sites-available/fod
Again if the directory conventions have been followed the file should
be:
<VirtualHost *:80>
ServerAdmin webmaster@localhost
ServerName fod.example.com
DocumentRoot /var/www
ErrorLog ${APACHE_LOG_DIR}/fod_error.log
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel debug
CustomLog ${APACHE_LOG_DIR}/fod_access.log combined
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule ^/(.*) https://fod.example.com/$1 [L,R]
</VirtualHost>
<VirtualHost *:443>
ServerName fod.example.com
ServerAdmin webmaster@localhost
ServerSignature On
SSLEngine on
SSLCertificateFile /etc/ssl/certs/fod.example.com.crt
SSLCertificateChainFile /etc/ssl/certs/example-chain.pem
SSLCertificateKeyFile /etc/ssl/private/fod.example.com.key
AddDefaultCharset UTF-8
IndexOptions +Charset=UTF-8
ShibConfig /etc/shibboleth/shibboleth2.xml
Alias /shibboleth-sp /usr/share/shibboleth
<Location /login>
AuthType shibboleth
ShibRequireSession On
ShibUseHeaders On
ShibRequestSetting entityID https://idp.example.com/idp/shibboleth
require valid-user
</Location>
# Shibboleth debugging CGI script
ScriptAlias /shibboleth/test /usr/lib/cgi-bin/shibtest.cgi
<Location /shibboleth/test>
AuthType shibboleth
ShibRequireSession On
ShibUseHeaders On
require valid-user
</Location>
<Location /Shibboleth.sso>
SetHandler shib
</Location>
# Shibboleth SP configuration
#SetEnv proxy-sendchunked
<Proxy *>
Order allow,deny
Allow from all
</Proxy>
SSLProxyEngine off
ProxyErrorOverride off
ProxyTimeout 28800
ProxyPass /static !
ProxyPass /shibboleth !
ProxyPass /Shibboleth.sso !
ProxyPass / http://localhost:8081/ retry=0
ProxyPassReverse / http://localhost:8081/
Alias /static /srv/flowspy/static
LogLevel warn
Now, enable your site. You might want to disable the default site if fod
is the only site you host on your server:
a2dissite default
a2ensite fod
You are not far away from deploying FoD. When asked for a super user,
create one:
cd /srv/flowspy
python manage.py syncdb
python manage.py migrate longerusername
python manage.py migrate flowspec
python manage.py migrate djcelery
python manage.py migrate accounts
python manage.py migrate
If you have not changed the values of the PEER\_\*\_TABLE variables to
False and thus you are going for a default installation (that is
PEER\_\*\_TABLE variables are set to True) , then run:
python manage.py migrate peers
If however you have set the PEER\_\*\_TABLE variables to False and by
accident you have ran the command above, then you have to cleanup you
database manually by dropping the peer\* tables plus the techc\_email
table. For MySQL the command is:
DROP TABLE `peer`, `peer_networks`, `peer_range`, `peer_techc_emails`, techc_email;
Restart, gunicorn and apache:
service gunicorn restart && service apache2 restart
doc/installation/generic.md 0 → 100644
+ 131
− 0
View file @ 8ec6776c
# Installing Flowspy
This guide provides general information about the installation of Flowspy. In case you use Debian Wheezy or Red Hat Linux, we provide detailed instructions for the installation.
Also it assumes that installation is carried out in `/srv/flowspy`
directory. If other directory is to be used, please change the
corresponding configuration files. It is also assumed that the `root` user
will perform every action.
## Requirements
### System Requirements
In order to install FoD properly, make sure the following software is installed on your computer.
- apache 2
- memcached
- libapache2-mod-proxy-html
- gunicorn
- beanstalkd
- mysql
- python
- pip
- libxml2-dev
- libxslt-dev
- gcc
- python-dev
### Pip
In order to install the required python packages for Flowspy you can use pip:
pip install -r requirements.txt
### Create a database
If you are using mysql, you should create a database:
mysql -u root -p -e 'create database fod'
### Download Flowspy
You can download Fod from GRNETs github repository. Then you have to unzip the file and place it under /srv.
cd /tmp
wget https://github.com/grnet/flowspy/archive/v1.2.zip
unzip v1.2.zip
mv flowspy-1.2 /srv/flowspy/
### Copy dist files
cd /srv/flowspy/flowspy
cp settings.py.dist settings.py
cp urls.py.dist urls.py
### Device Configuration
Flowspy generates and commits flowspec rules to a
device via netconf. You have to create an account
with rw access to flowspec and set these credentials
in settings.py. See Configuration for details.
### Adding some default data
Into `/srv/flowspy` you will notice that there is a directory called `initial_data`. In there, there is a file called `fixtures_manual.xml` which contains some default static pages for django's flatpages app. In this file we have placed 4 flatpages (2 for Greek, 2 for English) with Information and Terms of Service about the service. To import the flatpages, run from `/srv/flowspy`:
./manage.py loaddata initial_data/fixtures_manual.xml
### Beanstalkd
Just start beanstalk already!
service beanstalkd start
### Apache2
Apache proxies gunicorn. Things are more flexible here as you may follow your own configuration and conventions.
#### Example config
Here is an example configuration.
<VirtualHost *:80>
ServerName fod.example.org
DocumentRoot /var/www
ErrorLog /var/log/httpd/fod_error.log
LogLevel debug
CustomLog /var/log/httpd/fod_access.log combined
RewriteEngine On
RewriteRule ^/(.*) https://fod.example.com/$1 [L,R]
</VirtualHost>
<VirtualHost *:443>
SSLEngine on
SSLProtocol TLSv1
SSLCertificateFile /home/local/GEANT/dante.spatharas/filename.crt
SSLCertificateKeyFile /home/local/GEANT/dante.spatharas/filename.key
SSLCACertificateFile /home/local/GEANT/dante.spatharas/filename.crt
Alias /static /srv/flowspy/static
AddDefaultCharset UTF-8
IndexOptions +Charset=UTF-8
#SSLProxyEngine off
ProxyErrorOverride off
ProxyTimeout 28800
ProxyPass /static !
ProxyPass / http://localhost:8080/ retry=0
ProxyPassReverse / http://localhost:8080/
</VirtualHost>
`Important!` You have to comment out/disable the default `Virtualhost` defined on line 74 until the end of this block at `/etc/httpd/conf.d/ssl.conf `.
### Gunicorn
FoD is served via gunicorn and is then proxied by Apache. If the above
directory conventions have been followed so far, then your configuration
should be:
CONFIG = {
'mode': 'django',
'working_dir': '/srv/flowspy',
'args': (
'--bind=127.0.0.1:8081',
'--workers=1',
'--worker-class=egg:gunicorn#gevent',
'--timeout=30',
'--debug',
'--log-level=debug',
'--log-file=/var/log/gunicorn/fod.log',
),
}
doc/installation/redhat.md 0 → 100644
+ 485
− 0
View file @ 8ec6776c
# Fod v1.1.1 Installation Documentation
The following document describes the installation process of Firewall On Demand
on a redhat 6.5 machine with linux 2.6.32-431.17.1.el6.x86_64.
## Step 1: Installing Requirements
The system must have the following packages installed in order for fod to work:
yum install python-pip libmysqlclient-dev mysql-devel git python-setuptools gcc mysql-devel.x84_64 python-devel libevent-devel libxslt-devel libxml2-devel mysql-server memcached httpd mod_ssl beanstalkd
Then, the next step is to install specific python packages for fod. These packages can be installed via pip (python package manager):
pip install Django==1.4.5 MySQL-python==1.2.3 PyYAML==3.10 South==0.7.5 amqplib==1.0.2 anyjson==0.3.1 argparse==1.2.1 celery==2.5.3 cl==0.0.3 django-celery==2.5.5 django-picklefield==0.2.1 django-registration==0.8 django-tinymce==1.5 gevent==0.13.6 greenlet==0.3.1 gunicorn==0.14.5 ipaddr==2.1.10 kombu==2.1.8 lxml==3.4.2 mailer==0.7 ncclient==0.4.3 paramiko==1.7.7.1 pycrypto==2.6 pyparsing==1.5.6 python-dateutil==1.5 python-memcached==1.48 wsgiref==0.1.2
`Important!`
There is one package that does not exist in pypi. Its name is Nxpy.
One can install nxpy from the official GRNETs code repository with the following command:
pip install git+https://code.grnet.gr/git/nxpy
## Step2: Configuring Requirements
FoD Requires the following components to be set up and configured before the installation of FoD:
- Mysql
- A router with flowspec enabled and a user with rw permissions to flowspec and netconf access (port 830) to the device.
For now only mysql can be configured:
### Mysql
In order to configure mysql one has to type the following commands:
# service mysqld restart
# mysqladmin -u <user> password <type_a_strong_password>
# mysql -u root -p
> CREATE DATABASE fod;
> exit
After this action the database will accept connections to database `fod` from the root user with the password you typed above.
### Router
Configuring the router in order to accept connections via netconf for a specific user is mentioned just for the sake of completeness.
## Step3: Actuall Installation of FoD
### Downloading and installing
You can download Fod v1.1.1 from GRNETs github repository. Then you have to unzip the file and place it under /srv.
# cd /tmp
# wget https://github.com/grnet/flowspy/archive/v1.1.1.zip
# unzip v1.1.1.zip
# mv flowspy-1.1.1 /srv/flowspy/
### Additional directory creations
We have to create manually the root directory for the logs:
mkdir /var/log/fod
chown apache:apache /var/log/fod
## Step4: Patching files for RedHat
We have noticed that some changes must be made for FoD to work under RedHat.
### Patch the `manage.py` file
We have to change `/srv/flowspy/manage.py` file, and make it look like this:
import os
import sys
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "flowspy.settings")
from gevent import monkey
if not 'celery' in sys.argv and not 'celeryd' in sys.argv:
monkey.patch_all()
from django.core.management import execute_from_command_line
execute_from_command_line(sys.argv)
Actually we have just added three new lines of code, but make sure the `manage.py` file looks like this.
### Patch python-tinymce:
We now have to fix a bug from a python package in order to get FoD up and running.
Open
/usr/lib/python2.6/site-packages/tinymce/widgets.py
you have to replace:
from django.forms.util import smart_unicode
to
from django.utils.encoding import smart_unicode
### Syncing the database
To create all the tables needed by FoD we have to run the following commands:
cd /srv/flowspy
./manage.py syncdb --noinput
./manage.py migrate
### Creating a superuser
A superuser can be added by using the following command from `/srv/flowspy/`:
./manage.py createsuperuser
### Adding some default data
Into `/srv/flowspy` you will notice that there is a directory called `initial_data`. In there, there is a file called `fixtures_manual.xml` which contains some default static pages for django's flatpages app. In this file we have placed 4 flatpages (2 for Greek, 2 for English) with Information and Terms of Service about the service. To import the flatpages, run from `/srv/flowspy`:
./manage.py loaddata initial_data/fixtures_manual.xml
#### Default Application Site:
Django uses the [Sites Framework](https://docs.djangoproject.com/en/1.4/ref/contrib/sites/) and needs at least a site declared in the database in order to work. So we have to enter the required data in the db. To do that we have to create a file called `sites.json` under `/srv/flowspy/initial_data/`. This file should contain:
[
{
"pk": 1,
"model": "sites.site",
"fields": {
"domain": "my-domain.com",
"name": "my-domain.com" # its a display name for the domain
}
}
]
so please enter the name of the domain you would like. Then load the contents of this file to the database:
./manage.py loaddata initial_data/sites.json
Its crucial to load this file and load it to the database cause it wont work otherwise.
## Step5: Configuring all the other dependencies
### Beanstalkd
Just start beanstalk already!
service beanstalkd start
### Celery
Celery is a distributed task queue, which helps FoD run some async tasks, like applying a flowspec rule to a router.
`Note` In order to check if celery runs or even debug it, you can run:
./manage.py celeryd --loglevel=debug
#### Celery daemon
In order to be able to run celery as a daemon you have to create the celeryd daemon at /etc/init.d/celeryd if it does not already exist:
vim /etc/init.d/celeryd
The configuration should be:
#!/bin/sh -e
# ============================================
# celeryd - Starts the Celery worker daemon.
# ============================================
#
# :Usage: /etc/init.d/celeryd {start|stop|force-reload|restart|try-restart|status}
# :Configuration file: /etc/default/celeryd
#
# See http://docs.celeryq.org/en/latest/cookbook/daemonizing.html#init-script-celeryd
### BEGIN INIT INFO
# Provides: celeryd
# Required-Start: $network $local_fs $remote_fs
# Required-Stop: $network $local_fs $remote_fs
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: celery task worker daemon
# Description: Starts the Celery worker daemon for a single project.
### END INIT INFO
#set -e
DEFAULT_PID_FILE="/var/run/celery/%n.pid"
DEFAULT_LOG_FILE="/var/log/celery/%n.log"
DEFAULT_LOG_LEVEL="INFO"
DEFAULT_NODES="celery"
DEFAULT_CELERYD="-m celery.bin.celeryd_detach"
ENABLED="false"
[ -r "$CELERY_DEFAULTS" ] && . "$CELERY_DEFAULTS"
[ -r /etc/default/celeryd ] && . /etc/default/celeryd
if [ "$ENABLED" != "true" ]; then
echo "celery daemon disabled - see /etc/default/celeryd."
exit 0
fi
CELERYD_PID_FILE=${CELERYD_PID_FILE:-${CELERYD_PIDFILE:-$DEFAULT_PID_FILE}}
CELERYD_LOG_FILE=${CELERYD_LOG_FILE:-${CELERYD_LOGFILE:-$DEFAULT_LOG_FILE}}
CELERYD_LOG_LEVEL=${CELERYD_LOG_LEVEL:-${CELERYD_LOGLEVEL:-$DEFAULT_LOG_LEVEL}}
CELERYD_MULTI=${CELERYD_MULTI:-"celeryd-multi"}
CELERYD=${CELERYD:-$DEFAULT_CELERYD}
CELERYCTL=${CELERYCTL:="celeryctl"}
CELERYD_NODES=${CELERYD_NODES:-$DEFAULT_NODES}
export CELERY_LOADER
if [ -n "$2" ]; then
CELERYD_OPTS="$CELERYD_OPTS $2"
fi
CELERYD_LOG_DIR=`dirname $CELERYD_LOG_FILE`
CELERYD_PID_DIR=`dirname $CELERYD_PID_FILE`
if [ ! -d "$CELERYD_LOG_DIR" ]; then
mkdir -p $CELERYD_LOG_DIR
fi
if [ ! -d "$CELERYD_PID_DIR" ]; then
mkdir -p $CELERYD_PID_DIR
fi
# Extra start-stop-daemon options, like user/group.
if [ -n "$CELERYD_USER" ]; then
DAEMON_OPTS="$DAEMON_OPTS --uid=$CELERYD_USER"
chown "$CELERYD_USER" $CELERYD_LOG_DIR $CELERYD_PID_DIR
fi
if [ -n "$CELERYD_GROUP" ]; then
DAEMON_OPTS="$DAEMON_OPTS --gid=$CELERYD_GROUP"
chgrp "$CELERYD_GROUP" $CELERYD_LOG_DIR $CELERYD_PID_DIR
fi
if [ -n "$CELERYD_CHDIR" ]; then
DAEMON_OPTS="$DAEMON_OPTS --workdir=\"$CELERYD_CHDIR\""
fi
check_dev_null() {
if [ ! -c /dev/null ]; then
echo "/dev/null is not a character device!"
exit 1
fi
}
export PATH="${PATH:+$PATH:}/usr/sbin:/sbin"
stop_workers () {
$CELERYD_MULTI stop $CELERYD_NODES --pidfile="$CELERYD_PID_FILE"
}
start_workers () {
$CELERYD_MULTI start $CELERYD_NODES $DAEMON_OPTS \
--pidfile="$CELERYD_PID_FILE" \
--logfile="$CELERYD_LOG_FILE" \
--loglevel="$CELERYD_LOG_LEVEL" \
--cmd="$CELERYD" \
$CELERYD_OPTS
}
restart_workers () {
$CELERYD_MULTI restart $CELERYD_NODES $DAEMON_OPTS \
--pidfile="$CELERYD_PID_FILE" \
--logfile="$CELERYD_LOG_FILE" \
--loglevel="$CELERYD_LOG_LEVEL" \
--cmd="$CELERYD" \
$CELERYD_OPTS
}
case "$1" in
start)
check_dev_null
start_workers
;;
stop)
check_dev_null
stop_workers
;;
reload|force-reload)
echo "Use restart"
;;
status)
$CELERYCTL status $CELERYCTL_OPTS
;;
restart)
check_dev_null
restart_workers
;;
try-restart)
check_dev_null
restart_workers
;;
*)
echo "Usage: /etc/init.d/celeryd {start|stop|restart|try-restart|kill}"
exit 1
;;
esac
exit 0
#### Configuring Celeryd
Celeryd requires a /etc/default/celeryd file to be in place. Thus we are going to create this file (/etc/default/celeryd):
vim /etc/default/celeryd
The configuration should be
# Default: false
ENABLED="true"
# Name of nodes to start, here we have a single node
CELERYD_NODES="w1"
# or we could have three nodes:
#CELERYD_NODES="w1 w2 w3"
# Where to chdir at start.
CELERYD_CHDIR="/srv/flowspy"
# How to call "manage.py celeryd_multi"
CELERYD_MULTI="python $CELERYD_CHDIR/manage.py celeryd_multi"
# How to call "manage.py celeryctl"
CELERYCTL="python $CELERYD_CHDIR/manage.py celeryctl"
# Extra arguments to celeryd
#CELERYD_OPTS="--time-limit=300 --concurrency=8"
CELERYD_OPTS="-E -B --schedule=/var/run/celery/celerybeat-schedule --concurrency=1 --soft-time-limit=180 --time-limit=1800"
# Name of the celery config module.
CELERY_CONFIG_MODULE="celeryconfig"
# %n will be replaced with the nodename.
CELERYD_LOG_FILE="/var/log/celery/fod_%n.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_USER="root"
CELERYD_GROUP="root"
# Name of the projects settings module.
export DJANGO_SETTINGS_MODULE="flowspy.settings"
### Apache (Httpd)
Apache proxies gunicorn. Things are more flexible here as you may follow your own configuration and conventions.
#### Sites enabled
Add this line at the bottom of `/etc/httpd/conf/httpd.conf`
include sites_enabled
Create the directory `sites_enabled` under `/etc/httpd/` and create and edit `/etc/httpd/sites-enabled/flowspy.conf`. You should set <server_name> and <admin_mail> along with your certificates. If under testing environment, you can use the provided snakeoil certs. If you do not intent to use Shibboleth delete or comment the corresponding configuration parts inside Shibboleth configuration.
<VirtualHost *:80>
ServerName fod.example.org
DocumentRoot /var/www
ErrorLog /var/log/httpd/fod_error.log
LogLevel debug
CustomLog /var/log/httpd/fod_access.log combined
RewriteEngine On
RewriteRule ^/(.*) https://fod.example.com/$1 [L,R]
</VirtualHost>
<VirtualHost *:443>
SSLEngine on
SSLProtocol TLSv1
SSLCertificateFile /home/local/GEANT/dante.spatharas/filename.crt
SSLCertificateKeyFile /home/local/GEANT/dante.spatharas/filename.key
SSLCACertificateFile /home/local/GEANT/dante.spatharas/filename.crt
Alias /static /srv/flowspy/static
AddDefaultCharset UTF-8
IndexOptions +Charset=UTF-8
#SSLProxyEngine off
ProxyErrorOverride off
ProxyTimeout 28800
ProxyPass /static !
ProxyPass / http://localhost:8080/ retry=0
ProxyPassReverse / http://localhost:8080/
</VirtualHost>
`Important!` You have to comment out/disable the default `Virtualhost` defined on line 74 until the end of this block at `/etc/httpd/conf.d/ssl.conf `.
### Gunicorn
In order for Gunicorn to work properly, you have to create `/etc/init.d/gunicorn` with the following content:
#!/bin/sh
### BEGIN INIT INFO
# Provides: gunicorn
# Required-Start: $all
# Required-Stop: $all
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: starts the gunicorn server
# Description: starts gunicorn using start-stop-daemon
### END INIT INFO
# Gunicorn init.d script for redhat/centos
# Written originally by Wojtek 'suda' Siudzinski <admin@suda.pl>
# Adapted to redhat/centos by Daniel Lemos <xspager@gmail.com>
# Gist: https://gist.github.com/1511911
# Original: https://gist.github.com/748450
#
# Sample config (/etc/gunicorn/gunicorn.conf):
#
# SERVERS=(
# 'server_name socket_or_url project_path number_of_workers'
# )
#RUN_AS='apache'
RUN_AS='root'
#
# WARNING: user $RUN_AS must have +w on /var/run/gunicorn
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/bin/gunicorn_django
LOGGING=--debug
NAME=flowspy
DESC=gunicorn
SERVER="$2"
test -x $DAEMON || exit 0
# Source function library.
. /etc/rc.d/init.d/functions
# Source networking configuration.
. /etc/sysconfig/network
# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0
if [ -f /etc/gunicorn.d ] ; then
. /etc/gunicorn.d/*
fi
if [ ! -d /var/run/gunicorn ]; then
mkdir /var/run/gunicorn
fi
start () {
daemon --user $RUN_AS --pidfile /var/run/gunicorn/${data[0]}.pid $DAEMON /srv/flowspy $LOGGING -b 127.0.0.1:8080 -D -p /var/run/gunicorn/${data[0]}.pid --limit-request-fields=10000
return
}
stop () {
if [ -f /var/run/gunicorn/${data[0]}.pid ]; then
echo "Killing ${data[0]}"
kill $(cat /var/run/gunicorn/${data[0]}.pid)
fi
}
case "$1" in
start)
echo "Starting $DESC"
start
;;
stop)
echo "Stopping $DESC"
stop
;;
restart)
echo "Restarting $DESC"
stop
sleep 1
start
;;
*)
N=/etc/init.d/$NAME
echo "Usage: $N {start|stop|restart} [particular_server_to_restart]" >&2
exit 1
;;
esac
exit 0
flowspy/urls.py.dist
+ 19
− 1
View file @ 8ec6776c
from django.conf.urls import patterns, include, url from django.conf.urls import patterns, include, url
from django.views.generic.simple import direct_to_template from django.views.generic.simple import direct_to_template
# Uncomment the next two lines to enable the admin:
from django.contrib import admin from django.contrib import admin
from rest_framework import routers
from flowspec.viewsets import (
RouteViewSet,
PortViewSet,
ThenActionViewSet,
FragmentTypeViewSet,
MatchProtocolViewSet,
)
admin.autodiscover() admin.autodiscover()
# Routers provide an easy way of automatically determining the URL conf.
router = routers.DefaultRouter()
router.register(r'routes', RouteViewSet, base_name='route')
router.register(r'ports', PortViewSet)
router.register(r'thenactions', ThenActionViewSet)
router.register(r'fragmentypes', FragmentTypeViewSet)
router.register(r'matchprotocol', MatchProtocolViewSet)
urlpatterns = patterns( urlpatterns = patterns(
'', '',
...@@ -13,6 +29,7 @@ urlpatterns = patterns( ...@@ -13,6 +29,7 @@ urlpatterns = patterns(
url(r'^overview_ajax/?$', 'flowspec.views.overview_routes_ajax', name="overview-ajax"), url(r'^overview_ajax/?$', 'flowspec.views.overview_routes_ajax', name="overview-ajax"),
url(r'^dashboard/?$', 'flowspec.views.dashboard', name="dashboard"), url(r'^dashboard/?$', 'flowspec.views.dashboard', name="dashboard"),
url(r'^profile/?$', 'flowspec.views.user_profile', name="user-profile"), url(r'^profile/?$', 'flowspec.views.user_profile', name="user-profile"),
url(r'^profile/token/$', 'accounts.views.generate_token', name="user-profile-token"),
url(r'^add/?$', 'flowspec.views.add_route', name="add-route"), url(r'^add/?$', 'flowspec.views.add_route', name="add-route"),
url(r'^addport/?$', 'flowspec.views.add_port', name="add-port"), url(r'^addport/?$', 'flowspec.views.add_port', name="add-port"),
url(r'^edit/(?P<route_slug>[\w\-]+)/$', 'flowspec.views.edit_route', name="edit-route"), url(r'^edit/(?P<route_slug>[\w\-]+)/$', 'flowspec.views.edit_route', name="edit-route"),
...@@ -38,4 +55,5 @@ urlpatterns = patterns( ...@@ -38,4 +55,5 @@ urlpatterns = patterns(
{'template_name': 'overview/login.html'}, name="altlogin" {'template_name': 'overview/login.html'}, name="altlogin"
), ),
url(r'^overview/?$', 'flowspec.views.overview', name="overview"), url(r'^overview/?$', 'flowspec.views.overview', name="overview"),
url(r'^api/', include(router.urls)),
) )
mkdocs.yml 0 → 100644
+ 12
− 0
View file @ 8ec6776c
site_name: Firewall On Demand
repo_url: https://github.com/grnet/flowspy/
docs_dir: doc
site_author: Stavros Kroustouris
theme: readthedocs
pages:
- 'Introduction': 'index.md'
- 'Installation':
- 'Generic': 'installation/generic.md'
- 'Debian': 'installation/debian_wheezy.md'
- 'Red Hat': 'installation/redhat.md'
- 'Configuration': 'configuration.md'
requirements.txt
+ 1
− 0
View file @ 8ec6776c
...@@ -15,3 +15,4 @@ pyparsing==1.5.6 ...@@ -15,3 +15,4 @@ pyparsing==1.5.6
python-dateutil==1.5 python-dateutil==1.5
python-memcached==1.48 python-memcached==1.48
djangorestframework==2.3.14 djangorestframework==2.3.14
git+https://code.grnet.gr/git/nxpy
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment