EduGAIN Connectivity Check Service - ECCS

1. Introduction
2. Check Performed on the IdPs
3. Limitations
4. Disable Checks
5. On-line interface
6. Requirements Hardware
7. Requirements Software
8. HOWTO Install and Configure
   • Python 3
     • CentOS 7 requirements
     • Debian requirements
     • Install
   • Install the Chromedriver
   • Install Google Chrome needed by Selenium
   • ECCS Script
     • Install
     • Configure
     • Execute
9. ECCS API Server (UWSGI)
   • Install
   • Configure
   • Utility
10. ECCS API JSON
11. User Interface
    • User interface parameters
12. Utility for web interface
13. Utility for developers
    • ECCS API Development Server
14. Authors

Introduction

The purpose of the eduGAIN Connectivity Check is to identify eduGAIN Identity Providers (IdP) that are not properly configured. In particular it checks if an IdP properly loads and consumes SAML2 metadata which contains the eduGAIN Service Providers (SP). The check results are published on the public eduGAIN Connectivity Check web page (https://technical.edugain.org/eccs). The main purpose is to increase the service overall quality and user experience of the eduGAIN interfederation service by making Federation and Identity Provider operators aware of configuration problems.

The check is performed by sending a SAML authentication request to each eduGAIN IdP and then follow the various HTTP redirects until the user login form. The expected result is a login form that allows users to authenticate themselves (typically with username/password) or an error message of some form. For those Identity Providers that return an error message, it can be assumed that they don't consume eduGAIN metadata properly or that they suffer from another configuration problem. There are some cases where the check will generate false positives, therefore IdPs can be excluded from checks as is described below.

The Identity Providers are checked once per day. Therefore, the login requests should not have any significant effect on the log entries/statistics of an Identity Provider. Also, no actual login is performed because the check cannot authenticate users due to missing username and password for the IdPs. Only Identity Providers are checked but not the Service Providers.

Check Performed on the IdPs

The check follows the steps:

1. It retrieves the eduGAIN IdPs from eduGAIN Operator Team database via a JSON interface

2. For each IdP, that hasn't been disabled manually by the eduGAIN Operations Team or dynamically by robots.txt (explained below) and that has a valid SSL certificate on its HTTP-Redirect Location, it performs an IdP-initiated SSO with SAML Authentication Request for two SP belonging two different NREN, members of eduGAIN interfederation, and for another randomnly generated fake SP. It expects to find the HTML form with username and password fields for the NREN SPs and an error or other result for the fake one. If an IdP uses frames on the Login page, the check follows only the first one on each nested pages. If an IdP uses HTTP Basic Authentication, the check searches '401 Unauthorized' string into the web page content returned or 401 HTTP Status Code from the request. Therefore, no complete login will happen at the Identity Provider because the check stops at the login page. The SAML Authentication Request is not signed. Therefore, an authentication request for any eduGAIN SP could be created because the SP's private key is not needed. The SPs HTTP-Post Assertion Consumer Service URLs used by the check are retrieved by sps-metadata.xml frile from the "input" directory. The 'validation' method used to validate the "sps-metadata.xml" is a deployer decision, but a solution is provided on the README-SPS-METADATA.md file.

3. If the check fails for an IdP the first time, a second attempt will be done at the end of all other checked IdP, before exit.

4. The results are kept for the last 7 days, but the deployers can increase it as they wish.

Limitations

There are some situations where the check cannot work reliably. In those cases it is possible to disable the check for a particular IdP. The so far known cases where the check might generate a false negative are:

• IdP does not support HTTPS with at least SSLv3 or TLS1 or newer (these IdPs are insecure anyway)
• IdP is part of a Hub & Spoke federation (some of them manually have to first approve eduGAIN SPs)
• IdP does not use web-based login form (e.g. Account Chooser Authentication or X.509 login)
• IdP does not allow requests coming from the ECCS servers: technical-test.edugain.org / technical.edugain.org
• IdP that uses more than one nested <iframe> inside the login page.

Disable Checks

In cases where an IdP cannot be reliably checked, it is necessary to create or enrich the robots.txt file on the IdP's web root dir with:

User-agent: ECCS
Disallow: /

If an IdP is not able to create its own robots.txt, it can be disabled by an eduGAIN Operation Team member by setting the dictionary IDPS_DISABLED_DICT into eccs_properties.py with a line like:

<idp-entity-id>':'<eccs-check-disabling-reason>

On-line interface

The tool uses following status for IdPs:

• ERROR (red):
  • The IdP's response contains an error or the web page is not returned due to a Timeout, Connection or IdP Generic error.
    • Timeout: considers those IdPs that do not load a standard username/password login page within 60 seconds.
    • Connection-Error: considers those IdPs that are not reachable due to a connection problem. View the "Page Source" content to discover which problem has the IdP.
    • IdP-Generic-Error: considers those IdPs that the returned web page does not contain a Login Form, but an unspecified error such as "An error occured". This kind of error has been seen on Microsoft ADFS based IdPs.
    • 403-Forbidden: considers those IdPs that return 403 Forbidden status code while opening their login page through a testing SP.
  • The IdP most likely does not consume the eduGAIN metadata correctly.
    • No-SP-Metadata-Error: considers those IdPs that return a message like "No return endpoint available for relying party" or "No metadata found for relying party" instead of the Login Page.
  • The HTTP SSL certificate used by the IdP is invalid:
    • SSL-Error
• OK (green):
  • The IdP most likely correctly consumes eduGAIN metadata and returns a valid username/password login page. This is no guarantee that login on this IdP works for all eduGAIN services but if the check is passed for an IdP, this is probable.
• UNKNOWN (yellow):
  • The IdP can't be checked because the returned Login Page content is not recognized or the Login Page is always returned, also for the fake SP.
    • Unable-To-Check: considers those IdPs that do not load a standard username/password login page and do not return messages like "No return endpoint available for relying party" or "No metadata found for relying party".
• DISABLED (white)
  • The IdP is excluded because it cannot be checked reliably. The "Page Source" column, when an entity is disabled, is populated with the reason for the disabling.
  • Disabled: considers those IdPs that are disabled from the check by an eduGAIN Operation Team member or "robots.txt" file.

Requirements Hardware

• OS: Debian 11, CentOS 7.8 (tested)
• HDD: 10 GB
• RAM: 4 GB
• CPU: >= 2 vCPU (suggested)
• ARCH: 64 Bit

Requirements Software

• Apache Server + WSGI
• Python 3 (tested with v3.9.1, v3.10.4)
• Selenium (tested with v4.1.3)
• Google Chrome Web Brower (tested with v91.0.4472.164, v100.0.4896.127, v101.0.4951.64)
• Chromedriver (tested with v91.0.4472.101, v100.0.4896.60)
• Git
• PHP

HOWTO Install and Configure

Download ECCS Repository

• cd $HOME ; git clone https://gitlab.geant.org/edugain/eccs.git

Install Python 3

CentOS 7 requirements

1. Update the system packages:

   • sudo yum -y update

2. Install the YUM utils:

   • sudo yum install yum-utils

3. Install needed packages to build python:

   • sudo yum-builddep python3

   If you want to use Python 3.10, you need OpenSSL >= 1.1.1:

   • sudo yum install openssl11 openssl11-devel
   • sudo mkdir /usr/local/openssl11
   • sudo cd /usr/local/openssl11
   • sudo ln -s /usr/lib64/openssl11 lib
   • sudo ln -s /usr/include/openssl11 include

4. Install Git:

   • sudo yum -y install git

Debian requirements

1. Update the system packages:

   • sudo apt update ; sudo apt upgrade -y

2. Install needed packages to build python3:

   • sudo apt-get build-dep python3 libffi-dev libssl-dev zlib-dev

3. Install Git:

   • sudo apt install git

Install

1. Download the last version of Python 3 from https://www.python.org/downloads/source/ into your home:

   • wget https://www.python.org/ftp/python/3.10.4/Python-3.10.4.tgz -O $HOME/eccs/Python-3.10.4.tgz

2. Extract Python source package:

   • cd $HOME/eccs/
   • tar xzf Python-3.10.4.tgz

3. Build Python from the source package:

   • Debian:

     • cd $HOME/eccs/Python-3.10.4
     • ./configure --prefix=$HOME/eccs/python
     • make

   • Centos 7:

     • cd $HOME/eccs/Python-3.10.4
     • ./configure --prefix=$HOME/eccs/python --with-openssl=/usr/local/openssl11
     • make

4. Install Python 3 under $HOME/eccs/python:

   • make install
   • $HOME/eccs/python/bin/python3 --version
   • $HOME/eccs/python/bin/python3 -c "import ssl; print (ssl.OPENSSL_VERSION)"

   This will install python3 under your $HOME/eccs/python directory.

5. Remove useless things:

   • rm -Rf $HOME/eccs/Python-3.10.4 $HOME/eccs/Python-3.10.4.tgz

Install Google Chrome needed by Selenium

• Debian (64 bit):

  • cd $HOME/eccs
  • sudo wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
  • sudo apt install ./google-chrome-stable_current_amd64.deb

• CentOS (64 bit):

  • cd $HOME/eccs
  • sudo wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm
  • sudo yum install ./google-chrome-stable_current_x86_64.rpm

Install the Chromedriver

1. Find out which version of Chromium you are using:

   • Debian 9 (stretch):
     • google-chrome -version => Google Chrome 100.0.4896.127
   • CentOS 7.8:
     • google-chrome -version => Google Chrome 100.0.4896.127

2. Take the Chrome version number, remove the last part, and append the result to URL "https://chromedriver.storage.googleapis.com/LATEST_RELEASE_". For example, with Chrome version 100.0.4896.127, you'd get a URL "https://chromedriver.storage.googleapis.com/LATEST_RELEASE_100.0.4896".

3. Use the URL created in the last step to discover the version of ChromeDriver to use. For example, the above URL will get your a file containing "100.0.4896.60".

4. Use the version number retrieved from the previous step to construct the URL to download ChromeDriver. With version 100.0.4896.60, the URL would be "https://chromedriver.storage.googleapis.com/index.html?path=100.0.4896.60/"

5. Download the Chromedriver and extract it with:

   • cd $HOME/eccs
   • wget https://chromedriver.storage.googleapis.com/100.0.4896.60/chromedriver_linux64.zip
   • unzip chromedriver_linux64.zip
   • rm chromedriver_linux64.zip google-chrome-stable_current_amd64.deb

Note: After the initial download, it is recommended that you occasionally go through the above process again to see if there are any bug fix releases.

ECCS Script

Install and Configure the Virtual Environment

• cd $HOME/eccs
• ./python/bin/python3 -m pip install virtualenv
• $HOME/eccs/python/bin/virtualenv --python=$HOME/eccs/python/bin/python3 eccs-venv
• $HOME/eccs/eccs-venv/bin/python3 -m pip install --upgrade pip
• source eccs-venv/bin/activate (deactivate to exit Virtualenv)
  • python3 -m pip install -r requirements.txt

Configure ECCS

1. Configure ECCS properties:

   • cp $HOME/eccs/eccs_properties.py.template $HOME/eccs/eccs_properties.py
   • vim $HOME/eccs/eccs_properties.py (and change it upon your needs)

2. Change PATH by adding the virtualenv Python bin dir:

   • CentOS:

     • vim $HOME/.bash_profile

     • Add the following lines at the tail:

       # set PATH for ECCS
       if [ -d "$HOME/eccs" ]; then
          PATH="$HOME/eccs/eccs-venv/bin:$PATH"
       fi

     • source $HOME/.bash_profile

   • Debian:

     • vim $HOME/.bash_profile

     • Add the following lines at the tail:

       # set PATH for ECCS
       if [ -d "$HOME/eccs" ]; then
          PATH="$HOME/eccs/eccs-venv/bin:$PATH"
       fi

     • source $HOME/.bash_profile

3. Configure the cron job that runs the ECCS script:

   • crontab -e

     SHELL=/bin/bash
     
     0 4 * * * /bin/bash $HOME/eccs/cleanAndRunEccs.sh > $HOME/eccs/logs/eccs-cron.log 2>&1

     The script takes about 2 hours to check 4666 IDPs, so its execution is suggested in the early morning,before the users start using the tool. The eccs-cron.log file will contains:

     • The execution time of the entire ECCS script
     • Each failed IdP checked again and their results
     • The result of the entire ECCS script

Execute

• cd $HOME/eccs
• ./cleanAndRunEccs.py (to run a full and clean check)
• ./runEccs.py (to run a full check on the existing inputs)
• ./runEccs.py --idp <IDP-ENTITYID> (to run check on a single IdP)
• ./runEccs.py --test (to run a full check without effects)
• ./runEccs.py --idp <IDP-ENTITYID> --test (to run check on a single IdP without effects)
• ./runEccs.py --idp <IDP-ENTITYID> --replace (to run check on a single IdP and replace, or add, a result)

If something prevent the good execution of the ECCS's check, the logs/failed-cmd.sh file will be not empty at the end of the execution.

The "--test" parameter will not change the result of ECCS, but will write the output on the logs/stdout_idp_YYYY-MM-DD.log,logs/stderr_idp_YYYY-MM-DD.log and logs/failed-cmd-idp.sh files if the argument "--test" will be used.

ECCS API Server (uWSGI)

Install

1. Install requirements:
   • Debian:
     • sudo apt-get install libpcre3 libpcre3-dev libapache2-mod-proxy-uwsgi build-essentials python3-dev unzip
   • CentOS:
     • sudo yum install mod_proxy_uwsgi unzip
     • Configure SElinux to enable ECCS:
       • sudo semanage fcontext -a -t httpd_sys_content_t $HOME/eccs/eccs.conf
       • sudo restorecon -v $HOME/eccs/eccs.conf
       • sudo semanage fcontext -a -t httpd_sys_content_t $HOME/eccs/html(/.*)?
       • sudo restorecon -R -v "$HOME/eccs/html/"
       • sudo setsebool -P httpd_can_network_connect 1

Configure

1. Add the systemd service to enable ECCS API:

   • cd $HOME/eccs
   • cp eccs.ini.template eccs.ini
   • cp eccs.service.template eccs.service
   • vim eccs.ini (and change "uid", "gid" and "base" values opportunely)
   • vim eccs.service (and change "User","Group","WorkingDirectory","RuntimeDirectory","ExecStart" values opportunely)
   • sudo cp $HOME/eccs/eccs.service /etc/systemd/system/eccs.service
   • sudo systemctl daemon-reload
   • sudo systemctl enable eccs.service
   • sudo systemctl start eccs.service

2. Configure Apache for ECCS web side:

   • Debian:
     • sudo cp $HOME/eccs/eccs-debian.conf /etc/apache2/conf-available/eccs.conf
     • sudo vim /etc/apache2/conf-available/eccs.conf (and change the file opportunely)
     • sudo a2enconf eccs.conf
     • sudo a2enmod proxy_uwsgi
     • sudo chgrp www-data $HOME ; sudo chmod g+rx $HOME (Apache needs permission to access the $HOME dir)
     • sudo systemctl restart apache2.service
   • CentOS:
     • sudo cp $HOME/eccs/eccs-centos.conf /etc/httpd/conf.d/eccs.conf
     • sudo chgrp apache $HOME ; sudo apache g+rx $HOME (Apache needs permission to access the $HOME dir)
     • sudo systemctl restart httpd.service

3. Restart API WSGI server each day before the ECCS script:

   • crontab -e

     SHELL=/bin/bash
     
     0 3 * * * /usr/bin/touch $HOME/eccs/eccs.ini

     This cron job must be executed prior to the ECCS script because it updates the date to the current day.

Utility

To perform a restart after an API change use the following command:

• touch $HOME/eccs/eccs.ini

ECCS API JSON

• /api/eccsresults (Return the results of the last check ready for ECCS web interface)
• /api/eccsresults?<parameter1>=<value1>&<parameter2>=<value2>:
  • date=2020-02-20 (select date)
  • idp=https://idp.example.org/idp/shibboleth (select a specific idp)
  • status= (select specific ECCS status)
    • 'OK'
    • 'ERROR'
    • 'DISABLED'
    • 'UNKNOWN'
  • check_result=
    • OK
    • Timeout
    • Connection-Error
    • IdP-Generic-Error
    • No-SP-Metadata-Error
    • SSL-Error
    • Unable-To-Check
    • 403-Forbidden
    • Disabled
  • reg_auth=https://reg.auth.example.org (select a specific Registration Authority)
  • format=simple (retrieve results in a simple format)
• /api/fedstats?<parameter1>=<value1>&<parameter2>=<value2>
  • reg_auth=https://reg.auth.example.org:

User interface

The eduGAIN Connectivity Check Service web page is available at (https://technical.edugain.org/eccs)

User interface parameters

Example:

https://technical.edugain.org/eccs?reg_auth=http://www.idem.garr.it/&check_result=SSL-Error

Utility for web interface

The available dates are provided by the first and the last file created into the output/ directory, remember to change its path into web/eccs.php file.

Clean old results

To clean the ECCS results from files older than last 7 days use (modify it on your needs):

• crontab -e

  SHELL=/bin/bash
  
  0 10 * * * /bin/bash $HOME/eccs/clean7daysOldFiles.sh > $HOME/eccs/logs/clean7daysOldFiles.log 2>&1  

  This cron job is useful to reduce the considered days selectable on the ECCS Web GUI calendar.

  It is suggested to configure it after the execution of ECCS script to get the hoped result.

Utility for developers

ECCS API Development Server

• cd $HOME/eccs ; ./api.py

Search files created on the current date

• cd $HOME/eccs
• find . -name *$(date +%Y-%m-%d)*

Delete files created on the current date

• cd $HOME/eccs
• rm -rf html/$(date +%Y-%m-%d) output/eccs_$(date +%Y-%m-%d).log logs/*_$(date +%Y-%m-%d).log

Authors

Original Author

• Marco Malavolti (marco.malavolti@garr.it)

GUI Developers

• Valentin Pocotilenco (valentin.pocotilenco@renam.md)