* [Inventory Provider](#inventory-provider) * [Overview](#overview) * [Configuration](#configuration) * [Running this module](#running-this-module) * [Protocol Specification](#protocol-specification) * [Backend (Redis) Storage Schema](#backend-redis-storage-schema) # Inventory Provider ## Overview This module implements a Flask-based webservice which provides some access to GÉANT network router inventory information. The webservice is communicates with clients over HTTP. Responses to valid requests are returned as JSON messages. The server will therefore return an error unless `application/json` is in the `Accept` request header field. HTTP communication and JSON grammar details are beyond the scope of this document. Please refer to [RFC 2616](https://tools.ietf.org/html/rfc2616) and www.json.org for more details. ## Configuration Several configuration files must be provided when launching the web service. The runtime-accessible filename of the first configuration file must be stored in the environment variable `SETTINGS_FILENAME`. The configuration file is python and defines a single variable. The following is an example: ```python INVENTORY_PROVIDER_CONFIG_FILENAME = "/somepath/config.json" ENABLE_TESTING_ROUTES = True ``` - `INVENTORY_PROVIDER_CONFIG_FILENAME`: [REQUIRED] Run-time accessible filename of a json file containing the server configuration parameters. This file must be formatted according to the following json schema: ```json { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "database_credentials": { "type": "object", "properties": { "hostname": {"type": "string"}, "dbname": {"type": "string"}, "username": {"type": "string"}, "password": {"type": "string"} }, "required": ["hostname", "dbname", "username", "password"], "additionalProperties": False } }, "type": "object", "properties": { "alarms-db": {"$ref": "#/definitions/database_credentials"}, "ops-db": {"$ref": "#/definitions/database_credentials"}, "oid_list.conf": {"type": "string"}, "ssh": { "type": "object", "properties": { "username": {"type": "string"}, "private-key": {"type": "string"}, "known-hosts": {"type": "string"} }, "required": ["private-key", "known-hosts"], "additionalProperties": False }, "redis": { "type": "object", "properties": { "hostname": {"type": "string"}, "port": {"type": "integer"} }, "required": ["hostname", "port"], "additionalProperties": False }, "junosspace": { "api": {"type": "string"}, "username": {"type": "string"}, "password": {"type": "string"} }, "infinera-dna": { "type": "array", "items": { "type": "object", "properties": { "name": {"type": "string"}, "address": {"type": "string"} }, "required": ["name", "address"], "additionalProperties": False } }, "coriant-tnms": { "type": "array", "items": { "type": "object", "properties": { "name": {"type": "string"}, "address": {"type": "string"} }, "required": ["name", "address"], "additionalProperties": False } } }, "required": [ "alarms-db", "ops-db", "oid_list.conf", "ssh", "redis", "junosspace", "infinera-dna", "coriant-tnms"], "additionalProperties": False } ``` - `ENABLE_TESTING_ROUTES`: [OPTIONAL (default value: False)] Flat (can be any value that evaluates to True) to enable routes to special utilities used for testing. *This must never be enabled in a production environment.* ## Running this module This module has been tested in the following execution environments: - As an embedded Flask application. For example, the application could be launched as follows: ```bash $ export FLASK_APP=app.py $ export SETTINGS_FILENAME=settings.cfg $ flask run ``` - As an Apache/`mod_wsgi` service. - Details of Apache and `mod_wsgi` configuration are beyond the scope of this document. - As a `gunicorn` wsgi service. - Details of `gunicorn` configuration are beyond the scope of this document. ## Protocol Specification The following resources can be requested from the webservice. ### resources Any non-empty responses are JSON formatted messages. * `/data/version` The response will be an object containing the module and protocol versions of the running server and will be formatted as follows: ```json { "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "properties": { "api": { "type": "string", "pattern": r'\d+\.\d+' }, "module": { "type": "string", "pattern": r'\d+\.\d+' } }, "required": ["api", "module"], "additionalProperties": False } ``` * /data/routers The response will be a list of router hostnames for which information is available and will be formatted as follows: ```json { "$schema": "http://json-schema.org/draft-07/schema#", "type": "array", "items": {"type": "string"} } ``` * /data/interfaces/*`hostname`* The response will be a list of information about the interfaces present on the requested host and will be formatted as follows: ```json { "$schema": "http://json-schema.org/draft-07/schema#", "type": "array", "items": { "type": "object", "properties": { "name": {"type": "string"}, "description": {"type": "string"}, "ipv4": { "type": "array", "items": {"type": "string"} }, "ipv6": { "type": "array", "items": {"type": "string"} } }, "required": ["name", "description", "ipv4", "ipv6"], "additionalProperties": False } } ``` * /data/snmp/*`hostname`* The response will be a list of information about the interfaces discovered through snmp queries on the requested host and will be formatted as follows: ```json { "$schema": "http://json-schema.org/draft-07/schema#", "type": "array", "items": { "type": "object", "properties": { "index": {"type": "string"}, "name": {"type": "string"} }, "required": ["index", "name"], "additionalProperties": False } } ``` * /data/bgp/*`hostname`* The response will be a list of information about the bgp peerings configured for the requested host and will be formatted as follows: ```json { "$schema": "http://json-schema.org/draft-07/schema#", "type": "array", "items": { "type": "object", "properties": { "description": {"type": "string"}, "as": { "type": "object", "properties": { "peer": { "type": "string", "pattern": r'^\d+$' }, "local": { "type": "string", "pattern": r'^\d+$' }, }, "required": ["peer", "local"], "additionalProperties": False }, }, "required": ["description", "as"], "additionalProperties": False } } ``` * /jobs/update This resource updates the inventory network data for juniper devices. The function completes asynchronously and a list of outstanding task id's is returned so the caller can use `/jobs/check-task-status` to determine when all jobs are finished. The response will be formatted as follows: ```json { "$schema": "http://json-schema.org/draft-07/schema#", "type": "array", "items": {"type": "string"} } ``` * /jobs/reload-router-config/<equipment-name> This resource updates the inventory network data for the identified juniper device. This function completes asynchronously and returns the same value as `/jobs/update`, except the return contains exactly one task id. * /jobs/check-task-status/<task-id>" This resource returns the current status of an asynchronous task started by `/jobs/update` or `jobs/reload-router-config`. The return value will be formatted as follows: ```json { "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "properties": { "id": {"type": "string"}, "status": {"type": "string"}, "exception": {"type": "boolean"}, "ready": {"type": "boolean"}, "success": {"type": "boolean"}, "result": {"type": "object"} }, "required": ["id", "status", "exception", "ready", "success"], "additionalProperties": False } ``` * /jobs/update-interface-statuses This resource updates the last known statuses of all interfaces extracted from the external inventory system, status information is retrieved from the alarms database * /classifier/juniper-link-info/*`source-equipment`*/*`source-interface`* The source-equipment is the equipment that causes the trap, not the NMS that sends it. The response will be an object containing the metadata, formatted according to the following schema: ```json { "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "definitions": { "ip-address": { "type": "string", "oneOf": [ {"pattern": r'^(\d+\.){3}\d+$'}, {"pattern": r'^([a-f\d]{4}:){7}[a-f\d]{4}$'} ] }, "ipv4-interface-address": { "type": "string", "pattern": r'^(\d+\.){3}\d+/\d+$' }, "ipv6-interface-address": { "type": "string", "pattern": r'^[a-f\d:]+/\d+$' }, "interface-info": { "type": "object", "properties": { "name": {"type": "string"}, "description": {"type": "string"}, "ipv4": { "type": "array", "items": {"$ref": "#/definitions/ipv4-interface-address"} }, "ipv6": { "type": "array", "items": {"$ref": "#/definitions/ipv6-interface-address"} } }, "required": ["name", "description", "ipv4", "ipv6"], "additionalProperties": False }, "service-info": { "type": "object", "properties": { "id": {"type": "integer"}, "name": {"type": "string"}, "status": { "type": "string", "enum": ["operational", "installed", "planned", "ordered"] }, "circuit_type": { "type": "string", "enum": ["path", "service", "l2circuit"] }, "service_type": {"type": "string"}, "project": {"type": "string"}, "equipment": {"type": "string"}, "other_end_equipment": {"type": "string"}, "port": {"type": "string"}, "other_end_port": {"type": "string"}, "logical_unit": { "oneOf": [ {"type": "integer"}, {"type": "string", "maxLength": 0} ] }, "other_end_logical_unit": { "oneOf": [ {"type": "integer"}, {"type": "string", "maxLength": 0} ] }, "manufacturer": { "type": "string", "enum": ["juniper", "coriant", "infinera", "cisco", "hewlett packard", "corsa", "graham smith uk ltd", "unknown", ""] }, "card_id": {"type": "string"}, "other_end_card_id": {"type": "string"}, "interface_name": {"type": "string"}, "other_end_interface_name": {"type": "string"} }, "additionalProperties": False } }, "type": "object", "properties": { "services": { "type": "array", "items": {"$ref": "#/definitions/service-info"} }, "interface": {"$ref": "#/definitions/interface-info"}, "related-services": { "type": "array", "items": {"$ref": "#/definitions/service-info"} } }, "additionalProperties": False } ``` * /classifier/peer-info/*`address`* The `address` parameter should be the ip address of a remote peer. If this address is found in the system then information about the interface is returned, otherwise 404 is returned (or 422 if the address can't be parsed) The response will be formatted according to the following syntax: ```json { "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "definitions": { "ip-address": { "type": "string", "oneOf": [ {"pattern": r'^(\d+\.){3}\d+$'}, {"pattern": r'^([a-f\d]{4}:){7}[a-f\d]{4}$'} ] }, "interface-address": { "type": "string", "oneOf": [ {"pattern": r'^(\d+\.){3}\d+/\d+$'}, {"pattern": r'^[a-f\d:]+/\d+$'} ] }, "vpn-rr-peer": { "type": "object", "properties": { "name": {"$ref": "#/definitions/ip-address"}, "description": {"type": "string"}, "peer-as": {"type": "integer"}, "router": {"type": "string"} }, "required": ["name", "description"], "additionalProperties": False }, "ix-public-peer": { "type": "object", "properties": { "name": {"$ref": "#/definitions/ip-address"}, "description": {"type": "string"}, "router": {"type": "string"}, "as": { "type": "object", "properties": { "local": {"type": "integer"}, "peer": {"type": "integer"}, }, "required": ["local", "peer"], "additionalProperties": False } }, "required": ["name", "description", "as"], "additionalProperties": False }, "ix-public-peer-list": { "type": "array", "items": {"$ref": "#/definitions/ip-address"} }, "ix-public-peer-info": { "type": "object", "properties": { "peer": {"$ref": "#/definitions/ix-public-peer"}, "group": {"$ref": "#/definitions/ix-public-peer-list"}, "router": {"$ref": "#/definitions/ix-public-peer-list"} }, "required": ["peer", "group", "router"], "additionalProperties": False }, "interface-info": { "type": "object", "properties": { "name": {"$ref": "#/definitions/ip-address"}, "interface address": { "$ref": "#/definitions/interface-address"}, "interface name": {"type": "string"}, "router": {"type": "string"} }, "required": [ "name", "interface address", "interface name", "router"], "additionalProperties": False }, "service-info": { "type": "object" }, "interface-lookup-info": { "type": "object", "properties": { "interface": {"$ref": "#/definitions/interface-info"}, "services": { "type": "array", "items": {"$ref": "#/definitions/service-info"} } } } }, "type": "object", "properties": { "ix-public-peer-info": {"$ref": "#/definitions/ix-public-peer-info"}, "vpn-rr-peer-info": {"$ref": "#/definitions/vpn-rr-peer"}, "interfaces": { "type": "array", "items": {"$ref": "#/definitions/interface-lookup-info"} } }, "additionalProperties": False } ``` * /classifier/infinera-lambda-info/*`source-equipment`*/*`source-interface`* The source-equipment is the equipment that causes the trap, not the NMS that sends it. The response will be an object containing the metadata, formatted according to the following schema: ```json { "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "definitions": { "service-info": { "type": "object", "properties": { "id": {"type": "integer"}, "name": {"type": "string"}, "status": { "type": "string", "enum": ["operational", "installed", "planned", "ordered"] }, "circuit_type": { "type": "string", "enum": ["path", "service", "l2circuit"] }, "service_type": {"type": "string"}, "project": {"type": "string"}, "equipment": {"type": "string"}, "other_end_equipment": {"type": "string"}, "port": {"type": "string"}, "other_end_port": {"type": "string"}, "logical_unit": { "oneOf": [ {"type": "integer"}, {"type": "string", "maxLength": 0} ] }, "other_end_logical_unit": { "oneOf": [ {"type": "integer"}, {"type": "string", "maxLength": 0} ] }, "manufacturer": { "type": "string", "enum": ["juniper", "coriant", "infinera", "cisco", "hewlett packard", "corsa", "graham smith uk ltd", "unknown", ""] }, "card_id": {"type": "string"}, "other_end_card_id": {"type": "string"}, "interface_name": {"type": "string"}, "other_end_interface_name": {"type": "string"} }, "additionalProperties": False }, "geant-lambdas": { "type": "object", "properties": { "id": {"type": "integer"}, "name": {"type": "string"}, "status": { "type": "string", "enum": ["operational", "installed", "planned", "ordered"] }, }, "additionalProperties": False } }, "type": "object", "properties": { "services": { "type": "array", "items": {"$ref": "#/definitions/service-info"} }, "geant-lambdas": { "type": "array", "items": {"$ref": "#/definitions/geant-lambdas"} }, }, "additionalProperties": False } ``` * /poller/interfaces/*`hostname`* The response will be the list of active interfaces on the router `hostname`. Each element of the returned list contains information necessary for setting up snmp throughput counter polling. The response will be formatted according to the following schema: ```json { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "circuit": { "type": "object", "properties": { "name": {"type": "string"}, "status": {"type": "string"}, "type": {"type": "string"}, "id": {"type": "integer"} }, "required": ["name", "status", "type", "id"], "additionalProperties": False } }, "type": "array", "items": { "type": "object", "properties": { "circuits": { "type": "array", "items": {"$ref": "#/definitions/circuit"} }, "description": {"type": "string"}, "name": {"type": "string"}, "snmp-index": {"type": "integer"} }, "required": ["circuits", "description", "name", "snmp-index"], "additionalProperties": False } } ``` ### Testing utilities The following routes are only available if the server was started with the `ENABLE_TESTING_ROUTES` flag. * `/testing/flushdb` This method erases all data in the backend redis database. * /testing/infinera-dna-addresses, /testing/coriant-tnmp-addresses, /testing/juniper-server-addresses` All of these resources return lists of source addresses of known senders of snmp traps. Responses will be formatted as follows: ```json { "$schema": "http://json-schema.org/draft-07/schema#", "type": "array", "items": {"type": "string"} } ``` ## Backend (Redis) Storage Schema * `netconf:<hostname>` * key example * `netconf:mx1.ams.nl.geant.net` * value format * cf. validation in `inventory_provider.juniper.load_config` * `snmp-interfaces:<hostname>` * key example: * `snmp-interfaces:mx1.lon2.uk.geant.net` * value schema TODO: this needs to be checked ... ```json { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "v4ifc": { "type": "object", "properties": { "v4Address": { "type": "string", "pattern": r'^(\d+\.){3}\d+$' }, "v4Mask": { "type": "string", "pattern": r'^(\d+\.){3}\d+$' }, "v4InterfaceName": {"type": "string"}, "index": { "type": "string", "pattern": r'^\d+$' } }, "required": [ "v4Address", "v4Mask", "v4InterfaceName", "index"], "additionalProperties": False }, "v6ifc": { "type": "object", "properties": { "v6Address": { "type": "string", "pattern": r'^[a-f\d:]+$' }, "v6Mask": { "type": "string", "pattern": r'^\d+$' }, "v6InterfaceName": {"type": "string"}, "index": { "type": "string", "pattern": r'^\d+$' } }, "required": [ "v6Address", "v6Mask", "v6InterfaceName", "index"], "additionalProperties": False } }, "type": "array", "items": { "anyOf": [ {"$ref": "#/definitions/v4ifc"}, {"$ref": "#/definitions/v6ifc"} ] } } ``` * `opsdb:interface_services:<equipment name>:<interface name>` * key examples * `opsdb:interface_services:mx1.ams.nl.geant.net:ae15.1103` * `opsdb:interface_services:Lab node 2:1-1/5` * value schema ```json { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "interface": { "type": "object", "properties": { "id": {"type": "integer"}, "name": {"type": "string"}, "status": { "type": "string", "enum": ["operational", "installed", "planned", "ordered"] }, "circuit_type": { "type": "string", "enum": ["path", "service", "l2circuit"] }, "service_type": {"type": "string"}, "project": {"type": "string"}, "equipment": {"type": "string"}, "port": { "anyOf": [ { "type": "string", "pattern": r'^[\d:]+$' }, { "type": "string", "enum": ['console'] } ] }, "logical_unit": { "anyOf": [ {"type": "integer"}, {"type": "string", "maxLength": 0} ] }, "manufacturer": { "type": "string", "enum": ["juniper", "coriant", "infinera", "cisco", "hewlett packard", "corsa", "graham smith uk ltd", "unknown", ""] }, "card_id": {"type": "string"}, "interface_name": {"type": "string"} }, "required": ["id", "name", "status", "circuit_type", "service_type", "project", "equipment", "port", "logical_unit", "manufacturer", "card_id", "interface_name"], "additionalProperties": False } }, "type": "object", "patternProperties": { "^.+$": { "type": "array", "items": {"$ref": "#/definitions/interface"} } } } ``` * `opsdb:location:<equipment name>` * key examples * `opsdb:location:mx1.ams.nl.geant.net` * `opsdb:location:taas.srv1.par.fr.geant.net` * `opsdb:location:A test EX 3-Oct` * value schema ```json { "$schema": "http://json-schema.org/draft-07/schema#", "type": "array", "items": { "type": "object", "properties": { "absid": {"type": "integer"}, "equipment_name": {"type": "string"}, "pop_name": {"type": "string"}, "pop_abbreviation": {"type": "string"}, "pop_site_id": {"type": "string"}, "country": {"type": "string"}, "longitude": {"type": "number"}, "latitude": {"type": "number"}, }, "required": ["absid", "equipment_name", "pop_name", "pop_abbreviation", "pop_site_id", "country", "longitude", "latitude"], "additionalProperties": False } } ``` * `opsdb:services:children:<circuit db id>` * `opsdb:services:parents:<circuit db id>` * key examples * `opsdb:services:children:12363` * `opsdb:services:parents:14407` * value schema TODO: this needs to be checked ... ```json { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "circuit_relationship": { "type": "object", "properties": { "parent_circuit": {"type": "string"}, "parent_circuit_id": {"type": "integer"}, "parent_circuit_status": { "type": "string", "enum": ["operational", "installed", "planned", "ordered"] }, "child_circuit": {"type": "string"}, "child_circuit_id": {"type": "integer"}, "child_circuit_status": { "type": "string", "enum": ["operational", "installed", "planned", "ordered"] } "circuit_group": {"type": "integer"}, }, "required": [ "parent_circuit", "parent_circuit_id", "parent_circuit_status", "child_circuit", "child_circuit_id", "child_circuit_status", "circuit_group" ], "additionalProperties": False } }, "type": "object", "patternProperties": { "^\d+$": { "type": "array", "items": {"$ref": "#/definitions/circuit_relationship"} } } } ``` * `alarmsdb:interface_status:<equipment name>:<interface name>` * key examples * `alarmsdb:interface_status:Lab node 1:1-1/3` * `alarmsdb:interface_status:mx1.ams.nl.geant.net:ae15.1500` * valid values: TODO: verify these values * `unknown` * `up` * `down` * `ix_public_peer:<address>` * key examples * `ix_public_peer:193.203.0.203` * `ix_public_peer:2001:07f8:00a0:0000:0000:5926:0000:0002` * valid values: ```json { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "ip-address": { "type": "string", "oneOf": [ {"pattern": r'^(\d+\.){3}\d+$'}, {"pattern": r'^([a-f\d]{4}:){7}[a-f\d]{4}$'} ] } }, "type": "object", "properties": { "name": {"$ref": "#/definitions/ip-address"}, "description": {"type": "string"}, "as": { "type": "object", "properties": { "local": {"type": "integer"}, "peer": {"type": "integer"}, }, "required": ["local", "peer"], "additionalProperties": False } }, "required": ["name", "description", "as"], "additionalProperties": False } ``` * `vpn_rr_peers/<address>` * key examples * `vpn_rr_peers:193.203.0.203` * valid values: ```json { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "ip-address": { "type": "string", "oneOf": [ {"pattern": r'^(\d+\.){3}\d+$'}, {"pattern": r'^([a-f\d]{4}:){7}[a-f\d]{4}$'} ] } }, "type": "object", "properties": { "name": {"$ref": "#/definitions/ip-address"}, "description": {"type": "string"}, "peer-as": {"type": "integer"} }, "required": ["name", "description"], "additionalProperties": False } ``` * `reverse_interface_addresses/<address>` * key examples * `reverse_interface_addresses:193.203.0.203` * `reverse_interface_addresses:2001:07f8:00a0:0000:0000:5926:0000:0002` * valid values: ```json { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "v4a": { "type": "string", "pattern": r'^(\d+\.){3}\d+$' }, "v6a": { "type": "string", "pattern": r'^([a-f\d]{4}:){7}[a-f\d]{4}$' }, "v4i": { "type": "string", "pattern": r'^(\d+\.){3}\d+/\d+$' }, "v6i": { "type": "string", "pattern": r'^[a-f\d:]+/\d+$' } }, "type": "array", "items": { "type": "object", "properties": { "name": { "oneOf": [ {"$ref": "#/definitions/v4a"}, {"$ref": "#/definitions/v6a"} ] }, "interface address": { "oneOf": [ {"$ref": "#/definitions/v4i"}, {"$ref": "#/definitions/v6i"} ] }, "interface name": {"type": "string"}, }, "required": ["name", "interface address", "interface name"], "additionalProperties": False } } ```