Feature/add validation workflows

• Fix the product_validation task
• Add Router verification workflow
• Add IP Trunk verification workflow
• Update documentation
• Add modify_note workflow to all products
• Update translations
• Note: The router creation workflow now fails if IPAM is incorrectly configured, with the reasoning that the step can be retried after amending the problem. This can be reverted.

requested review from @simone.spinelli

assigned to @karel.vanklink

marked this merge request as draft

Marking this MR as draft, as it contains db migrations that will probably need to get adjusted by the time this is approved.

approved this merge request

unapproved this merge request

added 228 commits

• 335cb262...1f5d528e - 218 commits from branch develop
• edb0500a - add router validation task
• 87c4f186 - Add nightly schedule for router validation
• b613bd64 - convert scheduled router validation to celery task
• 07cd30e2 - add missing workflows to translations file
• a525634c - add modify_note workflow to all existing products
• 5e378a7f - add unit test for product validation workflow
• 37087a8f - add unit test for router validation workflow
• f467e7bf - Add IP Trunk validation workflow
• 205af46e - Add unit test for IP Trunk validation workflow
• 6cbfbcee - update documentation, include missing workflows and glossary entries

Compare with previous version

added 3 commits

• 2077ab60 - rename stray pp_interaction references
• cab33def - re-order database migrations to prevent double heads
• d8417ed4 - fixup! remove router validation task

Compare with previous version

DB migrations have been updated, should now be correct. Will further update the behaviour of the validation workflows.

For now however: a validation task is failing since there are workflows which are not registered in the database. This is something we do on purpose, as these are the import workflows for example. How do we want to handle this? We could disable the validation check, or have the tasks in the database after all.

added 4 commits

• a75b31b9 - add migration for adding modify_note workflow to superpop switch and officerouter products
• f58c65d9 - re-order steps in router validation workflow
• b75c3666 - extend the LSO client with an interaction that does not include an input step
• beb3a02d - Update router validation workflow

Compare with previous version

added 74 commits

• beb3a02d...2150cca4 - 47 commits from branch develop
• 2150cca4...6ef85a90 - 17 earlier commits
• 1f87a7e7 - Update IPAM validation for IP trunks
• b82b706f - Add NetBox check to IP trunk validation workflow
• f3f6daaf - Add check for enabled interfaces
• 6dc33bbe - Rename all uses of the init step to begin
• b1d81768 - modify input form for router validation workflow
• b7039429 - Add missing translation key, reorder db migrations
• 3d2449f3 - Replace dud migration for adding modify note workflow for a working one
• 29742f54 - remove data label from switch migration
• c8deee0e - Split up validation workflow for juniper and nokia routers
• fc6b48e6 - Update playbook validation steps

Compare with previous version

added 4 commits

• 632d46b5 - Speed up import unit test by mocking redundant sleeps
• e637837c - re-order db migrations to avoid branching
• aefaa8d1 - Add edit note workflow to Opengear product
• d3dfc157 - update unit test for validate iptrunk

Compare with previous version

added 45 commits

• d3dfc157...87fb70b2 - 35 earlier commits
• 21002ada - Split up validation workflow for juniper and nokia routers
• 7162f3f2 - Update playbook validation steps
• 661169c9 - Speed up import unit test by mocking redundant sleeps
• 175c9126 - re-order db migrations to avoid branching
• 15c2b6ab - Add edit note workflow to Opengear product
• 4cf9a5e7 - update unit test for validate iptrunk
• 34191389 - lag interface name on iptrunk side is mandatory for provisioning and active states
• e8affac4 - Replace calls to deprecated .dict() with .model_dump()
• b1a44100 - re-order db migrations
• ed8cca91 - Bump to python 3.12

Compare with previous version

resolved all threads

added 3 commits

• f0931459 - Update method names in router validation
• 02f4e757 - replace calls to += with append
• a800bd4a - Improve and use LibreNMS validation method

Compare with previous version

marked this merge request as ready

added 1 commit

• 88c42829 - Update extra vars for validation workflows

Compare with previous version

added 58 commits

• 88c42829...74019763 - 19 commits from branch develop
• 74019763...87b78270 - 29 earlier commits
• 4f6d315a - Add edit note workflow to Opengear product
• e3310af4 - update unit test for validate iptrunk
• 86c6ff31 - lag interface name on iptrunk side is mandatory for provisioning and active states
• 421a1792 - Replace calls to deprecated .dict() with .model_dump()
• 830ec7f0 - re-order db migrations
• 0bb7cb30 - Bump to python 3.12
• 4f0bee26 - Update method names in router validation
• d61f0425 - replace calls to += with append
• 93fe51e3 - Improve and use LibreNMS validation method
• b8513417 - Update extra vars for validation workflows

Compare with previous version

marked this merge request as draft from d9a6d351

added 52 commits

• b8513417...245132e1 - 13 commits from branch develop
• 245132e1...f529bd7a - 29 earlier commits
• 2a233fa4 - Add edit note workflow to Opengear product
• ce215177 - update unit test for validate iptrunk
• 84463f04 - lag interface name on iptrunk side is mandatory for provisioning and active states
• 1173a611 - Replace calls to deprecated .dict() with .model_dump()
• 3934bf67 - re-order db migrations
• 36d81973 - Bump to python 3.12
• ab0caf5a - Update method names in router validation
• e4ce9fe8 - replace calls to += with append
• d6dc787c - Improve and use LibreNMS validation method
• 83607d06 - Update extra vars for validation workflows

Compare with previous version

added 41 commits

• 83607d06...4ac6686a - 31 earlier commits
• b3890607 - lag interface name on iptrunk side is mandatory for provisioning and active states
• 80386a29 - Replace calls to deprecated .dict() with .model_dump()
• c5c538c7 - re-order db migrations
• 6c98cf6c - Bump to python 3.12
• 126d0c54 - Update method names in router validation
• 6935797f - replace calls to += with append
• 4c2cb7cf - Improve and use LibreNMS validation method
• 3179d91b - Update extra vars for validation workflows
• b669888c - validate_router: subscription serialization
• dda99bae - validate_router: FIX librenms

Compare with previous version

added 5 commits

• 3c01bd32 - validate_router: subscription serialization
• be977d6f - Externalise the generation of an active PE-router dict
• e473a973 - Only run validation workflows on subscriptions that are both ACTIVE and in sync
• 1ea7d1ac - Only run validation workflows on subscriptions that are both ACTIVE and in sync
• 243c2e5c - Split off workflow steps for adding all PE routers to a P router

Compare with previous version

added 1 commit

• 47c85c6a - remove tt_number from `add_all_pe_to_p_dry`

Compare with previous version

added 1 commit

• 05eecc25 - iBGP "table" -> "group"

Compare with previous version

added 1 commit

• aecb5b5b - validate_router

Compare with previous version

added 98 commits

• aecb5b5b...2323d627 - 50 commits from branch develop
• 2323d627...50974b01 - 38 earlier commits
• f8827c21 - Update extra vars for validation workflows
• 3013ff14 - validate_router: subscription serialization
• 82b0ac8a - Externalise the generation of an active PE-router dict
• 7ccaed9c - Only run validation workflows on subscriptions that are both ACTIVE and in sync
• 96121bc2 - Only run validation workflows on subscriptions that are both ACTIVE and in sync
• 91c5fd4d - Split off workflow steps for adding all PE routers to a P router
• 9e22894a - remove tt_number from `add_all_pe_to_p_dry`
• 932399b1 - iBGP "table" -> "group"
• eeb2f808 - validate_router
• eee0933c - Fixed the migration down revision after rebasing with develop and update the...

Compare with previous version

added 47 commits

• eee0933c...838d916b - 37 earlier commits
• 3bb0816a - Improve and use LibreNMS validation method
• 5ed7733f - Update extra vars for validation workflows
• ffb84be8 - validate_router: subscription serialization
• 516da16b - Externalise the generation of an active PE-router dict
• 084b87eb - Only run validation workflows on subscriptions that are both ACTIVE and in sync
• ba8bf336 - Only run validation workflows on subscriptions that are both ACTIVE and in sync
• e807a283 - Split off workflow steps for adding all PE routers to a P router
• f2b785a6 - remove tt_number from `add_all_pe_to_p_dry`
• d2e2e7bb - iBGP "table" -> "group"
• 0851a669 - validate_router

Compare with previous version

added 47 commits

• 0851a669...6935797f - 37 earlier commits
• 4c2cb7cf - Improve and use LibreNMS validation method
• 3179d91b - Update extra vars for validation workflows
• 3c01bd32 - validate_router: subscription serialization
• be977d6f - Externalise the generation of an active PE-router dict
• e473a973 - Only run validation workflows on subscriptions that are both ACTIVE and in sync
• 1ea7d1ac - Only run validation workflows on subscriptions that are both ACTIVE and in sync
• 243c2e5c - Split off workflow steps for adding all PE routers to a P router
• 47c85c6a - remove tt_number from `add_all_pe_to_p_dry`
• 05eecc25 - iBGP "table" -> "group"
• aecb5b5b - validate_router

Compare with previous version

added 1 commit

• 48352f7c - Corrected `validate_iptrunk`

Compare with previous version

added 98 commits

• 48352f7c...2323d627 - 50 commits from branch develop
• 2323d627...4ec97895 - 38 earlier commits
• a30335d1 - Update extra vars for validation workflows
• e87cda1e - validate_router: subscription serialization
• 2aa36195 - Externalise the generation of an active PE-router dict
• 72462069 - Only run validation workflows on subscriptions that are both ACTIVE and in sync
• c9ea4eda - Only run validation workflows on subscriptions that are both ACTIVE and in sync
• 2eb1d161 - Split off workflow steps for adding all PE routers to a P router
• 2df011eb - remove tt_number from `add_all_pe_to_p_dry`
• 86b1bdaf - iBGP "table" -> "group"
• 48992593 - validate_router
• 825cacee - Corrected `validate_iptrunk`

Compare with previous version

added 1 commit

• 0f7d232d - Fixed the migration down revision after rebasing with develop and update the...

Compare with previous version

added 6 commits

• 1a5f2d6f - replace call to deprecated dict method
• e4cdb4c5 - Add missing translation keys
• c675b274 - Split off task_validate_products
• 8f27bdaf - Update docs, clean up validation task file
• 9d070ca7 - remove modify_note migrations from history
• 3edfb440 - Comment out a check that will never pass at this moment

Compare with previous version

approved this merge request

added 59 commits

• 3edfb440...552452e3 - 4 commits from branch develop
• 552452e3...39860601 - 45 earlier commits
• 4eba0ed0 - iBGP "table" -> "group"
• b981428a - validate_router
• 2786b6b1 - Corrected `validate_iptrunk`
• fcc894f4 - Fixed the migration down revision after rebasing with develop and update the...
• 7f25e136 - replace call to deprecated dict method
• 3036855f - Add missing translation keys
• 5d0e303e - Split off task_validate_products
• c381e721 - Update docs, clean up validation task file
• e7c8f2a0 - remove modify_note migrations from history
• 1fe8764b - Comment out a check that will never pass at this moment

Compare with previous version

marked this merge request as ready

enabled an automatic merge when the pipeline for 1fe8764b succeeds

merged