sdx-controller v3.0.0-alpha

What's Changed

This release note covers the most relevant changes across the other three repos in the SDX Middleware Software system: sdx-LC, PCE, and Datamodel.

Major New Features

1. Conformity to the latest AW-SDX data model specification: Topology Data Model Specification 2.0.0 and Service Provisioning Data Model Specification 1.0

2. L2VPN Provisioning workflow enhancement in reliability, performance, fault tolerance, and resource (vlan and bandwidth) management.

2.1 Formalized the lifetime state machine of L2VPN connections and implemented it in the datamodel library. This included defining the:

• Connection state transitions
  Connection Transition

• OXP topology state transitions
  Topology Transition

2.2 Completed the L2VPN provisioning workflow across the sdx-controller, sdx-lc, and oxp sub-systems with correct API calls and response counting for the connection state transition.

2.3 Enhanced the OXP topology update handling to account for the resource and status change events in ports and links. Implemented failure handling to deal with OXP topology port failure on top of the existing link failure handling in both inter- and intra-OXP cases. The functions are located in the PCE library (https://github.com/atlanticwave-sdx/pce)

2.5 Implemented provisioning function on the same switch.

3. AW-SDX Middleware enhancement in reliability and fault-tolerance

3.1 Thread-safe, synchronization, and communication performance enhancements across the distributed subsystems.

3.2 Crash-safe SDX-controller restart to account for existing connections and resource usage

3.3 Message queue reliability and security enhancements

3.4 MongoDB reliability and security enhancements

4. Implemented a pluggable XOP provisioning system client mechanism in sdx-LC (https://github.com/atlanticwave-sdx/sdx-lc).

https://sdx-docs.readthedocs.io/en/latest/sdx_lc_oxp.html

Now sdx-lc supports both aw-sdx datamodel-compliant and non-compliant OXPs

An example implementation of an OXP client for the DoE SENSE system (https://sense.es.net/sdn-for-end-to-end-networked-science-at-the-exascale-sense) can be found in the repo https://github.com/atlanticwave-sdx/sense-sdx.

**5. End-to-end system emulation testing Jupyter notebooks **
Some examples to use the NSF FABRIC for the test can be found in this repo:

https://github.com/atlanticwave-sdx/sdx-jupyter-notebooks

6. Software engineering and CI/CD
6.1 unitest: tox-based automatic virtual system testing
6.2 Implementation of GitHub Action-based CI/CD workflow, including the PR process.
6.3 Updated Dockerfile and automated image publication to Docker Hub

7. Automated software document publication
https://sdx-docs.readthedocs.io/en/latest/
https://github.com/atlanticwave-sdx/sdx-docs

What's Changed in the code

1. SDX-Controller API endpoint changes: compliant with Topology Data Model Specification 2.0.0, enhanced failure handling.
2. SDX-Controller MongoDB changes: save data as MongoDB compatible raw JSON objects.
3. Message queue changes: enable password authentication
4. sdx-lc changes (see sdx-lc release notes: https://github.com/atlanticwave-sdx/sdx-lc/releases)
5. pce changes (see pce release notes: https://github.com/atlanticwave-sdx/pce/releases)
6. datamodel changes (see data model release notes: https://github.com/atlanticwave-sdx/datamodel/releases)
7. Other improvements and bug fixes

Reference: Closed PRs

• Avoid a busy wait by @sajith in #175
• Use pce 2.0.2 by @sajith in #181
• Clean up imports by @sajith in #185
• Remove top-level tests and test data by @sajith in #183
• Enable save log to file by @congwang09 in #188
• Read latest topo from db only if entry exists by @congwang09 in #187
• Update docker compose by @congwang09 in #189
• Use pce 2.0.3 by @sajith in #193
• Add mongo port binding by @congwang09 in #194
• change time_stamp to timestamp by @congwang09 in #190
• link failure handling, part 1 by @congwang09 in #195
• Enable adding connection after link failure by @congwang09 in #196
• Fix typo in "connection" by @sajith in #203
• Use pce 2.0.6 by @sajith in #198
• Add elastic search to compose by @sajith in #201
• Avoid busy wait in bapm_server by @sajith in #200
• Split compose configuration by @sajith in #205
• Update black by @sajith in #211
• Remove unused files by @sajith in #208
• Upgrade MongoDB by @sajith in #202
• Use pyproject.toml by @sajith in #212
• Rename swagger_server to sdx_controller by @sajith in #214
• Make sdx controller a WSGI app by @sajith in #215
• Docker and app fixes by @sajith in #218
• Add return breakdown for connection_handler by @congwang09 in #226
• Add ISO3166 2 lvl4 by @congwang09 in #228
• Fix return for place connection by @congwang09 in #231
• Add link to sample failure topology request by @congwang09 in #233
• Use tox-docker to run services when testing by @sajith in #242
• Enable RabbitMQ when testing with tox by @sajith in #246
• Test place_connection() more by @sajith in #209
• Additional changes to allow RabbitMQ authentication by @italovalcy in #250
• Use password authentication with RabbitMQ by @sajith in #223
• Fix pika auth for rpc_queue_consumer by @italovalcy in #254
• Use request connection id when saving to db by @congwang09 in #258
• Update OXP domain name according to the new standard on SDX-LC by @italovalcy in #255
• Handle connection deletions in PCE by @sajith in #225
• Remove MongoDB internal _id from the user response and change status code for graph validation by @italovalcy in #262
• Add end point to list all connections by @congwang09 in #264
• Changing data returned by list connection by @italovalcy in #267
• use the new tagged pce with the enhanced user port support in breakdown by @YufengXin in #269
• Add a test for GET /connection/:connection_id by @sajith in #271
• To pce rc1 by @YufengXin in #272
• Add connection deletion by @congwang09 in #273
• Exclude deleted connections when listing all connections by @congwang09 in #274
• CI housekeeping: use the right coveralls client to finalize by @sajith in #275
• Use MongoDB 7.0.11 by @sajith in #277
• Multi-stage Docker image build by @sajith in #279
• Remove obsolete version field from compose spec by @sajith in #280
• Remove unused lc_queue_name by @sajith in #285
• Make MongoDB log to a file by @sajith in #284
• Remove unused messaging code by @sajith in #289
• Delete connection by @congwang09 in #288
• Use tox-docker expose directive by @sajith in #292
• Update failure handling by @congwang09 in #293
• Use correct env file format by @sajith in #295
• Use connection request v2 spec by @sajith in #297
• Use service_id instead of connection_id by @congwang09 in #301
• Use MONGODB_CONNSTRING when MONGO_HOST is not set by @italovalcy in #300
• Also delete breakdown when deleting connection by @congwang09 in #303
• Save connection request only when getting 200 code by @congwang09 in #304
• Add traceback information when exception on breakdown by @italovalcy in #306
• Adjustment of the mandatory fields in the Topology data model by @italovalcy in #307
• Modify link failure handler to leverage topology update messages by @italovalcy in #310
• 299 sdx controller must have new endpoints according to specification 20 by @YufengXin in #318
• Housekeeping updates by @sajith in #321
• Update SDX message when no topology is available yet by @congwang09 in #319
• Unbreak container image by @sajith in #322
• Add version to l2vpn by @congwang09 in #323
• Implement PATCH connection by @congwang09 in #324
• Add list domains api by @congwang09 in #325
• Reuse service id for patch by @congwang09 in #326
• Handle connection requests with VLAN set to "any" by @sajith in #327
• Use topology from database when initialize te manager by @congwang09 in #328
• Use te_manager delete_connection by @congwang09 in #330
• Use LC-X for topo db key by @congwang09 in #329
• Add archived connection api by @congwang09 in #331
• Use get_connection_status when listing connections by @congwang09 in #338
• Do not duplicate service_id when listing connections by @congwang09 in #339
• Add qos_metrics, scheduling, notifications to GET connection request by @congwang09 in #341
• Add archived_date as 0 for all active connections by @congwang09 in #344
• Update success code by @congwang09 in #345
• Cleanup topology handling logic by @congwang09 in #346
• Bump pce version up by @sajith in #353
• Add connection after failure handling by @congwang09 in #354
• Handle out-of-range VLANs gracefully by @sajith in #357
• Roll back patch when fail by @congwang09 in #358
• Listen to connection OXP response by @congwang09 in #377
• 375 stricter connection field check and error codes by @YufengXin in #378
• bump up to pce tag 3.0.0.dev7 by @YufengXin in #379
• Add or update connection status based on OXP response code by @congwang09 in #380
• New tag PCE for error codes by @YufengXin in #408
• use 3.11 for docker by @YufengXin in #410
• Better error handling by @congwang09 in #411
• Change strings to consts by @congwang09 in #416
• Adding log level configuration from env variable by @italovalcy in #415
• Use constants from datamodel by @congwang09 in #417
• Access link id by property by @congwang09 in #419
• Aggregate-topology-update by @YufengXin in #413
• Change link to dict access by @congwang09 in #428
• dealing with requests on the same node or same port by @YufengXin in #423
• 420 aggregate issue for connection status formalization by @YufengXin in #421
• oxp response counter initial value by @YufengXin in #432
• Only store list of service IDs, instead of objects by @congwang09 in #434
• Link failure handling error by @YufengXin in #433
• Fix inconsistent Link to Service IDs mapping by @italovalcy in #439
• Enhance l2vpn body persistency to avoid inconsistent value for oxp_success_count by @italovalcy in #440
• Fix PATCH L2VPN by @italovalcy in #445
• Adding support for displaying L2VPN status according to spec by @italovalcy in #447
• bump PCE version to benefit from recent changes by @italovalcy in #454
• bump pce version 3.1.0.dev3 => 3.1.0.dev4 by @italovalcy in #455
• Save raw JSON to MongoDB by @congwang09 in #453
• Using getField to fetch data from mongo collections with field names with periods by @italovalcy in #464
• Save domain topology to mongo indexed by topology ID by @italovalcy in #467
• Save domain topologies in raw python dict into MongoDB by @italovalcy in #469
• Avoid loading LATEST_TOPOLOGY into TEManager by @italovalcy in #471
• initial implementation of changing connection status to down if uni p… by @YufengXin in #463
• Bump pce to dev8 by @congwang09 in #474
• reject request on NNI port by @YufengXin in #475
• Fixed documents key for collection historical_connections that must be string by @italovalcy in #481
• Fix nested documents on ports collection by @italovalcy in #483
• bump pce version by @italovalcy in #484
• Save the full solution path of a request connection to DB by @YufengXin in #476
• Update to 2.0 API format by @congwang09 in #477
• Add generic exception handler to avoid SDX-Controller queue consumer crash on failures by @italovalcy in #494
• Change domain list to dict by @congwang09 in #499

New Contributors

• @italovalcy made their first contribution in #250

Full Changelog: 2.0.0...3.1.0.dev5