updating biblib to python 3.10

.github/workflows/python_actions.yml

@@ -11,13 +11,12 @@ jobs:
     - uses: actions/checkout@v2
     - uses: actions/setup-python@v2
       with:
-        python-version: 3.8
+        python-version: '3.10'
 
     - name: Install dependencies
       run: |
-        python -m pip install --upgrade wheel setuptools==57 pip
-        pip install -U -r requirements.txt
-        pip install -U -r dev-requirements.txt
+        python -m pip install "pip==24" setuptools==57.5.0 wheel
+        pip install ".[dev]"
 
     - name: Test with pytest
       run: |

.gitignore

@@ -20,3 +20,9 @@ python
 docker-compose*
 Dockerfile
 scripts/pytest/*
+build/
+*.egg-info/
+__pycache__/
+.pytest_cache/
+.tox/
+dist/

README.md

@@ -19,14 +19,20 @@ To load and enter the VM: `vagrant up && vagrant ssh`
 
 ### tests
 
-Run the tests using `py.test`:
+Run the tests using `pytest`:
 ```bash
-docker run --name some-postgres -e POSTGRES_USER="postgres" POSTGRES_PASSWORD="postgres" -p 5432:5432 --name postgres
-virtualenv python
+docker run -d --name postgres -e POSTGRES_USER="postgres" -e POSTGRES_PASSWORD="postgres" -p 5432:5432 postgres:12.6
+
+# Setup environment
+python3 -m venv python
 source python/bin/activate
-pip install -r requirements.txt
-pip install -r dev-requirements.txt
-py.tests biblib/tests/
+
+# Install with legacy build support (requires pip 24 and specific setuptools)
+python -m pip install "pip==24" setuptools==57.5.0 wheel
+pip install -e ".[dev]"
+
+# Run tests
+pytest
 ```
 
 ### Layout
@@ -42,33 +48,50 @@ All tests have been written top down, or in a Test-Driven Development approach,
 ### Running Biblib Locally
 
 To run a version of Biblib locally, a postgres database needs to be created and properly formatted for use with Biblib. This can be done with a local postgres instance or in a docker container using the following commands.
-`config.py` must also be copied to `local_config.py` and the environment variables must be adjusted to reflect the local environment.
+`local_config.py` should be created in `biblib/` and the environment variables must be adjusted to reflect the local environment.
+
 ```bash
-docker run -d -e POSTGRES_USER="postgres" -e POSTGRES_PASSWORD="postgres" -p 5432:5432 --name postgres  postgres:12.6
-docker exec -it postgres bash -c "psql -c \"CREATE ROLE biblib_service WITH LOGIN PASSWORD 'biblib_service';\""
-docker exec -it postgres bash -c "psql -c \"CREATE DATABASE biblib_service;\""
-docker exec -it postgres bash -c "psql -c \"GRANT CREATE ON DATABASE biblib_service TO biblib_service;\""
+# Setup database
+docker run -d -e POSTGRES_USER="postgres" -e POSTGRES_PASSWORD="postgres" -p 5432:5432 --name postgres postgres:12.6
+docker exec -it postgres psql -U postgres -c "CREATE ROLE biblib_service WITH LOGIN PASSWORD 'biblib_service';"
+docker exec -it postgres psql -U postgres -c "CREATE DATABASE biblib_service;"
+docker exec -it postgres psql -U postgres -c "GRANT ALL PRIVILEGES ON DATABASE biblib_service TO biblib_service;"
 ```
 
-Once the database has been created, `alembic` can be used to upgrade the database to the correct alembic revision
-```bash
-#In order for alembic to have access to the models metadata, the biblib-service directory must be added to the PYTHONPATH
+# Run migrations
+# In order for alembic to have access to the models metadata, the biblib-service directory must be added to the PYTHONPATH
 export PYTHONPATH=$(pwd):$PYTHONPATH
+python biblib/manage.py syncdb  # This will sync users and can be used to initialize schema via alembic indirectly or directly:
 alembic upgrade head
 ```
 
-A new revision can be created by doing the following:
+A test version of the microservice can then be deployed using:
 ```bash
-#In order for alembic to have access to the models metadata, the biblib-service directory must be added to the PYTHONPATH
-export PYTHONPATH=$(pwd):$PYTHONPATH
-alembic revision -m "<revision-name>" --autogenerate
+export FLASK_APP=biblib/app.py
+flask run --port 4000
+```
+or via the legacy entrypoint:
+```bash
+python wsgi.py
 ```
 
-A test version of the microservice can then be deployed using
+### Database versioning
+
+Database versioning is managed using Alembic. You can upgrade to the latest revision or downgrade to a previous one using the following commands:
+
 ```bash
-python3 wsgi.py
+# Upgrade to latest revision
+alembic upgrade head
+
+# Downgrade revision
+alembic downgrade <revision>
+
+# Create a new revision
+alembic revision --autogenerate -m "revision description"
 ```
 
+New revisions of libraries and notes are created automatically by `sqlalchemy-continuum` whenever a record is updated and committed to the database.
+
 ## deployment
 
 The only thing to take care of when making a deployment is the migration of the backend database. Libraries uses specific features of PostgreSQL, such as `UUID` and `JSON`-store, so you should think carefully if you wish to change the backend. **The use of `flask-migrate` for database migrations has been replaced by directly calling `alembic`.**  

biblib/manage.py

@@ -1,147 +1,171 @@
-"""
-Alembic migration management file
-"""
-from datetime import datetime
-from dateutil.relativedelta import relativedelta
 import os
 import sys
-PROJECT_HOME = os.path.abspath(
-    os.path.join(os.path.dirname(__file__), '..'))
-sys.path.append(PROJECT_HOME)
-from flask import current_app
-from flask_script import Manager, Command, Option
-from flask_migrate import Migrate, MigrateCommand
-from biblib.models import Base, User, Permissions, Library, Notes
+import click
 from biblib.app import create_app
-from sqlalchemy import create_engine, desc
-from sqlalchemy.orm import sessionmaker, scoped_session
+from biblib.models import User, Permissions, Library, Notes
+from flask import current_app
+from datetime import datetime
+from dateutil.relativedelta import relativedelta
 import sqlalchemy_continuum
 
-# Load the app with the factory
-app = create_app()
-
-class DeleteStaleUsers(Command):
+class DeleteStaleUsers:
     """
     Compares the users that exist within the API to those within the
-    microservice and deletes any stale users that no longer exist. The logic
-    also takes care of the associated permissions and libraries depending on
-    the cascade that has been implemented.
+    microservice and deletes any stale users that no longer exist.
     """
-    @staticmethod
-    def run(app=app):
-        """
-        Carries out the deletion of the stale content
-        """
+    def run(self, app=None):
+        if app is None:
+            app = create_app()
         with app.app_context():
             with current_app.session_scope() as session:
                 # Obtain the list of API users
                 postgres_search_text = 'SELECT id FROM users;'
                 result = session.execute(postgres_search_text).fetchall()
                 list_of_api_users = [int(r[0]) for r in result]
 
-                # Loop through every use in the service database
+                # Loop through every user in the service database
                 removal_list = []
                 for service_user in session.query(User).all():
                     if service_user.absolute_uid not in list_of_api_users:
                         try:
                             # Obtain the libraries that should be deleted
-                            permissions = session.query(Permissions).filter(Permissions.user_id == service_user.id).all()
-                            libraries = [session.query(Library).filter(Library.id == permission.library_id).one() for permission in permissions if permission.permissions['owner']]
+                            permissions = session.query(Permissions).filter(
+                                Permissions.user_id == service_user.id
+                            ).all()
+
+                            libraries = [
+                                session.query(Library).filter(Library.id == permission.library_id).one() 
+                                for permission in permissions if permission.permissions['owner']
+                            ]
 
                             # Delete all the libraries found
                             # By cascade this should delete all the permissions
-                            d = [session.delete(library) for library in libraries]
-                            p = [session.delete(permission) for permission in permissions]
-                            d_len = len(d)
-
+                            for library in libraries:
+                                session.delete(library)
+                            for permission in permissions:
+                                session.delete(permission)
+
                             session.delete(service_user)
                             session.commit()
+
+                            d_len = len(libraries)
                             current_app.logger.info('Removed stale user: {} and {} libraries'.format(service_user, d_len))
+
                             removal_list.append(service_user)
-
+                            
                         except Exception as error:
                             current_app.logger.info('Problem with database, could not remove user {}: {}'
                                                     .format(service_user, error))
                             session.rollback()
+
                 current_app.logger.info('Deleted {} stale users: {}'.format(len(removal_list), removal_list))
 
-class DeleteObsoleteVersionsTime(Command):
+class DeleteObsoleteVersionsTime:
     """
     Clears obsolete library and notes versions older than chosen time.
     """
-    @staticmethod
-    def run(app=app, n_years=None):
-        """
-        Carries out the deletion of older versions
-        """
+    def run(self, n_years=None, app=None):
+        if app is None:
+            app = create_app()
         with app.app_context():
-
-            if not n_years: n_years = current_app.config.get('REVISION_TIME', 7)
+            if not n_years:
+                n_years = current_app.config.get('REVISION_TIME', 7)
 
             with current_app.session_scope() as session:
-                # Obtain a list of all versions older than 1 year.
+                # Obtain a list of all versions older than chosen time.
                 LibraryVersion = sqlalchemy_continuum.version_class(Library)
                 NotesVersion = sqlalchemy_continuum.version_class(Notes)
+
                 current_date = datetime.now()
                 current_offset = current_date - relativedelta(years=n_years)
-                try:
-                    library_results = session.query(LibraryVersion).filter(LibraryVersion.date_last_modified<current_offset).all()
-                    notes_results = session.query(NotesVersion).filter(NotesVersion.date_last_modified<current_offset).all() 
 
-                    d_library = [session.delete(revision) for revision in library_results]
-                    d_notes = [session.delete(revision) for revision in notes_results] 
+                try:
+                    library_results = session.query(LibraryVersion).filter(
+                        LibraryVersion.date_last_modified < current_offset
+                    ).all()
+                    notes_results = session.query(NotesVersion).filter(
+                        NotesVersion.date_last_modified < current_offset
+                    ).all()
+
+                    for revision in library_results:
+                        session.delete(revision)
+                    for revision in notes_results:
+                        session.delete(revision)
 
-                    d_library_len = len(d_library)
-                    d_notes_len = len(d_notes)
                     session.commit()
-                    current_app.logger.info('Removed {} obsolete library revisions'.format(d_library_len))
-                    current_app.logger.info('Removed {} obsolete notes revisions'.format(d_notes_len))
+
+                    current_app.logger.info('Removed {} obsolete library revisions'.format(len(library_results)))
+                    current_app.logger.info('Removed {} obsolete notes revisions'.format(len(notes_results)))
+
                 except Exception as error:
-                        current_app.logger.info('Problem with database, could not remove revisions: {}'
-                                                .format(error))
-                        session.rollback()
+                    current_app.logger.info('Problem with database, could not remove revisions: {}'
+                                            .format(error))
+                    session.rollback()
 
-class DeleteObsoleteVersionsNumber(Command):
+class DeleteObsoleteVersionsNumber:
     """
     Limits number of revisions saved per entity to n_revisions.
     """
-    @staticmethod
-    def limit_revisions(session, entity_class, n_revisions):
-        VersionClass = sqlalchemy_continuum.version_class(entity_class)
-        entities = session.query(entity_class).all()
-
-        for entity in entities:
-            try:
-                revisions = session.query(VersionClass).filter_by(id=entity.id).order_by(VersionClass.date_last_modified.asc()).all()
-                current_app.logger.debug('Found {} revisions for entity: {}'.format(len(revisions), entity.id))
-                obsolete_revisions = revisions[:-n_revisions]
-                deleted_revisions = [session.delete(revision) for revision in obsolete_revisions]
-                deleted_revisions_len = len(deleted_revisions)
-                session.commit()
-                current_app.logger.info('Removed {} obsolete revisions for entity: {}'.format(deleted_revisions_len, entity.id))
-
-            except Exception as error:
-                current_app.logger.info('Problem with the database, could not remove revisions for entity {}: {}'
-                                        .format(entity, error))
-                session.rollback()
-
-    @staticmethod
-    def run(app=app, n_revisions=None):
-        if not n_revisions:
-            n_revisions = current_app.config.get('NUMBER_REVISIONS', 7)
-
+    def run(self, n_revisions=None, app=None):
+        if app is None:
+            app = create_app()
         with app.app_context():
-            with current_app.session_scope() as session:
-                DeleteObsoleteVersionsNumber.limit_revisions(session, Library, n_revisions)
-                DeleteObsoleteVersionsNumber.limit_revisions(session, Notes, n_revisions)
-
-
+            if not n_revisions:
+                n_revisions = current_app.config.get('NUMBER_REVISIONS', 7)
+
+            def limit_revisions(session, entity_class, n_revisions):
+                VersionClass = sqlalchemy_continuum.version_class(entity_class)
+                entities = session.query(entity_class).all()
+
+                for entity in entities:
+                    try:
+                        revisions = session.query(VersionClass).filter_by(id=entity.id).order_by(
+                            VersionClass.date_last_modified.asc()
+                        ).all()
+
+                        current_app.logger.debug('Found {} revisions for entity: {}'.format(len(revisions), entity.id))
+
+                        if len(revisions) > n_revisions:
+                            obsolete = revisions[:-n_revisions]
+                            for r in obsolete:
+                                session.delete(r)
+
+                            session.commit()
+
+                            current_app.logger.info('Removed {} obsolete revisions for entity: {}'.format(len(obsolete), entity.id))
+
+                    except Exception as error:
+                        current_app.logger.info('Problem with the database, could not remove revisions for entity {}: {}'
+                                                .format(entity, error))
+                        session.rollback()
 
-# Setup the command line arguments using Flask-Script
-manager = Manager(app)
-manager.add_command('syncdb', DeleteStaleUsers())
-manager.add_command('clean_versions_time', DeleteObsoleteVersionsTime())
-manager.add_command('clean_versions_number', DeleteObsoleteVersionsNumber())
+            with current_app.session_scope() as session:
+                limit_revisions(session, Library, n_revisions)
+                limit_revisions(session, Notes, n_revisions)
+
+# CLI part for backward compatibility running as script
+@click.group()
+def manager():
+    """Management script for the Biblib service."""
+    pass
+
+@manager.command()
+def syncdb():
+    """Compares microservice users to API users and deletes stale users."""
+    DeleteStaleUsers().run()
+
+@manager.command(name='clean_versions_time')
+@click.option('--years', default=None, type=int, help='Number of years to keep')
+def clean_versions_time(years):
+    """Clears obsolete revisions older than chosen time."""
+    DeleteObsoleteVersionsTime().run(n_years=years)
+
+@manager.command(name='clean_versions_number')
+@click.option('--revisions', default=None, type=int, help='Maximum revisions to keep')
+def clean_versions_number(revisions):
+    """Limits number of revisions saved per entity."""
+    DeleteObsoleteVersionsNumber().run(n_revisions=revisions)
 
 if __name__ == '__main__':
-    manager.run()
+    manager()
+

biblib/models.py

@@ -36,6 +36,7 @@ class GUID(TypeDecorator):
     as Flask cannot serialise UUIDs correctly.
 
     """
+    cache_ok = True
     # Refers to the class of type being decorated
     impl = CHAR