Add configuration for deployment and update privacy configuration/documentation

.github/workflows/ci.yml

@@ -6,10 +6,12 @@ on:
   pull_request:
     branches: [ "main" ]
 
+permissions:
+  pull-requests: write # allow posting comments on PRs by actions
 jobs:
   unittests:
     runs-on: ubuntu-24.04
-    
+
     strategy:
       max-parallel: 4
       matrix:
@@ -57,9 +59,25 @@ jobs:
       run: |
         python manage.py loaddata fixtures/test_data.json
 
+    - name: Run deploy checks
+      run: |
+        python -Wa manage.py check --deploy
+
     - name: Run Tests
       run: |
-        python -Wa manage.py test tests
+        coverage run --source='publications' --omit='*/migrations/**' manage.py test tests
+
+    - name: Check coverage and save it to files
+      run: |
+        coverage html
+        coverage xml
+        coverage report --show-missing --fail-under=70
+
+    #- name: Post code coverage
+    #  uses: orgoro/coverage@v3.2
+    #  with:
+    #      coverageFile: coverage.xml
+    #      token: ${{ secrets.GITHUB_TOKEN }}
 
   uitests:
     # disable until UI tests are fixed, see https://github.com/GeoinformationSystems/optimap/issues/142

.gitignore

@@ -133,7 +133,11 @@ geodjango/
 
 tests-ui/screenshots/
 
-static/
+/static/
 
 # nano's edit files on the server
 *.swp
+
+*.docx#
+
+publications/static/OPTIMAP_Data-Privacy-Concept.docx

Dockerfile

@@ -34,11 +34,12 @@ RUN set -ex && \
     pip install --no-cache-dir -r /tmp/requirements.txt && \
     rm -rf /root/.cache/
 
-COPY . /code/
+COPY . /code
 
-RUN python manage.py collectstatic --noinput
+RUN chmod a+x /code/etc/manage-and-run.sh
 
 EXPOSE 8000
 
 #CMD ["gunicorn", "--bind", ":8000", "--workers", "2", "optimap.wsgi"]
-CMD ["python", "manage.py", "runserver", "0.0.0.0:8000" ]
+#CMD ["python", "manage.py", "runserver", "0.0.0.0:8000" ]
+ENTRYPOINT [ "/code/etc/manage-and-run.sh" ]

README.md

@@ -29,37 +29,38 @@ A complete list of existing parameters is provided in the file `optimap/.env.exa
 
 ## Run with Docker
 
-The project is containerized using Docker, with services defined in `docker-compose.yml`. To start all services, run:
+The project is containerized using Docker, with services defined in `docker-compose.(deploy.)yml`. To start all services, run:
 
 ```bash
 docker compose up
-```
 
-### Initial Setup
+docker compose run --entrypoint python app manage.py loaddata fixtures/test_data.json
+```
 
-After starting the containers, apply database migrations:
+The database migrations are applied as part of the startup script, see file `etc/manage-and-run.sh`.
+You can still run the commands below manually if need be, e.g., during development.
 
 ```bash
-# run migrations, in the directory where docker-compose is to resolve the name "web"
-docker compose run app python manage.py makemigrations # should not detect and changes, otherwise your local config might be outdated
-docker compose run app python manage.py migrate
-docker compose run app python manage.py collectstatic --noinput
-docker compose run app python manage.py loaddata fixtures/test_data.json
+docker compose run --entrypoint python app manage.py makemigrations # should not detect and changes, otherwise your local config might be outdated
+docker compose run --entrypoint python app manage.py migrate
+docker compose run --entrypoint python app manage.py collectstatic --noinput
 ```
 
 Now open a browser at <http://localhost:80/>.
 
-#### Services Overview
+### Services Overview
 
 - db: Runs a PostgreSQL database with PostGIS extensions. Data is persisted in a Docker volume named db_data.
 - app: Our primary Django web application.
 - webserver: An Nginx server for serving static files and test files.
 
-#### Ports
+### Ports
 
-- 5434: Database (PostgreSQL/PostGIS)
-- 8000: App (Django server)
-- 8080: Webserver (Nginx)
+Not all of these ports are exposed by default, but they are available for local development - just uncomment the matching lines in the `docker-compose.yml` file.
+
+- `5432`: Database (PostgreSQL/PostGIS)
+- `8000`: App (Django server)
+- `80`: Webserver (Nginx)
 
 ## Development
 
@@ -137,6 +138,9 @@ python manage.py createcachetable
 # Collect static files
 python manage.py collectstatic --noinput
 
+# If you need to run tasks (harvesting, data export) then start a cluster in a separate shell
+python manage.py qcluster
+
 # Start the Django development server
 python manage.py runserver
 
@@ -256,14 +260,33 @@ python manage.py test tests
 python -Wa manage.py test
 
 # configure logging level for cleaner test progress output
-OPTIMAP_LOGGING_CONSOLE_LEVEL=WARNING python manage.py test tests
+OPTIMAP_LOGGING_LEVEL=WARNING python manage.py test tests
+```
+
+### Run UI tests
 
-# running UI tests needs either compose configuration or a manage.py runserver in a seperate shell
+Running UI tests needs either compose configuration or a manage.py runserver in a seperate shell.
+
+```bash
 docker-compose up --build
 
 python -Wa manage.py test tests-ui
 ```
 
+### Check test coverage
+
+```bash
+# run the tests and capture coverage
+coverage run --source='publications' --omit='*/migrations/**' manage.py test tests
+
+# show coverage report
+coverage report --show-missing --fail-under=70
+
+# save the reports
+coverage html
+coverage xml
+```
+
 ### Develop tests
 
 For developing the UI tests, you can remove the `headless=True` in the statements for starting the browsers so you can "watch along" and inspect the HTML when a breakpoint is hit as the tests are executed.
@@ -309,7 +332,7 @@ The changelog follows the [Keep a Changelog](https://keepachangelog.com/en/1.1.0
 
 The version is managed in `optimap/__init__.py`.
 
-### Design colours and logos
+## Design colours and logos
 
 Optimeta colour = _primary colour_: #158F9B
 
@@ -323,7 +346,7 @@ The **logos** and favicon are in the repository in the folder [`publications/sta
 
 ## Deploy
 
-Deploy using `docker-compose` or see [`fly.io.md`](fly.io.md) for notes on deploying to Fly.io.
+~~Deploy using `docker-compose` or see [`fly.io.md`](fly.io.md) for notes on deploying to Fly.io.~~
 
 ## Operation
 
@@ -344,6 +367,32 @@ Deploy using `docker-compose` or see [`fly.io.md`](fly.io.md) for notes on deplo
    - Go to `/admin/auth/user/`
    - Select users → Choose **"Delete user and block email/domain"** → Click **Go**.
 
+### Tasks
+
+We use [Django Q2](https://django-q2.readthedocs.io/) for scheduling (repeated) tasks.
+
+#### Run the cluster
+
+```bash
+python manage.py qcluster
+```
+
+#### Monitor
+
+Details: <https://django-q2.readthedocs.io/en/master/monitor.html>
+
+tl;dr:
+
+```bash
+python manage.py qmonitorq
+
+python manage.py qinfo
+```
+
+### Trigger creation of data export files
+
+TODO
+
 ## License
 
 This software is published under the GNU General Public License v3.0 (see file `LICENSE`).

docker-compose.deploy.yml

@@ -6,29 +6,39 @@ services:
       POSTGRES_USER: optimap
       POSTGRES_PASSWORD: optimap
       #POSTGRES_PASSWORD_FILE: database_password
-    ports:
-      - "5434:5432"
+    #ports:
+    #  - "5432:5432"
     volumes:
       - optimap-volume:/var/lib/postgresql/data
     #secrets:
     #  - database_password
 
   app:
-    build:
-      context: .
-      dockerfile: Dockerfile
+    build: .
     environment:
       #see local .env file on the server
       DATABASE_URL: postgis://optimap:optimap@db:5432/optimap?sslmode=disable
       OPTIMAP_DEBUG: "False"
     volumes:
-      - ".:/code" # need to run collectstatic
-      - "./fixtures:/fixtures"
-    ports:
-      - "8000:8000"
+      - "./optimap/.env:/code/optimap/.env:ro"
+    #ports:
+    #  - "8000:8000"
     depends_on:
       - db
 
+  djangoq:
+    build: .
+    environment:
+      #see local .env file on the server
+      DATABASE_URL: postgis://optimap:optimap@db:5432/optimap?sslmode=disable   
+    entrypoint: ["python"] # do not run manage-and-run.sh script
+    command: ["manage.py", "qcluster"]
+    volumes:
+      - "./optimap/.env:/code/optimap/.env:ro"
+    depends_on:
+      - db
+      - app
+
   webserver:
     image: nginx:1.27-alpine
     volumes:

docker-compose.yml

@@ -5,26 +5,36 @@ services:
       POSTGRES_DB: optimap
       POSTGRES_USER: optimap
       POSTGRES_PASSWORD: optimap
-    ports:
-      - "5434:5432"
+    #ports:
+    #  - "5434:5432"
     volumes:
       - db_data:/var/lib/postgresql/data
 
   app:
-    build:
-      context: .
-      dockerfile: Dockerfile
+    build: .
     environment:
       DATABASE_URL: postgis://optimap:optimap@db:5432/optimap?sslmode=disable
       OPTIMAP_CACHE: "dummy"
-      OPTIMAP_DEBUG: "True"
     volumes:
-      - ".:/code:ro"
+      - ".:/code" # mount the code for development and testing
       - "./fixtures:/fixtures:ro"
-    ports:
-      - "8000:8000"
+    #ports:
+    #  - "8000:8000"
+    depends_on:
+      - db
+
+  djangoq:
+    build: .
+    environment:
+      DATABASE_URL: postgis://optimap:optimap@db:5432/optimap?sslmode=disable
+      OPTIMAP_CACHE: "dummy"
+    entrypoint: ["python"] # do not run manage-and-run.sh script
+    command: ["manage.py", "qcluster"]
+    volumes:
+      - ".:/code:ro"
     depends_on:
       - db
+      - app
 
   webserver:
     image: nginx:1.27-alpine

etc/manage-and-run.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+set -e
+
+# Collect static files
+echo "OPTIMAP ENTRYPOINT | Collect static files"
+python manage.py collectstatic --noinput
+
+# Create a cache table
+echo "OPTIMAP ENTRYPOINT | Create cache table"
+python manage.py createcachetable
+
+# Apply database migrations
+echo "OPTIMAP ENTRYPOINT | Apply database migrations"
+python manage.py migrate
+
+# Start server
+echo "OPTIMAP ENTRYPOINT | Starting server"
+# make the next process the main process, cf. https://www.baeldung.com/ops/docker-exec-process-replacement
+exec python manage.py runserver 0.0.0.0:8000
+
+# TODO: revisit entrypoint approach if there are any issues, e.g., by using an extra migration container, cf. https://stackoverflow.com/questions/33992867/how-do-you-perform-django-database-migrations-when-using-docker-compose, or by manually running selected commands after installation, cf. https://www.baeldung.com/ops/django-database-migrations-docker-compose

fly.io.md

@@ -186,6 +186,20 @@ fly proxy 15432:5432 -a optimap-db
 
 Connect to database locally at port `15432`, e.g., with pgAdmin.
 
+## Get a shell in the app container
+
+```bash
+flyctl --app optimap console
+```
+
+Inspect the settings,  load test data, etc.
+
+```bash
+python manage.py loaddata fixtures/test_data.json
+
+python manage.py diffsettings
+```
+
 ## Disable the app
 
 For example, when you want to manipulate the database without any open connections:

fly.toml

@@ -25,11 +25,12 @@ kill_timeout = "5s"
   OPTIMAP_EMAIL_HOST = "smtp.ionos.de"
   OPTIMAP_EMAIL_HOST_IMAP = "imap.ionos.de"
   OPTIMAP_EMAIL_HOST_USER = "login@optimap.science"
-  OPTIMAP_EMAIL_IMAP_SENT_FOLDER = "\"Gesendete Objekte\""
+  OPTIMAP_EMAIL_IMAP_SENT_FOLDER = "Gesendete Objekte"
   OPTIMAP_EMAIL_PORT_IMAP = "993"
   OPTIMAP_EMAIL_PORT_SMTP = "587"
   OPTIMAP_EMAIL_USE_TLS = "true"
-  OPTIMAP_LOGGING_CONSOLE_LEVEL = "DEBUG"
+  OPTIMAP_LOGGING_LEVEL = "DEBUG"
+  DJANGO_LOGGING_LEVEL = "WARNING"
   PORT = "8000"
 
 [[services]]

optimap/.env.example

@@ -23,6 +23,7 @@ OPTIMAP_EMAIL_PORT_IMAP=993
 OPTIMAP_EMAIL_USE_TLS=True
 OPTIMAP_EMAIL_IMAP_SENT_FOLDER=""
 
-OPTIMAP_LOGGING_CONSOLE_LEVEL=INFO
+OPTIMAP_LOGGING_LEVEL=INFO
+DJANGO_LOGGING_LEVEL=ERROR
 
 OPTIMAP_DATA_DUMP_RETENTION=3

optimap/__init__.py

@@ -1,2 +1,2 @@
-__version__ = "0.1.0"
+__version__ = "0.1.1"
 VERSION = __version__