Adds Zenodo data archival functionality#249
Open
nuest wants to merge 4 commits into
Open
Conversation
Implements functionality to deposit OPTIMAP data to Zenodo by creating/updating draft records. This feature enables automated archival and versioning of research data for long-term preservation and citation. Features: - Two Django management commands: - `render_zenodo`: Generates metadata files and data archives - `deposit_zenodo`: Uploads files and merges metadata to Zenodo drafts - Updates existing drafts only (requires deposition ID) - Never publishes automatically - manual approval required in Zenodo UI - Uploads: README.md, optimap-main.zip, latest GeoJSON and GeoPackage files - Merges metadata non-destructively without overwriting stable fields - Configurable via environment variables (ZENODO_API_TOKEN, etc.) - Comprehensive test coverage for rendering and deposition New files: - works/management/commands/deposit_zenodo.py - Upload to Zenodo - works/management/commands/render_zenodo.py - Generate metadata/archives - works/templates/README.md.j2 - Jinja2 template for README - data/README.md, data/last_version.txt, data/zenodo_dynamic.json - tests/test_deposit_zenodo.py - Deposition tests - tests/test_render_zenodo.py - Render tests Modified files: - .gitignore - Ignore Zenodo artifacts - optimap/settings.py - Add Zenodo configuration - requirements.txt - Add zenodo-client, markdown, jinja2 dependencies This implementation is adapted from PR #214 to work with the refactored codebase (publications/ → works/ directory structure). Closes ifgi#63 Co-authored-by: BharatVe <bharatveauli@live.com> Co-authored-by: BharatVe <150399011+BharatVe@users.noreply.github.com> 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Adds comprehensive integration test suite for Zenodo deposition functionality with support for testing against the actual Zenodo sandbox API. Changes: - Fixed model references in tests (Publication → Work, publications → works) - Added tests/.env.template with configuration instructions - Created test_zenodo_integration.py with tagged integration tests - Tests can run against real Zenodo sandbox API with proper credentials - Added .env file to .gitignore to protect secrets Test categories: - Unit tests: Mock-based tests (existing) - Integration tests: Real API tests (new, tagged as 'integration') - Full deposit tests: End-to-end upload tests (tagged as 'slow' and 'upload') Usage: # Run only unit tests (no API calls): python manage.py test tests.test_deposit_zenodo tests.test_render_zenodo # Run integration tests (requires tests/.env): python manage.py test tests.test_zenodo_integration # Run specific test tags: python manage.py test --tag=integration python manage.py test --exclude-tag=slow Setup: 1. Copy tests/.env.template to tests/.env 2. Add Zenodo sandbox API token from https://sandbox.zenodo.org 3. Create a draft deposition and add its ID to .env 4. Run: python manage.py test tests.test_zenodo_integration 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Implements automated data archival to Zenodo for long-term preservation and citability. - Introduces a new `zenodo` app with functions for rendering metadata, depositing data, and managing Zenodo records. - Creates new management commands (`render_zenodo`, `deposit_zenodo`, and `zenodo_deposit`) for simplified workflow. - Adds a new `ZenodoDepositionLog` model to track deposition history and status. - Enhances the Django admin interface with actions to trigger depositions and view logs. - Includes comprehensive documentation in `README.md` on setting up and using the Zenodo integration.
Refs ifgi#63. - untrack data/README.md, data/zenodo_dynamic.json, data/last_version.txt (sandbox render output from local runs leaked into the branch); extend .gitignore to cover them plus CSV dump variants - fix the README.md.j2 sources loop — was unpacking dicts as (label, url) tuples so every entry rendered as "[name](url)" with no newline between items; iterate over Source dicts properly - switch tests/test_deposit_zenodo.py and tests/test_render_zenodo.py from unittest.TestCase to django.test.TestCase so the in-test ZenodoDepositionLog.save() and ORM-created Source rows hit a real test DB instead of crashing (deposit) or polluting the dev DB (render) - refresh the 0009 migration header timestamp - CHANGELOG entry under Unreleased describing the deposit groundwork
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enables automatic data package generation and deposition to Zenodo, ensuring long-term preservation and citability for OPTIMAP data.
This comprehensive feature introduces:
Data Package Generation
render_zenodocommand that builds essential artifacts: a dynamicREADME.md(generated from a Jinja2 template with live statistics and source information), anoptimap-main.zipcontaining the project's source code snapshot, and azenodo_dynamic.jsonfile for flexible metadata updates.last_version.txt.Zenodo Deposition Management
deposit_zenodocommand for updating existing Zenodo draft depositions. It intelligently merges metadata, protecting crucial identifiers like DOIs, and ensures a clean slate by deleting previous files before uploading new ones.zenodo_depositcommand simplifies the workflow by executing both the rendering and deposition steps sequentially.Logging, Monitoring, and Notifications
ZenodoDepositionLogmodel records every deposition attempt, tracking status, uploaded files, total size, duration, and any errors encountered./datapublic page now prominently displays information about the latest successful Zenodo deposition, with environment-aware display (sandbox in DEBUG, production otherwise).Streamlined Administration
Configuration
ZENODO_API_TOKEN,ZENODO_SANDBOX_DEPOSITION_ID,ZENODO_API_BASE) are introduced for flexible environment configuration, supported by atests/.env.template.Enhanced Testing
Relates to #63