Metadata Handling Improvements
- Fixed duplicate fields in latest.json by removing redundant "latest_incremental_" fields
- Improved JSON field ordering for better readability
- Added genesis.tar.bz2 auto-upload functionality
Retention Management
- Implemented configurable count-based retention (keep N most recent snapshots)
- Added time-based retention (keep snapshots newer than X hours)
- Modified cleanup to run after each successful upload
- Added support for multi-instance retention management
Performance Optimizations
- Custom upload concurrency and chunk size parameters
- Fixed cleanup operations that would hang indefinitely
- Implemented timeouts for S3 operations
- Added detailed logging throughout the process
Error Handling
- Added timeout contexts to prevent hanging operations
- Enhanced error reporting and recovery mechanisms
- Implemented graceful handling of S3 API failures
Code Organization
- Improved code structure and readability
- Added comprehensive logging with duration tracking
- Better error handling and context propagation