Add Hand Reconstruction op (HaWoR)

[Qirui-jiao]

• video_hand_reconstruction_hawor_mapper: Use HaWoR and MoGe-2 for hand reconstruction.
  (The unittest is skipped because users need to download MANO_RIGHT.pkl.)

[gemini-code-assist]

Summary of Changes

Hello @Qirui-jiao, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly expands the video processing capabilities of the data_juicer library by introducing several new mappers. The core addition is a sophisticated 3D hand reconstruction operator utilizing the HaWoR and MoGe-2 models. Complementing this, new functionalities for static camera calibration (using both DeepCalib and MoGe-2) and video undistortion have been added, providing a more robust toolkit for preparing and analyzing complex video datasets. These additions are fully integrated with necessary configuration updates, model loading mechanisms, and comprehensive unit tests.

Highlights

• HaWoR Hand Reconstruction: Introduced a new video_hand_reconstruction_hawor_mapper that leverages HaWoR and MoGe-2 models for advanced 3D hand reconstruction from video data, providing detailed hand pose and shape parameters.
• Static Camera Calibration: Added two new mappers, video_camera_calibration_static_deepcalib_mapper and video_camera_calibration_static_moge_mapper, to accurately compute camera intrinsics and field of view (FOV) for static cameras using DeepCalib and the more precise MoGe-2 models, respectively.
• Video Undistortion: Implemented a video_undistort_mapper to correct raw video frames by applying camera intrinsics and distortion coefficients, producing undistorted video output.
• Infrastructure Integration: Integrated these new video processing mappers into the system's configuration (config_all.yaml), __init__.py imports, MetaKeys for consistent metadata tagging, and model_utils.py for streamlined model loading and management, including backup links for new models.
• Comprehensive Testing: Included dedicated unit tests for each new video processing mapper to ensure their functionality, accuracy, and reliability within the data processing pipeline.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces several new video processing mappers to data_juicer, including video_camera_calibration_static_deepcalib_mapper, video_camera_calibration_static_moge_mapper, video_hand_reconstruction_hawor_mapper, and video_undistort_mapper. It also modifies the configuration file and imports the new mappers. The review focuses on potential issues related to dependency management, code clarity, and the handling of external model paths, with a severity level of medium or higher.

[Qirui-jiao]

Updates on Feb 3:

• Merge main.
• Updated test.