Add package-spec install flow with UniDep artifact metadata#276
Open
basnijholt wants to merge 22 commits intomainfrom
Open
Add package-spec install flow with UniDep artifact metadata#276basnijholt wants to merge 22 commits intomainfrom
basnijholt wants to merge 22 commits intomainfrom
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #276 +/- ##
==========================================
+ Coverage 98.07% 98.37% +0.30%
==========================================
Files 11 13 +2
Lines 2285 2711 +426
==========================================
+ Hits 2241 2667 +426
Misses 44 44 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Critical fixes: 1. Extra dependency encoding: filter pip entries subsumed by conda in select_unidep_dependencies to prevent duplicate pip+conda installs when an extra moves a dep across channels. 2. Defer python_executable/conda_run resolution until after conda install so package-spec installs don't crash for not-yet-created target envs. 3. Remove duplicated _dedupe from _index_install; import from _artifact_metadata. 4. Move CondaExecutable type alias to _artifact_metadata as single source of truth; _cli and _index_install import from there. 5. Remove duplicated _build_pip_install_command from _cli; delegate to _index_install version. Warning fixes: 1. Add dry-run disclaimer in _download_package_artifact noting that metadata cannot be inspected without downloading. 2. Remove pointless _classify_install_targets wrapper from _cli; call _index_install.classify_install_targets directly. 3. Fix fallback_flags: only use --no-deps when no_dependencies is set, not when --skip-pip is set alone.
Critical fixes: - Hatch build hook: use self.metadata.version instead of build variant string - argparse: parse install targets as str, not Path, to preserve URL specifiers - Marker-gated requirements: return None instead of RuntimeError when pip downloads nothing (e.g. markers evaluate to false) Warning fixes: - Extras lookup: normalise extra names per PEP 685 (case/separator insensitive) - Catch zipfile.BadZipFile during wheel metadata extraction - Remove dead/misleading pip_names code in select_unidep_dependencies - Add tests for URL specifier preservation, marker-gated fallback, extras normalisation, and BadZipFile handling
Critical fixes: 1. Move _dedupe to utils.dedupe (public) — eliminates cross-module private function import between _artifact_metadata and _index_install. 2. Pin with_metadata installs to exact metadata version — prevents version drift between inspected artifact and installed artifact for range specifiers (e.g. pkg>=1.0). 3. When --skip-conda is set, move packages with conda deps to pip fallback — prevents silently leaving dependencies missing when the guided recovery path (--skip-conda) is followed. Warning fixes: 1. Add tests for Hatch UnidepBuildHook (new test_hatch_build_hook.py). 2. Extract duplicated _Metadata/_Distribution/_Cmd test helpers to module-level _StubMetadata/_StubDistribution/_StubCmd classes in test_setuptools_integration.py. 3. Replace hardcoded "linux-64" default platform in InstallRuntime fallback with real identify_current_platform from utils.
Critical: Extra with no delta on current platform was incorrectly treated as 'missing', causing base conda deps to be silently discarded in favor of a pip-only fallback. Now distinguish 'extra not defined at all' (truly missing → fallback) from 'extra defined but no contribution on this platform' (empty delta → proceed with base deps). Warning: PEP 685-normalized extra-name collisions (e.g. dev-extra and dev_extra) were silently overwritten in the normalised lookup. Now validated at parse time with an UnidepMetadataError on collision.
Critical: Zero-delta extras (whose deps duplicate the base set) were dropped from metadata by the `if extra_platform_payload:` guard in build_unidep_metadata. At install time, the missing extra was misclassified as undefined, causing the entire package to fall back to pip-only and silently losing base conda deps. Fix: always record the extra in the payload, even when the per-platform delta is empty. Warning: Document the local_dependencies limitation in build_unidep_metadata's docstring — it always resolves local deps recursively but does not emit PyPI alternative package references, which diverges from get_python_dependencies/install_requires behavior when UNIDEP_SKIP_LOCAL_DEPS is set.
Fix conda→pip dependency movement lost by extras in metadata selection. The dedup rule in select_unidep_dependencies now tracks which channel an extra contributed deps to. When an extra adds a pip entry for a name that already exists in base conda (and does NOT also add it to conda), the extra's pip version takes precedence — removing the base conda entry. Previously the dedup rule always preferred conda, silently dropping pip entries that extras intentionally introduced to replace conda deps.
Validate metadata.project against req.name using canonicalize_name() before constructing the --no-deps install spec. On mismatch, fall back to plain pip install instead of risking installing a different package. Use req.name (the user's requested name) instead of metadata.project when building the pinned spec, so normalisation differences (e.g. underscores vs hyphens) never cause the wrong package to be resolved. Added regression tests for both the mismatch fallback and the normalisation-compatible match case.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Installing Python packages via
pipis not Conda-aware. For mixed dependency stacks (Conda + pip), this can pull the wrong runtime variant (for example CPU-only where Conda would resolve CUDA-enabled packages).This made dependency management via source checkouts/submodules attractive, but heavy.
Solution
Teach
unidep install <package-spec>to resolve dependencies from package artifacts.Each built wheel can embed a
unidep.jsonmanifest. At install time, UniDep inspects that metadata and performs a Conda-first installation flow before installing the package itself.10,000-foot Flow
setuptools: writes.dist-info/unidep.jsonhatchling: writes.dist-info/extra_metadata/unidep.jsonunidep install "pkg==X"(orpkg @ URL/path):unidep.json--no-depsExact Use Cases
What Changed
unidep install <package-spec>).unidep.json.Notes / Current Limitation
--dry-runcurrently does not perform full artifact inspection, so dry-run output may show pip fallback path even when metadata is available.Validation
test_cli,test_setuptools_integration,test_artifact_metadata).📚 Documentation preview 📚: https://unidep--276.org.readthedocs.build/en/276/