Feat/uepr 535 introduce script for migrating packages to monorepo#505
Feat/uepr 535 introduce script for migrating packages to monorepo#505KManolov3 wants to merge 3 commits into
Conversation
| run: npm publish --access=public --tag="$NPM_TAG" --ignore-scripts --workspace=@scratch/task-herder | ||
|
|
||
| - name: Publish scratch-media-lib-scripts | ||
| run: | |
There was a problem hiding this comment.
Those were some leftovers from the publish update PR, the build and token set happen earlier currently.
Test report for scratch-vm 1 files ±0 886 suites ±0 1m 59s ⏱️ -1s Results for commit 725c499. ± Comparison against base commit ace70ed. This pull request removes 2 and adds 2 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
| -not -path '*/.git/*' \ | ||
| -not -path '*/node_modules/*' \ | ||
| -exec grep -Il "$REPO_NAME" {} \; 2>/dev/null | while read -r file; do | ||
| sed -i '' -e "s:\(require(\|from \|resolve(\|node_modules\)\(['\"/]\)${REPO_NAME}\(['\"/]\):\1\2${NPM_ORGANIZATION}/${REPO_NAME}\3:g" "$file" |
There was a problem hiding this comment.
sed -i '' -e seems to be a macOS-only syntax. Consider using perl -pi -e
|
|
||
| # Rewrite all paths in the cloned repo to live under a subdirectory | ||
| # Handles repos with and without submodules | ||
| move_repository_subdirectory() { |
There was a problem hiding this comment.
This was taken from the build-monorepo script. We don't (yet) have a case of repo with submodules.
| ) "${PACKAGE_PATH}/package.json" | sponge "${PACKAGE_PATH}/package.json" | ||
| fi | ||
|
|
||
| ### Step 5: Update root workspaces list ### |
There was a problem hiding this comment.
Consider using npm init as an alternative (as in https://github.com/scratchfoundation/scratch-editor/pull/434/changes#diff-c0835ea4a8034dbcc4211a854aa9ee66ca48777eca6294856e5e02e60ad0b9fdR1-R358)
kbangelov
left a comment
There was a problem hiding this comment.
Looks good to me overall! Left a few small comments.
| --source-branch) | ||
| SOURCE_BRANCH="$2" | ||
| shift 2 | ||
| ;; | ||
| --org) | ||
| GITHUB_ORG="$2" | ||
| shift 2 | ||
| ;; | ||
| --cache-dir) | ||
| BUILD_CACHE="$2" | ||
| shift 2 |
There was a problem hiding this comment.
Perhaps we should consider adding checks for whether the second argument exists and whether it is not another flag, which would lead to nonsense instead of error.
if [ -z "$2" ] || [[ "$2" == --* ]]; then
echo "Error: --source-branch requires a value" >&2
usage 1
fi
| fi | ||
|
|
||
| if [ -d "$PACKAGE_PATH" ]; then | ||
| echo "Error: ${PACKAGE_DIR} already exists in the monorepo." >&2 |
| for DEP in $DEPS; do | ||
| npm install --force --save --save-exact \ | ||
| "${NPM_ORGANIZATION}/${DEP}" -w "${NPM_ORGANIZATION}/${PACKAGE}" \ | ||
| || package_replacement_error "$PACKAGE" "$DEP" | ||
| done | ||
| for DEP in $DEVDEPS; do | ||
| npm install --force --save-dev --save-exact \ | ||
| "${NPM_ORGANIZATION}/${DEP}" -w "${NPM_ORGANIZATION}/${PACKAGE}" \ | ||
| || package_replacement_error "$PACKAGE" "$DEP" | ||
| done | ||
| for DEP in $OPTDEPS; do | ||
| npm install --force --save-optional --save-exact \ | ||
| "${NPM_ORGANIZATION}/${DEP}" -w "${NPM_ORGANIZATION}/${PACKAGE}" \ | ||
| || package_replacement_error "$PACKAGE" "$DEP" | ||
| done | ||
| for DEP in $PEERDEPS; do | ||
| npm install --force --save-peer --save-exact \ | ||
| "${NPM_ORGANIZATION}/${DEP}" -w "${NPM_ORGANIZATION}/${PACKAGE}" \ | ||
| || package_replacement_error "$PACKAGE" "$DEP" | ||
| done |
There was a problem hiding this comment.
I wonder if this might lead to a situation where the npm install fails but package_replacement_error just logs the error and continues.
| run: npm publish --access=public --tag="$NPM_TAG" --ignore-scripts --workspace=@scratch/task-herder | ||
|
|
||
| - name: Publish scratch-media-lib-scripts | ||
| run: | |
| echo "***ERROR***" | ||
| echo "Could not replace a dependency with the local monorepo version." | ||
| echo "Failed to replace $3 in $1#$2" | tee -a "monorepo.errors.log" | ||
| #exit 1 # uncomment this to make it a fatal error |
There was a problem hiding this comment.
nicpick: Should we actually uncomment this? Or maybe make it a flag that is passed to the script? Because I think right now we could get to a point where something fails to install correctly, but the script still succeeds. Which makes it easier for the person running the script to miss it?
…and #505 Replaces the script work of two prior PRs that independently solved "add an existing scratchfoundation repo to the monorepo": scripts/add-repo.sh — collapses add-to-monorepo.sh (#434) and add-repo.sh (#505) into a single canonical entry point. Takes the CLI shape and auto-CI-regen from #505 and the source-branch auto-detect (develop -> main -> master), normalization (sort-package-json), and clean-tree pre-flight from #434. scripts/update-gha-workflows.ts — replaces the CircleCI-era build-gha-workflows.ts and workspace-template.yml. Regenerates .github/path-filters.yml and incrementally updates publish.yml. Open review comments on #505 are addressed: - sed -i portability bug: rewrites now use perl -pi -e (works on macOS, NixOS, Ubuntu/Debian/Arch, WSL). Perl is added to the explicit prereq check next to git-filter-repo, sponge, and jq. - Arg parser now rejects flag-stealing: --source-branch --org foo errors instead of silently using --org as the branch name. - npm-install failures hard-fail by default. --continue-on-error opts into the prior soft-fail behavior; failures are still logged to add-repo.errors.log when continuing. - The submodules-aware branch in move_repository_subdirectory keeps a TODO comment marking that it has never been exercised against a real submodules-bearing repo and should be run carefully when one lands. No repo additions are included; this PR is script-tooling only.
Replaces the script work of two prior PRs that independently solved "add an existing scratchfoundation repo to the monorepo": scripts/add-repo.sh — collapses add-to-monorepo.sh (#434) and add-repo.sh (#505) into a single canonical entry point. Takes the CLI shape and auto-CI-regen from #505 and the source-branch auto-detect (develop -> main -> master), normalization (sort-package-json), and clean-tree pre-flight from #434. scripts/update-gha-workflows.ts — replaces the CircleCI-era build-gha-workflows.ts and workspace-template.yml. Regenerates .github/path-filters.yml and incrementally updates publish.yml. Open review comments on #505 are addressed: - sed -i portability bug: rewrites now use perl -pi -e (works on macOS, NixOS, Ubuntu/Debian/Arch, WSL). Perl is added to the explicit prereq check next to git-filter-repo, sponge, and jq. - Arg parser now rejects flag-stealing: --source-branch --org foo errors instead of silently using --org as the branch name. - npm-install failures hard-fail by default. --continue-on-error opts into the prior soft-fail behavior; failures are still logged to add-repo.errors.log when continuing. - The submodules-aware branch in move_repository_subdirectory keeps a TODO comment marking that it has never been exercised against a real submodules-bearing repo and should be run carefully when one lands. Cross-workspace dep handling: rewires both bare-name and already-prefixed @scratch/<name> deps in the new package to exact-pinned current monorepo versions. Matches the existing convention in scratch-gui/scratch-vm/ scratch-render. npm's workspace: protocol would be the ideal here but npm does not actually support it (npm/cli#8845 - EUNSUPPORTEDPROTOCOL at install time). Workspaces array ordering: the new package is inserted just after the latest existing workspace it depends on, so 'npm run --workspaces build' builds its deps first. Falls back to position 0 (prepend) when the new package has no monorepo deps. This is a "last dep wins" heuristic, not a full topo sort; a real topo sort can be added later by extending update-gha-workflows.ts if needed. bash 3.2 compatibility: macOS ships bash 3.2 at /bin/bash, which lacks the 'mapfile' builtin. The find/grep output is collected via a 'while IFS= read -r' loop so the script runs on macOS as well as Linux. No repo additions are included; this PR is script-tooling only.
…and #505 Replaces the script work of two prior PRs that independently solved "add an existing scratchfoundation repo to the monorepo": scripts/add-repo.sh — collapses add-to-monorepo.sh (#434) and add-repo.sh (#505) into a single canonical entry point. Takes the CLI shape and auto-CI-regen from #505 and the source-branch auto-detect (develop -> main -> master), normalization (sort-package-json), and clean-tree pre-flight from #434. scripts/update-gha-workflows.ts — replaces the CircleCI-era build-gha-workflows.ts and workspace-template.yml. Regenerates .github/path-filters.yml and incrementally updates publish.yml. Open review comments on #505 are addressed: - sed -i portability bug: rewrites now use perl -pi -e (works on macOS, NixOS, Ubuntu/Debian/Arch, WSL). Perl is added to the explicit prereq check next to git-filter-repo, sponge, and jq. - Arg parser now rejects flag-stealing: --source-branch --org foo errors instead of silently using --org as the branch name. - npm-install failures hard-fail by default. --continue-on-error opts into the prior soft-fail behavior; failures are still logged to add-repo.errors.log when continuing. - The submodules-aware branch in move_repository_subdirectory keeps a TODO comment marking that it has never been exercised against a real submodules-bearing repo and should be run carefully when one lands. Cross-workspace dep handling: rewires both bare-name and already-prefixed @scratch/<name> deps in the new package to exact-pinned current monorepo versions. Matches the existing convention in scratch-gui/scratch-vm/ scratch-render. npm's workspace: protocol would be the ideal here but npm does not actually support it (npm/cli#8845 - EUNSUPPORTEDPROTOCOL at install time). Workspaces array ordering: the new package is inserted just after the latest existing workspace it depends on, so 'npm run --workspaces build' builds its deps first. Falls back to position 0 (prepend) when the new package has no monorepo deps. This is a "last dep wins" heuristic, not a full topo sort; a real topo sort can be added later by extending update-gha-workflows.ts if needed. bash 3.2 compatibility: macOS ships bash 3.2 at /bin/bash, which lacks the 'mapfile' builtin. The find/grep output is collected via a 'while IFS= read -r' loop so the script runs on macOS as well as Linux. No repo additions are included; this PR is script-tooling only.
Replaces the script work of two prior PRs that independently solved "add an existing scratchfoundation repo to the monorepo": scripts/add-repo.sh — collapses add-to-monorepo.sh (#434) and add-repo.sh (#505) into a single canonical entry point. Takes the CLI shape and auto-CI-regen from #505 and the source-branch auto-detect (develop -> main -> master), normalization (sort-package-json), and clean-tree pre-flight from #434. scripts/update-gha-workflows.ts — replaces the CircleCI-era build-gha-workflows.ts and workspace-template.yml. Regenerates .github/path-filters.yml and incrementally updates publish.yml. Open review comments on #505 are addressed: - sed -i portability bug: rewrites now use perl -pi -e (works on macOS, NixOS, Ubuntu/Debian/Arch, WSL). Perl is added to the explicit prereq check next to git-filter-repo, sponge, and jq. - Arg parser now rejects flag-stealing: --source-branch --org foo errors instead of silently using --org as the branch name. - npm-install failures hard-fail by default. --continue-on-error opts into the prior soft-fail behavior; failures are still logged to add-repo.errors.log when continuing. Scoped to the per-dep rewire loop; the final lockfile install always hard-fails by design. - The submodules-aware branch in move_repository_subdirectory keeps a TODO comment marking that it has never been exercised against a real submodules-bearing repo and should be run carefully when one lands. Cross-workspace dep handling: rewires both bare-name and already-prefixed @scratch/<name> deps in the new package to exact-pinned current monorepo versions. Matches the existing convention in scratch-gui/scratch-vm/ scratch-render. npm's workspace: protocol would be the ideal here but npm does not actually support it (npm/cli#8845 - EUNSUPPORTEDPROTOCOL at install time). Workspaces array ordering: the new package is inserted just after the latest existing workspace it depends on, so 'npm run --workspaces build' builds its deps first. Falls back to position 0 (prepend) when the new package has no monorepo deps. This is a "last dep wins" heuristic, not a full topological sort; it relies on the existing workspaces array already being in valid build order. A real topo sort can be added later (e.g. by extending update-gha-workflows.ts) if it ever becomes necessary. bash 3.2 compatibility: macOS ships bash 3.2 at /bin/bash, which lacks the 'mapfile' builtin. The find/grep output is collected via a 'while IFS= read -r' loop so the script runs on macOS as well as Linux. Bare-clone alternates path: bare clones store the alternates file at objects/info/alternates, not .git/objects/info/alternates. The previous path was a no-op; the working disconnection comes from --dissociate in the clone command. Path corrected so the line is defensive belt-and- braces rather than dead code. update-gha-workflows.ts: throws with a clear error if npm query returns a workspace not present in the root "workspaces" array (previously would silently misorder via indexOf returning -1). The comment in resolveWorkspaces is corrected to reflect that the function returns Map insertion order, with the caller re-sorting into declared order. No repo additions are included; this PR is script-tooling only.
…and #505 Replaces the script work of two prior PRs that independently solved "add an existing scratchfoundation repo to the monorepo": scripts/add-repo.sh — collapses add-to-monorepo.sh (#434) and add-repo.sh (#505) into a single canonical entry point. Takes the CLI shape and auto-CI-regen from #505 and the source-branch auto-detect (develop -> main -> master), normalization (sort-package-json), and clean-tree pre-flight from #434. scripts/update-gha-workflows.ts — replaces the CircleCI-era build-gha-workflows.ts and workspace-template.yml. Regenerates .github/path-filters.yml and incrementally updates publish.yml. Open review comments on #505 are addressed: - sed -i portability bug: rewrites now use perl -pi -e (works on macOS, NixOS, Ubuntu/Debian/Arch, WSL). Perl is added to the explicit prereq check next to git-filter-repo, sponge, and jq. - Arg parser now rejects flag-stealing: --source-branch --org foo errors instead of silently using --org as the branch name. - npm-install failures hard-fail by default. --continue-on-error opts into the prior soft-fail behavior; failures are still logged to add-repo.errors.log when continuing. Scoped to the per-dep rewire loop; the final lockfile install always hard-fails by design. - The submodules-aware branch in move_repository_subdirectory keeps a TODO comment marking that it has never been exercised against a real submodules-bearing repo and should be run carefully when one lands. Cross-workspace dep handling: rewires both bare-name and already-prefixed @scratch/<name> deps in the new package to exact-pinned current monorepo versions. Matches the existing convention in scratch-gui/scratch-vm/ scratch-render. npm's workspace: protocol would be the ideal here but npm does not actually support it (npm/cli#8845 - EUNSUPPORTEDPROTOCOL at install time). Workspaces array ordering: the new package is inserted just after the latest existing workspace it depends on, so 'npm run --workspaces build' builds its deps first. Falls back to position 0 (prepend) when the new package has no monorepo deps. This is a "last dep wins" heuristic, not a full topological sort; it relies on the existing workspaces array already being in valid build order. A real topo sort can be added later (e.g. by extending update-gha-workflows.ts) if it ever becomes necessary. bash 3.2 compatibility: macOS ships bash 3.2 at /bin/bash, which lacks the 'mapfile' builtin. The find/grep output is collected via a 'while IFS= read -r' loop so the script runs on macOS as well as Linux. Bare-clone alternates path: bare clones store the alternates file at objects/info/alternates, not .git/objects/info/alternates. The previous path was a no-op; the working disconnection comes from --dissociate in the clone command. Path corrected so the line is defensive belt-and- braces rather than dead code. update-gha-workflows.ts: throws with a clear error if npm query returns a workspace not present in the root "workspaces" array (previously would silently misorder via indexOf returning -1). The comment in resolveWorkspaces is corrected to reflect that the function returns Map insertion order, with the caller re-sorting into declared order. No repo additions are included; this PR is script-tooling only.
Replaces the script work of two prior PRs that independently solved "add an existing scratchfoundation repo to the monorepo": scripts/add-repo.sh — collapses add-to-monorepo.sh (#434) and add-repo.sh (#505) into a single canonical entry point. Takes the CLI shape and auto-CI-regen from #505 and the source-branch auto-detect (develop -> main -> master), normalization (sort-package-json), and clean-tree pre-flight from #434. scripts/update-gha-workflows.ts — replaces the CircleCI-era build-gha-workflows.ts and workspace-template.yml. Regenerates .github/path-filters.yml and incrementally updates publish.yml. Open review comments on #505 are addressed: - sed -i portability bug: rewrites now use perl -pi -e (works on macOS, NixOS, Ubuntu/Debian/Arch, WSL). Perl is added to the explicit prereq check next to git-filter-repo, sponge, and jq. - Arg parser now rejects flag-stealing: --source-branch --org foo errors instead of silently using --org as the branch name. - npm-install failures hard-fail by default. --continue-on-error opts into the prior soft-fail behavior; failures are still logged to add-repo.errors.log when continuing. Scoped to per-dep package.json rewrite failures in the rewire step; the final lockfile install always hard-fails by design. - The submodules-aware branch in move_repository_subdirectory keeps a TODO comment marking that it has never been exercised against a real submodules-bearing repo and should be run carefully when one lands. Cross-workspace dep handling: rewires both bare-name and already-prefixed @scratch/<name> deps in the new package to exact-pinned current monorepo versions. Matches the existing convention in scratch-gui/scratch-vm/ scratch-render. npm's workspace: protocol would be the ideal here but npm does not actually support it (npm/cli#8845 - EUNSUPPORTEDPROTOCOL at install time). Workspaces array ordering: the new package is inserted just after the latest existing workspace it depends on, so 'npm run --workspaces build' builds its deps first. Falls back to position 0 (prepend) when the new package has no monorepo deps. This is a "last dep wins" heuristic, not a full topological sort; it relies on the existing workspaces array already being in valid build order. A real topo sort can be added later (e.g. by extending update-gha-workflows.ts) if it ever becomes necessary. bash 3.2 compatibility: macOS ships bash 3.2 at /bin/bash, which lacks the 'mapfile' builtin. The find/grep output is collected via a 'while IFS= read -r' loop so the script runs on macOS as well as Linux. Bare-clone alternates path: bare clones store the alternates file at objects/info/alternates, not .git/objects/info/alternates. The previous path was a no-op; the working disconnection comes from --dissociate in the clone command. Path corrected so the line is defensive belt-and- braces rather than dead code. update-gha-workflows.ts: throws with a clear error if npm query returns a workspace not present in the root "workspaces" array (previously would silently misorder via indexOf returning -1). The comment in resolveWorkspaces is corrected to reflect that the function returns Map insertion order, with the caller re-sorting into declared order. The npm query exec call now requests a 64 MiB maxBuffer; Node's default 1 MiB is fine for the current repo size but would silently truncate as the monorepo grows. Path anchoring: BUILD_TMP, BUILD_CACHE, and any user-supplied --cache-dir value are resolved to absolute paths up front. A CWD-relative default would have put add-repo.tmp outside the repo (and outside the .gitignore pattern) if the script were run from a subdirectory of the monorepo. No repo additions are included; this PR is script-tooling only.
…and #505 Replaces the script work of two prior PRs that independently solved "add an existing scratchfoundation repo to the monorepo": scripts/add-repo.sh — collapses add-to-monorepo.sh (#434) and add-repo.sh (#505) into a single canonical entry point. Takes the CLI shape and auto-CI-regen from #505 and the source-branch auto-detect (develop -> main -> master), normalization (sort-package-json), and clean-tree pre-flight from #434. scripts/update-gha-workflows.ts — replaces the CircleCI-era build-gha-workflows.ts and workspace-template.yml. Regenerates .github/path-filters.yml and incrementally updates publish.yml. Open review comments on #505 are addressed: - sed -i portability bug: rewrites now use perl -pi -e (works on macOS, NixOS, Ubuntu/Debian/Arch, WSL). Perl is added to the explicit prereq check next to git-filter-repo, sponge, and jq. - Arg parser now rejects flag-stealing: --source-branch --org foo errors instead of silently using --org as the branch name. - npm-install failures hard-fail by default. --continue-on-error opts into the prior soft-fail behavior; failures are still logged to add-repo.errors.log when continuing. Scoped to per-dep package.json rewrite failures in the rewire step; the final lockfile install always hard-fails by design. - The submodules-aware branch in move_repository_subdirectory keeps a TODO comment marking that it has never been exercised against a real submodules-bearing repo and should be run carefully when one lands. Cross-workspace dep handling: rewires both bare-name and already-prefixed @scratch/<name> deps in the new package to exact-pinned current monorepo versions. Matches the existing convention in scratch-gui/scratch-vm/ scratch-render. npm's workspace: protocol would be the ideal here but npm does not actually support it (npm/cli#8845 - EUNSUPPORTEDPROTOCOL at install time). Workspaces array ordering: the new package is inserted just after the latest existing workspace it depends on, so 'npm run --workspaces build' builds its deps first. Falls back to position 0 (prepend) when the new package has no monorepo deps. This is a "last dep wins" heuristic, not a full topological sort; it relies on the existing workspaces array already being in valid build order. A real topo sort can be added later (e.g. by extending update-gha-workflows.ts) if it ever becomes necessary. bash 3.2 compatibility: macOS ships bash 3.2 at /bin/bash, which lacks the 'mapfile' builtin. The find/grep output is collected via a 'while IFS= read -r' loop so the script runs on macOS as well as Linux. Bare-clone alternates path: bare clones store the alternates file at objects/info/alternates, not .git/objects/info/alternates. The previous path was a no-op; the working disconnection comes from --dissociate in the clone command. Path corrected so the line is defensive belt-and- braces rather than dead code. update-gha-workflows.ts: throws with a clear error if npm query returns a workspace not present in the root "workspaces" array (previously would silently misorder via indexOf returning -1). The comment in resolveWorkspaces is corrected to reflect that the function returns Map insertion order, with the caller re-sorting into declared order. The npm query exec call now requests a 64 MiB maxBuffer; Node's default 1 MiB is fine for the current repo size but would silently truncate as the monorepo grows. Path anchoring: BUILD_TMP, BUILD_CACHE, and any user-supplied --cache-dir value are resolved to absolute paths up front. A CWD-relative default would have put add-repo.tmp outside the repo (and outside the .gitignore pattern) if the script were run from a subdirectory of the monorepo. No repo additions are included; this PR is script-tooling only.
TODO:
Resolves
Part of https://scratchfoundation.atlassian.net/browse/UEPR-535
Proposed Changes
Add two scripts -
add-repo.shandrefresh-gh-workflowsto assist in adding new packages to monorepo.Reason for Changes
We want to expand the scratch-editor monorepo with other packages that make sense to be there (e.g. scratch-paint, scratch-audio, ...)
Test Coverage
This is for development use, and the results of it should always undergo manual developer review, so I'm unsure if automated tests make sense in terms of effort.