feat(core): add initial impl of task io service#34205
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
✅ Deploy Preview for nx-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
View your CI Pipeline Execution ↗ for commit b1f7987
☁️ Nx Cloud last updated this comment at |
🐳 We have a release for that!This PR has a release associated with it. You can try it out using this command: npx create-nx-workspace@0.0.0-pr-34205-7a03d44 my-workspaceOr just copy this version and use it in your own command: 0.0.0-pr-34205-7a03d44
To request a new release for this pull request, mention someone from the Nx team or the |
AgentEnder
force-pushed
the
feat/task-io-service
branch
from
7a03d44 to
1277938
Compare
![nx-cloud[bot]](https://avatars.githubusercontent.com/in/80458?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
| all_workspace_files: &'b [FileData], | ||
| ) -> napi::Result<impl ParallelIterator<Item = &'b FileData>> { | ||
| let globs = globs_from_workspace_inputs(workspace_file_sets); | ||
| let globs = expand_workspace_globs(workspace_file_sets); |
There was a problem hiding this comment.
Bad name from AI, not expanding glob, just removing {workspaceRoot}
|
|
||
| /// Expands project file set patterns by replacing `{projectRoot}` with the actual project root. | ||
| /// For root projects (project_root == "."), strips `{projectRoot}/` instead. | ||
| pub fn expand_project_globs(project_root: &str, file_sets: &[String]) -> Vec<String> { |
There was a problem hiding this comment.
globs_from_project_globs
🐳 We have a release for that!This PR has a release associated with it. You can try it out using this command: npx create-nx-workspace@0.0.0-pr-34205-604567c my-workspaceOr just copy this version and use it in your own command: 0.0.0-pr-34205-604567c
To request a new release for this pull request, mention someone from the Nx team or the |
🐳 We have a release for that!This PR has a release associated with it. You can try it out using this command: npx create-nx-workspace@0.0.0-pr-34205-6b29262 my-workspaceOr just copy this version and use it in your own command: 0.0.0-pr-34205-6b29262
To request a new release for this pull request, mention someone from the Nx team or the |
AgentEnder
force-pushed
the
feat/task-io-service
branch
from
6b29262 to
93f4176
Compare
AgentEnder
force-pushed
the
feat/task-io-service
branch
from
fd4e191 to
5a7204a
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
packages/nx/src/config/task-graph.ts
Outdated
| /** | ||
| * Structured inputs used for hashing (file paths, env vars, etc.) | ||
| */ | ||
| hashInputs?: { |
packages/nx/src/hasher/hash-task.ts
Outdated
| : hasher.hashTask(task, taskGraph, env)); | ||
| task.hash = value; | ||
| task.hashDetails = details; | ||
| task.hashInputs = inputs; |
packages/nx/src/hasher/hash-task.ts
Outdated
| } as any); | ||
| task.hash = value; | ||
| task.hashDetails = details; | ||
| task.hashInputs = inputs; |
packages/nx/src/hasher/hash-task.ts
Outdated
| for (let i = 0; i < tasksWithoutCustomHashers.length; i++) { | ||
| tasksWithoutCustomHashers[i].hash = hashes[i].value; | ||
| tasksWithoutCustomHashers[i].hashDetails = hashes[i].details; | ||
| tasksWithoutCustomHashers[i].hashInputs = hashes[i].inputs; |
| /** | ||
| * Structured inputs used for hashing | ||
| */ | ||
| export interface HashInputs { |
There was a problem hiding this comment.
Import this from rust
| * may have spawned multiple processes. To get all PIDs for a task, | ||
| * you need to correlate spawned processes via their parent PID. | ||
| */ | ||
| getAllTaskPids(): Map<string, number> { |
| * The task must have been hashed (task.hashInputs must be populated). | ||
| * Returns file inputs and outputs. | ||
| */ | ||
| getTaskIOInfo(task: string): TaskIOInfo | null { |
| * Get expanded output paths for a task. | ||
| * Resolves {projectRoot} and {workspaceRoot} placeholders. | ||
| */ | ||
| getExpandedOutputs(task: Task): string[] { |
| } | ||
| } | ||
|
|
||
| class DebugTaskIOService extends TaskIOService { |
There was a problem hiding this comment.
Remove this before landing
| export function getTaskIOService(): TaskIOService { | ||
| if (!instance) { | ||
| // TODO: Remove debug service option once stable | ||
| instance = process.env.NX_TASK_IO_DEBUG |
There was a problem hiding this comment.
Remove this before landing
AgentEnder
force-pushed
the
feat/task-io-service
branch
from
5a7204a to
3fa4326
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
AgentEnder
force-pushed
the
feat/task-io-service
branch
from
4c5cbba to
474dca6
Compare
There was a problem hiding this comment.
Important
At least one additional CI pipeline execution has run since the conclusion below was written and it may no longer be applicable.
Nx Cloud is proposing a fix for your failed CI:
These changes fix the test failures in project-configuration-utils.spec.ts by updating the error message to match the test expectations. The PR updated the test assertions to expect "at the beginning of a string" but the implementation code was not updated accordingly, causing the tests to fail with a mismatch between expected and actual error messages.
Tip
✅ We verified this fix by re-running nx:test.
diff --git a/packages/nx/src/project-graph/utils/project-configuration-utils.ts b/packages/nx/src/project-graph/utils/project-configuration-utils.ts
index 1a1c99b622..c6c36639be 100644
--- a/packages/nx/src/project-graph/utils/project-configuration-utils.ts
+++ b/packages/nx/src/project-graph/utils/project-configuration-utils.ts
@@ -1155,7 +1155,7 @@ export function resolveNxTokensInOptions<T extends Object | Array<unknown>>(
}
if (value.includes('{workspaceRoot}')) {
throw new Error(
- `${NX_PREFIX} The {workspaceRoot} token is only valid at the beginning of an option. (${key})`
+ `${NX_PREFIX} The {workspaceRoot} token is only valid at the beginning of a string. (${key})`
);
}
value = value.replace(/\{projectRoot\}/g, project.root);
Or Apply changes locally with:
npx nx-cloud apply-locally IaNW-EOc8
Apply fix locally with your editor ↗ View interactive diff ↗
🎓 Learn more about Self-Healing CI on nx.dev
AgentEnder
force-pushed
the
feat/task-io-service
branch
from
474dca6 to
a933748
Compare
|
Failed to publish a PR release of this pull request, triggered by @AgentEnder. |
🐳 We have a release for that!This PR has a release associated with it. You can try it out using this command: npx create-nx-workspace@0.0.0-pr-34205-a933748 my-workspaceOr just copy this version and use it in your own command: 0.0.0-pr-34205-a933748
To request a new release for this pull request, mention someone from the Nx team or the |
| class TaskIOService { | ||
| // Used to call subscribers that were late to the party | ||
| protected taskToPids: Map<string, number> = new Map(); | ||
| protected taskToInputs: Map<string, TaskInputInfo> = new Map(); |
There was a problem hiding this comment.
You need a 3rd for outputs right?
|
|
||
| trace!(parent: &span, "hash_tsconfig: {:?}", now.elapsed()); | ||
| ts_hash | ||
| (ts_hash, vec![]) |
There was a problem hiding this comment.
Should this not be the tsconfig file?
Update the native task hasher to return expanded file paths instead of glob patterns. This implements "glob once, use twice" optimization where files are collected during the hashing operation and returned alongside the hash value. - Add hash_project_files_with_inputs returning ProjectFilesHashResult - Add hash_workspace_files_with_inputs returning WorkspaceFilesHashResult - Update HashInputs to use files: Vec<String> instead of file_sets - Remove original hash_project_files/hash_workspace_files functions - Remove workspace_files_cache as it's no longer needed
Wire up the expanded hash inputs from the Rust hasher to the TaskIOService subscription system. This enables cloud consumption of task input file paths. - Update HashInputs interface to use files: string[] - Add subscribeToHashInputs/notifyHashInputs to TaskIOService - Call notifyHashInputs from hash-task.ts when inputs are computed - Update TaskIOInfo to use files instead of fileSets - Update test snapshots to expect expanded file paths
TaskOutput hashing now returns both the hash and the list of files that were actually hashed. This allows depOutputs in HashInputs to contain actual file paths instead of the task output string pattern. - Add TaskOutputHashResult struct with hash and files - Update task_hasher to use files from TaskOutput for depOutputs - Separate file tracking for WorkspaceFileSet/ProjectFileSet vs TaskOutput
… type - Import HashInputs from native module instead of defining manually - Remove hashInputs field from Task type (not needed externally) - Remove task.hashInputs assignments (data flows via TaskIOService) - Update snapshots for new file ordering in depOutputs
- Change init() to constructor with projectGraph/taskGraph parameters - Remove getter methods (getPidForTask, getAllTaskPids, getTaskIOInfo, getExpandedOutputs) - Remove outputGlobs from TaskIOInfo (outputs come from DbCache now) - Add notifyTaskOutputs and subscribeToTaskOutputs methods - Update DebugTaskIOService for new constructor pattern
…cribe Change subscribe() return type from ProcessMetricsService to void as method chaining is not used.
DbCache now calls TaskIOService.notifyTaskOutputs() after caching task outputs with the expanded file paths. This provides real file paths to subscribers instead of glob patterns with tokens.
The resolveNxTokensInString extraction is no longer needed since outputs will be real file paths from DbCache, not globs with tokens.
Rename subscribeToTaskIO to subscribeToTaskInputs, notifyTaskIO to notifyTaskInputs, and TaskIOInfo to TaskInputInfo for better clarity about what these methods handle.
AgentEnder
force-pushed
the
feat/task-io-service
branch
from
65f30e9 to
e485237
Compare
AgentEnder
force-pushed
the
feat/task-io-service
branch
from
e485237 to
09161b9
Compare
AgentEnder
force-pushed
the
feat/task-io-service
branch
from
09161b9 to
b1f7987
Compare
2 participants
Current Behavior
There's not an easy to use service to track PIDs being registered to nx tasks
Expected Behavior
There's a service to track this stuff
Example outputs of the debug logs are below:
Related Issue(s)
Fixes #