[PULP-1117] Add task-specific log capturing to task diagnostics mechanism #7217
Conversation
dralley
force-pushed
the
task-logging
branch
2 times, most recently
from
5d9ca0f to
21eb4d6
Compare
dralley
force-pushed
the
task-logging
branch
2 times, most recently
from
f8b12d0 to
5532ddc
Compare
|
@bmbouter @mdellweg Since last push:
|
dralley
force-pushed
the
task-logging
branch
2 times, most recently
from
3b2aafa to
291e7b6
Compare
These logs just aren't very helpful, and they're extremely verbose, to the point where they inflate log size by orders of magnitude and meaningfully slow execution.
TASK_DIAGNOSTICS and X-Task-Diagnostics can now capture all logs for specific tasks. Because the logs are specific to the task, the logs are linear and not broken up by unrelated logs from other services/workers/tasks. Assisted By: Claude closes pulp#7215
Temporarily change the root logger's log level, but make sure that the console logger is still only logging at INFO level. closes pulp#7214
dralley
changed the title
There was a problem hiding this comment.
Not technically related, but let's just silently fix this (it's a separate commit anyway)...
There was a problem hiding this comment.
Right. I had seen the markdown-header- prefix thing elsewhere, but it turns out it is just unnecessary and actually doesn't work on this particular markdown flavor.
pulpcore/tasking/tasks.py
Outdated
| ): | ||
| """Create arguments for creation of a new task""" | ||
| current_task = Task.current() | ||
| profile_options = x_task_diagnostics_var.get(None) |
There was a problem hiding this comment.
x_task_diagnostics_var is a context var. Wouldn't it make sense to set it up again when entering the task context instead of applying extra logic for subtasks?
There was a problem hiding this comment.
Look at "temp commit", is that what you mean? (maybe that code would be moved to contexts.py, but in general principle)
pulpcore/tasking/_util.py
Outdated
| return __memray_diagnostic_decorator | ||
|
|
||
|
|
||
| def _logging_decorator(temp_dir, func): |
There was a problem hiding this comment.
I am still a little concerned that this will capture any logs happening during task execution.
As long as we call a task in a subprocess this will probably work. But we already have (fast, small) tasks "run in the forground" and we might introduce running async tasks concurrently. I don't have an answer for this ready.
There was a problem hiding this comment.
You bring up a decent point, I could try to test this with tasks that are forced to run in immediate mode.
3 participants
TASK_DIAGNOSTICS and X-Task-Diagnostics can now capture all logs for specific tasks and present those via the profile artifacts mechanism. Because the logs are specific to the task, the logs are linear and not broken up by unrelated logs from other services/workers/tasks.
Assisted By: Claude
closes #7214