Skip to content

add env worker logging#1561

Merged
samsja merged 10 commits intomainfrom
daniel/env-worker-logs
Jan 13, 2026
Merged

add env worker logging#1561
samsja merged 10 commits intomainfrom
daniel/env-worker-logs

Conversation

@rasdani
Copy link
Contributor

@rasdani rasdani commented Jan 8, 2026

enable in orch.toml like this

writes to run_0/env_workers/ENV_NAME/worker_X.log

[env.log]
level = "debug"
vf_level = "debug"

Note

Introduces optional, per-environment worker logging and routes both prime-rl and verifiers logs to per-worker files.

  • Config: New EnvLogConfig and orchestrator.env.log field with level and vf_level; if unset, logging stays disabled
  • Scheduler/Orchestrator: Scheduler now receives output_dir; creates get_env_worker_log_dir(output_dir, env_name) and passes log_level, vf_log_level, log_file to EnvWorker
  • EnvWorker: Worker process resets and sets up logger; redirects verifiers to the same log file; accepts new logging args
  • Utils: Added get_env_worker_log_dir in utils/pathing.py
  • Docs: Updated CHANGELOG.md with orchestrator.env.log entry

Written by Cursor Bugbot for commit 347d19c. This will update automatically on new commits. Configure here.

@rasdani rasdani marked this pull request as ready for review January 8, 2026 20:54
@rasdani rasdani marked this pull request as draft January 9, 2026 09:57
@rasdani rasdani force-pushed the daniel/env-worker-logs branch from e7e400f to 25fe4ee Compare January 13, 2026 11:35
@rasdani rasdani marked this pull request as ready for review January 13, 2026 15:31
Documents the new EnvLogConfig class and log field added to EnvConfig,
which provides logging configuration for environment workers.
When logging is disabled (default), reset_logger() sets _LOGGER to None
but setup_logger() was not called. This caused EventLoopLagMonitor to
crash with RuntimeError when calling get_logger().

Fix: Always set up a logger in worker processes. When logging is disabled,
use CRITICAL level which produces no output (critical logging is disabled
in setup_logger).
rasdani and others added 2 commits January 13, 2026 21:24
The FileHandler added for the verifiers logger was missing a Formatter,
causing log messages to be written as raw text without timestamps, log
levels, or logger names. This makes the verifiers logs difficult to
correlate or debug effectively.

Added a Formatter with timestamp (HH:MM:SS) and log level formatting,
matching the format used by the prime-rl logger for consistency.
cursoragent and others added 3 commits January 13, 2026 16:14
On Linux with fork start method, worker subprocesses inherit _LOGGER
from parent. Both branches need reset_logger() before setup_logger()
to avoid RuntimeError("Logger already set").
Copy link
Member

@mikasenghaas mikasenghaas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

log_file: str | None,
):
"""Main entry point for worker process."""
# Reset logger inherited from parent process, then setup fresh logger for this worker
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lol the logger singleton is shared across mp boundaries?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually would surprise me too

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but i needed reset_logger() for sure.

samsja pushed a commit that referenced this pull request Jan 13, 2026
samsja pushed a commit that referenced this pull request Jan 13, 2026
Copy link
Member

@samsja samsja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@samsja samsja merged commit 3436f42 into main Jan 13, 2026
8 checks passed
@cursor cursor bot mentioned this pull request Jan 15, 2026
4 tasks
Jackmin801 pushed a commit that referenced this pull request Jan 15, 2026
* add env worker logging

* reset logger before env logger

* redirect verifiers logger to file

* Add CHANGELOG entry for orchestrator.env.log configuration

Documents the new EnvLogConfig class and log field added to EnvConfig,
which provides logging configuration for environment workers.

* Fix worker crash when logging disabled

When logging is disabled (default), reset_logger() sets _LOGGER to None
but setup_logger() was not called. This caused EventLoopLagMonitor to
crash with RuntimeError when calling get_logger().

Fix: Always set up a logger in worker processes. When logging is disabled,
use CRITICAL level which produces no output (critical logging is disabled
in setup_logger).

* fix

* Add formatter to verifiers logger FileHandler

The FileHandler added for the verifiers logger was missing a Formatter,
causing log messages to be written as raw text without timestamps, log
levels, or logger names. This makes the verifiers logs difficult to
correlate or debug effectively.

Added a Formatter with timestamp (HH:MM:SS) and log level formatting,
matching the format used by the prime-rl logger for consistency.

* Add missing reset_logger() in else branch

On Linux with fork start method, worker subprocesses inherit _LOGGER
from parent. Both branches need reset_logger() before setup_logger()
to avoid RuntimeError("Logger already set").

* Revert "Fix worker crash when logging disabled"

This reverts commit f37ab4b.

* refactor

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants