additional clarifying and infrastructure details as suggestd by gemini and claude

Author: jlnav

AGENTS.md

@@ -3,51 +3,70 @@ Agent Contributor Guidelines and Information
 
 Read the ``README.rst`` for an overview of libEnsemble.
 
+- libEnsemble uses a manager-worker architecture. Points are generated by a generator and sent to a worker, which runs a simulator.
+- The manager determines how and when points get passed to workers via an allocation function.
+
 Repository Layout
 -----------------
 
-- ``libensemble/`` - source code.
--   ``/alloc_funcs`` - allocation functions. Policies for passing work between the manager and workers.
--   ``/comms`` - modules and abstractions for communication between the manager and workers.
--   ``/executors`` - an interface for launching executables, often simulations.
--   ``/gen_classes`` - generators that adhere to the `gest-api` standard.
+- ``libensemble/`` - Source code.
+-   ``/alloc_funcs`` - Allocation functions. Policies for passing work between the manager and workers.
+-   ``/comms`` - Modules and abstractions for communication between the manager and workers.
+-   ``/executors`` - An interface for launching executables, often simulations.
+-   ``/gen_classes`` - Generators that adhere to the `gest-api` standard.
                        Recommended over entries from ``/gen_funcs`` that perform similar functionality.
--   ``/gen_funcs`` - generator functions. Modules for producing points for simulations.
--   ``/resources`` - classes and functions for managing compute resources for MPI tasks, libensemble workers.
--   ``/sim_funcs`` - simulator functions. Modules for running simulations or performing experiments.
--   ``/tests`` - tests.
-    - ``/functionality_tests`` primarily tests libEnsemble code only.
-    - ``/regression_tests`` tests libEnsemble code with external code. Often more closely resembles actual use-cases.
-    - ``/unit_tests`` tests for individual modules.
--   ``/tools`` - tools. misc functions and classes to ease development.
--   ``/utils`` - utilities. misc functions and classes used internally by multiple modules.
+-   ``/gen_funcs`` - Generator functions. Modules for producing points for simulations.
+-   ``/resources`` - Classes and functions for managing compute resources for MPI tasks, libensemble workers.
+-   ``/sim_funcs`` - Simulator functions. Modules for running simulations or performing experiments.
+-   ``/tests`` - Tests.
+    - ``/functionality_tests`` - Primarily tests libEnsemble code only.
+    - ``/regression_tests`` - Tests libEnsemble code with external code. Often more closely resembles actual use-cases.
+    - ``/unit_tests`` - Tests for individual modules.
+-   ``/tools`` - Tools. Misc functions and classes to ease development.
+-   ``/utils`` - Utilities. Misc functions and classes used internally by multiple modules.
 -   ``ensemble.py`` - The primary interface for parameterizing and running libEnsemble.
--   ``generators.py`` - base classes for generators that adhere to the `gest-api` standard.
--   ``history.py`` - module for recording points that have been generated and simulation results. NumPy array.
+-   ``generators.py`` - Base classes for generators that adhere to the `gest-api` standard.
+-   ``history.py`` - Module for recording points that have been generated and simulation results. NumPy array.
 -   ``libE.py`` - libE main file. Previous primary interface for parameterizing and running libEnsemble.
--   ``logger.py`` - logging configuration
--   ``manager.py`` - module for maintaining the history array and passing points between the workers.
--   ``message_numbers.py`` - constants that represent states of the ensemble.
--   ``specs.py`` - Dataclasses for parameterizing the ensemble.
--   ``worker.py`` - module for running generators and simulators. Communicates with the manager.
--   ``version.py`` - version file.
+-   ``logger.py`` - Logging configuration
+-   ``manager.py`` - Module for maintaining the history array and passing points between the workers.
+-   ``message_numbers.py`` - Constants that represent states of the ensemble.
+-   ``specs.py`` - Dataclasses for parameterizing the ensemble. Most importantly, contains ``LibeSpecs, SimSpecs, GenSpecs``.
+-   ``worker.py`` - Module for running generators and simulators. Communicates with the manager.
+-   ``version.py`` - Version file.
 
 - ``.github/`` - GitHub actions. See ``.github/workflows/`` for the CI.
-- ``.docs/`` - Documentation. Check here first for information before reading the source code.
-- ``examples/`` - Symlinks to examples further inside the ``libensemble/`` directory.
+- ``docs/`` - Documentation. Check here first for information before reading the source code.
+- ``examples/`` - The ``*_funcs`` and ``calling_scripts`` directories contain symlinks to examples further in the source code.
+-   ``/libE_submission_scripts`` - Example scripts for submitting libEnsemble jobs to HPC systems.
+-   ``/tutorials`` - Tutorials on how to use libEnsemble.
+- ``pyproject.toml`` - Project configuration file. Contains information about the project and its dependencies.
 
 Other files in the root directory should be self-documenting.
 
 Familiarize yourself with ``libensemble/tests/regression_tests/test_1d_sampling.py`` for a simple example of the libEnsemble interface.
 
+Information about Generators
+----------------------------
+
+- Generators are functions or objects that produce points for simulations.
+- The History array is a numpy structured array that stores points that have been generated and simulation results. Its fields match ``sim_specs/gen_specs["out"]`` or ``vocs`` attributes.
+- Prior to libEnsemble v1.6.0, generators were plain functions. They often ran in "persistent" mode, meaning they executed in a long-running loop, sending and receiving points to and from the manager until the ensemble was complete.
+- A ``gest-api`` or "standardized" generator is a class that at a minimum implements ``suggest`` and ``ingest`` methods, and is parameterized by a ``vocs``.
+- See ``libensemble/generators.py`` for more information about the ``gest-api`` standard.
+
 General Guidelines
 ------------------
 
-- If using a generator that adheres to the `gest-api` standard, use the ``start_only_persistent`` allocation function.
+- If using classic ``sim_specs`` and ``gen_specs``, then ensure that ``sim_specs["out"]`` and ``gen_specs["in"]`` field names match, and vice-versa.
+- As-of libEnsemble v1.6.0, ``SimSpecs`` and ``GenSpecs`` can also be parameterized by a ``vocs`` object, imported from ``gest_api.vocs`` (NOT xopt.vocs).
+- ``VOCS`` contains variables, objectives, constraints, and other settings that define the problem. See ``libensemble/tests/regression_tests/test_xopt_EI.py`` for an example of how to use it.
+- If using a generator that adheres to the ``gest-api`` standard, or a classic persistent generator, use the ``start_only_persistent`` allocation function.
 - An MPI distribution is not required for libEnsemble to run, but is required to use the ``MPIExecutor``.  ``mpich`` is recommended.
 - New tests are heavily encouraged for new features, bug fixes, or integrations. See ``libensemble/tests/regression_tests`` for examples.
 - Never use destructive git commands unless explicitly requested.
 - Code is in the ``black`` style. This should be enforced by ``pre-commit``.
+- When writing new code, prefer the ``LibeSpecs``, ``SimSpecs``, and ``GenSpecs`` dataclasses over the classic ``sim_specs`` and ``gen_specs`` bare dictionaries.
 - Read ``CONTRIBUTING.md`` for more information.
 
 Development Environment
@@ -57,6 +76,7 @@ Development Environment
 of dependencies and the available testing environments.
 
 - Enter the development environment with ``pixi shell -e dev``. This environment contains the most common dependencies for development and testing.
+- For one-off commands, use ``pixi run -e dev``. This will run a single command in the development environment.
 
 - If ``pixi`` is not available or not preferred by the user, ``pip`` can be used instead, but the user will need to manually install other dependencies.
 
@@ -66,9 +86,11 @@ the configuration and ``pyproject.toml`` for other configuration.
 Testing
 -------
 
-- Run tests with the ``run-tests.py`` script.  See ``libensemble/tests/run-tests.py`` for usage information.
+- Run tests with the ``run-tests.py`` script: ``python libensemble/tests/run-tests.py``.  See ``libensemble/tests/run-tests.py`` for usage information.
 
 - Some tests require third party software to be installed. When developing a feature or fixing a bug the entire test suite may not necessarily need to be run,
 since it will be on Github Actions.
 
-- Individual unit tests can be run with ``pytest``.
+- Individual unit tests can be run with ``pixi run -e dev pytest path/to/test_file``.
+- A libEnsemble run typically outputs an ``ensemble.log`` and ``libE_stats.txt`` file in the working directory. Check these files for tracebacks or run statistics.
+- An "ensemble" or "workflow" directory may also be created, often containing per-simulation output directories