refactor: Reorganize project structure by moving documentation and generator/test files into dedicated subdirectories.

Author: dschwartz-ftadvisory

README.md

@@ -5,7 +5,7 @@
 
 # Rune Python Generator
 
-The Rune Python Generator creates Python code from [Rune](https://github.com/finos/rune-dsl).  It fully supports the Rune type syntax and has comprehensive [expression coverage](./EXPRESSION_SUPPORT.md), though function generation is not yet fully implemented.
+The Rune Python Generator creates Python code from [Rune](https://github.com/finos/rune-dsl).  It fully supports the Rune type syntax and has comprehensive [expression coverage](./docs/EXPRESSION_SUPPORT.md), though function generation is not yet fully implemented.
 
 The generated code requires the [RunePythonRuntime](https://github.com/finos/rune-python-runtime) library and Python 3.11+.
 
@@ -28,21 +28,19 @@ The Renovate bot automatically handles DSL updates by creating a pull request. M
 ## Repository Organization
 
 - `README.md` - this file, for documentation purposes
-- `BUILDANDTEST.md` - instructions on building and testing the repo
 - `RELEASE.md` - information about the current release
 - `src/main`  - Java/Xtend code to generate Python from Rune
 - `src/test`  - Java/Xtend code to run JUnit tests on the code generation process
-- `build` - configuration scripts to setup and tear down the Python unit testing environment
-- `build/build_cdm.sh` - used to create a Python package using CDM Rune definitions
-- `test` - Python unit tests and scripts to run the tests
+- `docs/BUILD_AND_TEST.md` - instructions on building and testing the repo
+- `test` - Python unit tests of the generated code
 
 ## Development setup
 
 ### For developers
 
 This guide is meant for everyone who wants to contribute to the Rune Python Generator and needs to get things up and running.
 
-Detailed build and testing instructions can be found in [BUILDANDTEST.md](./BUILDANDTEST.md)
+Detailed build and testing instructions can be found in [docs/BUILD_AND_TEST.md](./docs/BUILD_AND_TEST.md)
 
 Troubleshooting: If you run into issues, please open an [issue](https://github.com/finos/rune-python-generator/issues). This helps us improve the guide for everyone.
 
@@ -61,7 +59,7 @@ To build the project, run `mvn clean package`.
 
 #### UNIT Testing
 
-Building the project using Maven will run JUNIT-based unit tests.  All tests should pass.  To run the Python unit tests follow the instructions in [BUILDANDTEST.md](./BUILDANDTEST.md)
+Building the project using Maven will run JUNIT-based unit tests.  All tests should pass.  To run the Python unit tests follow the instructions in [docs/BUILD_AND_TEST.md](./docs/BUILD_AND_TEST.md)
 
 ### 2. Setting things up in Eclipse
 

BUILDANDTEST.md docs/BUILD_AND_TEST.mdBUILDANDTEST.md renamed to docs/BUILD_AND_TEST.md

@@ -65,4 +65,25 @@
 | RosettaNumberLiteral                   | ✓       | Decimal numeric literal value |[RosettaExpression.xcore#L73](https://github.com/finos/rune-dsl/blob/38dd20e8f3f8145588de7a9586972c84caa23951/rosetta-lang/model/RosettaExpression.xcore#L73) |
 | RosettaOnlyElement                     | ✓       | Unary list operation asserting the only element |[RosettaExpression.xcore#L300](https://github.com/finos/rune-dsl/blob/38dd20e8f3f8145588de7a9586972c84caa23951/rosetta-lang/model/RosettaExpression.xcore#L300) |
 | RosettaOnlyExistsExpression            | ✓       | Checks that only the listed expressions exist |[RosettaExpression.xcore#L247](https://github.com/finos/rune-dsl/blob/38dd20e8f3f8145588de7a9586972c84caa23951/rosetta-lang/model/RosettaExpression.xcore#L247) |
-| RosettaReference                       | ✓       | Abstract base for reference expressions |[RosettaExpression.xcore#L123](https://github.com/finos/rune-dsl/blob/38dd20e8f3f8145588de7a9586972c84caa23951/rosetta-lang/model/RosettaExpression.xcore#L123) |
+| RosettaReference                       | ✓       | Abstract base for reference expressions |[RosettaExpression.xcore#L123](https://github.com/finos/rune-dsl/blob/38dd20e8f3f8145588de7a9586972c84caa23951/rosetta-lang/model/RosettaExpression.xcore#L123) |
+
+## Proposed Changes to ExpressionGeneration
+
+The following changes are proposed to enhance the `ExpressionGeneration` functionality and address identified gaps:
+
+1.  **Direct Function Calling Support**: Implement support for `RosettaCallableWithArgs` to allow direct execution of other Rosetta functions within an expression.
+2.  **Date/Time & Math Standards**:  Create explicit tests and potentially refine implementation for `ToDate`, `ToDateTime`, `Min`, `Max`, `Sort`, and `Reverse` operations to ensure full compatibility with Python's standard library.
+3.  **List Operations**: Add comprehensive tests for `JoinOperation` and `ReverseOperation`.
+4.  **Refined Existence Logic**:  Investigate and correctly implement/test the `single exists` and `multiple exists` operators if present in the Rosetta DSL, ensuring they map correctly to Python logic (possibly extending `rune_attr_exists` or `rune_count`).
+5.  **Clean up `RosettaExistsExpressionTest`**:  Enable the disabled tests, add assertions, and migrate relevant test cases to the main test suite if they represent core functionality.
+
+## Evaluation of Current Testing
+
+### PythonExpressionGeneratorTest
+-   **Coverage**:  Strong coverage for arithmetic, boolean logic, basic list operations (`count`, `flatten`), `if-then-else`, and `switch`.
+-   **Gaps**:  Missing specific tests for `To*` conversions (Date/Time), `Min/Max/Sort`, and nested function calls.
+
+### RosettaExistsExpressionTest
+-   **Status**:  Currently disabled (`@Disabled`) and non-functional (no assertions).
+-   **Analysis**:  Contains valuable DSL examples for complex `exists` logic (`only exists`, `single exists`, `multiple exists` combined with `or`/`and`).
+-   **Plan**:  Migrate valid DSL cases to `PythonExpressionGeneratorTest`. Specifically, the complex boolean logic combinations should be asserted against expected Python output to verify that `rune_attr_exists` interacts correctly with Python's `and`/`or` operators. The `single/multiple` exists concepts need to be verified against the `PythonExpressionGenerator` implementation (or lack thereof) to decide if they are supported features or need implementation.

EXPRESSION_SUPPORT.md docs/EXPRESSION_SUPPORT.mdEXPRESSION_SUPPORT.md renamed to docs/EXPRESSION_SUPPORT.md

@@ -1,15 +1,15 @@
 package com.regnosys.rosetta.generator.python;
 // TODO: collect imports as a set rather than an array
+
 // TODO: re-engineer type generation to use an object that has the features carried throughout the generation (imports, etc.)
 // TODO: function support
 // TODO: review and consolidate unit tests
 // TODO: review migrating choice alias processor to PythonModelObjectGenerator
 
-
 import jakarta.inject.Inject;
 import com.regnosys.rosetta.generator.external.AbstractExternalGenerator;
 import com.regnosys.rosetta.generator.python.enums.PythonEnumGenerator;
-import com.regnosys.rosetta.generator.python.func.PythonFunctionGenerator;
+import com.regnosys.rosetta.generator.python.functions.PythonFunctionGenerator;
 import com.regnosys.rosetta.generator.python.object.PythonModelObjectGenerator;
 import com.regnosys.rosetta.generator.python.util.PythonCodeGeneratorUtil;
 
@@ -28,38 +28,49 @@
 import java.util.stream.Collectors;
 
 /**
- * PythonCodeGenerator is an external generator for the Rosetta DSL that produces Python code
- * from Rosetta model definitions. It supports the generation of Python classes, enums, and functions
+ * PythonCodeGenerator is an external generator for the Rosetta DSL that
+ * produces Python code
+ * from Rosetta model definitions. It supports the generation of Python classes,
+ * enums, and functions
  * based on the structure and semantics of the input Rosetta models.
  * <p>
- * This generator is designed to be used as part of the Rosetta code generation pipeline and is
- * typically invoked by the Rosetta build tools or CLI. It processes Rosetta models and outputs
- * Python source files, including project metadata such as <code>pyproject.toml</code>.
+ * This generator is designed to be used as part of the Rosetta code generation
+ * pipeline and is
+ * typically invoked by the Rosetta build tools or CLI. It processes Rosetta
+ * models and outputs
+ * Python source files, including project metadata such as
+ * <code>pyproject.toml</code>.
  * </p>
  *
  * <h2>Features</h2>
  * <ul>
- *   <li>Generates Python classes from Rosetta Data types</li>
- *   <li>Generates Python enums from Rosetta enumerations</li>
- *   <li>Generates Python functions from Rosetta function definitions</li>
- *   <li>Handles Rosetta model namespaces and organizes output into appropriate Python packages</li>
- *   <li>Produces project files for Python packaging (e.g., <code>pyproject.toml</code>)</li>
+ * <li>Generates Python classes from Rosetta Data types</li>
+ * <li>Generates Python enums from Rosetta enumerations</li>
+ * <li>Generates Python functions from Rosetta function definitions</li>
+ * <li>Handles Rosetta model namespaces and organizes output into appropriate
+ * Python packages</li>
+ * <li>Produces project files for Python packaging (e.g.,
+ * <code>pyproject.toml</code>)</li>
  * </ul>
  *
  * <h2>Usage</h2>
  * <p>
- * Typically, this class is not used directly, but is invoked by the Rosetta code generation
- * infrastructure. It can be integrated into build pipelines or called from a CLI tool.
+ * Typically, this class is not used directly, but is invoked by the Rosetta
+ * code generation
+ * infrastructure. It can be integrated into build pipelines or called from a
+ * CLI tool.
  * </p>
  *
  * <h2>Thread Safety</h2>
  * <p>
- * This class is not thread-safe and should be used in a single-threaded context.
+ * This class is not thread-safe and should be used in a single-threaded
+ * context.
  * </p>
  *
  * <h2>Extensibility</h2>
  * <p>
- * The generator is designed to be extensible. Additional features or customizations can be
+ * The generator is designed to be extensible. Additional features or
+ * customizations can be
  * implemented by extending this class or its collaborators.
  * </p>
  *
@@ -73,20 +84,23 @@ public class PythonCodeGenerator extends AbstractExternalGenerator {
 
     private static final Logger LOGGER = LoggerFactory.getLogger(PythonCodeGenerator.class);
 
-    @Inject private PythonModelObjectGenerator pojoGenerator;
-    @Inject private PythonFunctionGenerator funcGenerator;
-    @Inject private PythonEnumGenerator enumGenerator;
-    
+    @Inject
+    private PythonModelObjectGenerator pojoGenerator;
+    @Inject
+    private PythonFunctionGenerator funcGenerator;
+    @Inject
+    private PythonEnumGenerator enumGenerator;
+
     private List<String> subfolders;
     private Map<String, Map<String, CharSequence>> objects = null; // Python code for types by namespace, by type name
 
-
     public PythonCodeGenerator() {
         super("python");
     }
 
     @Override
-    public Map<String, ? extends CharSequence> beforeAllGenerate(ResourceSet set, Collection<? extends RosettaModel> models, String version) {
+    public Map<String, ? extends CharSequence> beforeAllGenerate(ResourceSet set,
+            Collection<? extends RosettaModel> models, String version) {
         subfolders = new ArrayList<>();
         pojoGenerator.beforeAllGenerate();
         objects = new HashMap<>();
@@ -113,9 +127,9 @@ public PythonCodeGenerator() {
                 .map(Function.class::cast).collect(Collectors.toList());
 
         if (!rosettaClasses.isEmpty() ||
-        	!metaDataItems.isEmpty() ||
-        	!rosettaEnums.isEmpty() || 
-        	!rosettaFunctions.isEmpty()) {
+                !metaDataItems.isEmpty() ||
+                !rosettaEnums.isEmpty() ||
+                !rosettaFunctions.isEmpty()) {
             addSubfolder(model.getName());
             if (!rosettaFunctions.isEmpty()) {
                 addSubfolder(model.getName() + ".functions");
@@ -127,8 +141,8 @@ public PythonCodeGenerator() {
         String namespace = PythonCodeGeneratorUtil.getNamespace(model);
         Map<String, CharSequence> currentObject = objects.get(namespace);
         if (currentObject == null) {
-        	currentObject = new HashMap<String, CharSequence>();
-        	objects.put(namespace, currentObject);
+            currentObject = new HashMap<String, CharSequence>();
+            objects.put(namespace, currentObject);
         }
         currentObject.putAll(pojoGenerator.generate(rosettaClasses, cleanVersion));
         result.putAll(enumGenerator.generate(rosettaEnums, cleanVersion));
@@ -139,9 +153,9 @@ public PythonCodeGenerator() {
 
     @Override
     public Map<String, ? extends CharSequence> afterAllGenerate(
-    		ResourceSet set,
-    		Collection<? extends RosettaModel> models, 
-    		String version) {
+            ResourceSet set,
+            Collection<? extends RosettaModel> models,
+            String version) {
         String cleanVersion = cleanVersion(version);
         Map<String, CharSequence> result = new HashMap<>();
 
@@ -153,7 +167,7 @@ public PythonCodeGenerator() {
             Map<String, CharSequence> currentObject = objects.get(namespace);
             if (currentObject != null && !currentObject.isEmpty()) {
                 result.put("pyproject.toml", PythonCodeGeneratorUtil.createPYProjectTomlFile(namespace, cleanVersion));
-                result.putAll (pojoGenerator.afterAllGenerate(namespace, currentObject));
+                result.putAll(pojoGenerator.afterAllGenerate(namespace, currentObject));
             }
         }
         return result;

src/main/java/com/regnosys/rosetta/generator/python/PythonCodeGenerator.java

@@ -1,4 +1,4 @@
-package com.regnosys.rosetta.generator.python.func;
+package com.regnosys.rosetta.generator.python.functions;
 
 import jakarta.inject.Inject
 import com.regnosys.rosetta.rosetta.RosettaEnumValueReference

…on/func/FunctionDependencyProvider.xtend …nctions/FunctionDependencyProvider.xtendsrc/main/java/com/regnosys/rosetta/generator/python/func/FunctionDependencyProvider.xtend renamed to src/main/java/com/regnosys/rosetta/generator/python/functions/FunctionDependencyProvider.xtend src/main/java/com/regnosys/rosetta/generator/python/func/FunctionDependencyProvider.xtend renamed to src/main/java/com/regnosys/rosetta/generator/python/functions/FunctionDependencyProvider.xtend

@@ -1,4 +1,4 @@
-package com.regnosys.rosetta.generator.python.func
+package com.regnosys.rosetta.generator.python.functions
 
 import jakarta.inject.Inject
 import com.regnosys.rosetta.generator.python.expressions.PythonExpressionGenerator