Phlex program startup

[knoepfel]

This is the central hub for discussing design issues for program startup, and related aspects of Phlex.

[knoepfel]

@marcpaterno, @beojan, and @knoepfel met April 9, 2026 to discuss program startup. My summary of the meeting:

1. The "Program Startup" section of the TDR will be adjusted so that output ports and input ports are connected by an edge when the product query of the input port matches the data product of the output port.
2. We had some discussion about the terms computational node, pure node, IO node, etc. That is not yet settled.
   • There is a similarity between Haskell's IO Monad and Phlex's driver and providers
3. A product query with an absent layer specification implies that any data products that match the product query will be presented to the node.
   • This is in contrast to a formerly suggested behavior, where a product query that resolves to multiple data products is an ambiguity.
4. We must be able to detect cycles among computational nodes (likely involves a third-party technology)
5. Forming the graph in Python may be easier than doing it in C++.
   • It may be possible to delay the construction of TBB flow-graph nodes until after the graph-formation mechanics are largely complete.
6. We must provide user-friendly error messages for users to easily understand how a program has been misconfigured.

[brettviren]

Perhaps this needs it's own thread, but @knoepfel what do you and others think about adding to the phlex command the ability to pass Jsonnet TLAs (both in string and "code" form)?

This is incredibly useful in wire-cell to pass things such as input/output file names or high level options. It avoids the need for ugly hacks of, eg, writing a shell script that bakes its CLI options into a generated config file.

The main problem I see with this idea is I read in the design doc that languages that do not support TLAs (YAML, JSON, FHiCL) are also supported. At worse, use of TLAs and these lesser languages would simply lead to phlex exiting with an error.

[brettviren]

In addition to supporting the passing of Jsonnet TLAs from phlex CLI to Jsonnet, I think phlex really needs a PATH like environment variable and CLI options to define a search path for Jsonnet files.

Wire-Cell Toolkit uses WIRECELL_PATH and the wire-cell CLI almost follows jsonnet in the options.

One caution: WIRECELL_PATH is "first one wins" while jsonnet has JSONNET_PATH which is "last one wins". I broke from jsonnet as "first one wins" is what we are used to with other PATH-like vars.

I would emphasize the use of TLA over "extVar" as the latter has no way to provide default values and thus requires specifying all variables that show up with std.extVar("name") in the entire Jsonnet corpus.

The "str" and "code" variants are needed at least to allow passing an integer value as an integer type. The "code" variant also allows passing aggregate types.

For reference, the CLI options for wire-cell and jsonnet:

$ wire-cell --help
Command line interface to the Wire-Cell Toolkit

Usage:
	wire-cell [options] [configuration ...]

Options:
  -h [ --help ]         produce help message
  -l [ --logsink ] arg  set log sink as <filename> or 'stdout' or 'stderr', a 
                        log level for the sink may be given by appending 
                        ':<level>'
  -L [ --loglevel ] arg set lowest log level for a log in form 'name:level' or 
                        just give 'level' value for all (level one of: 
                        critical,error,warn,info,debug,trace)
  -a [ --app ] arg      application component to invoke
  -c [ --config ] arg   provide a configuration file
  -p [ --plugin ] arg   specify a plugin as name[:lib]
  -V [ --ext-str ] arg  specify a Jsonnet external variable=<string>
  -C [ --ext-code ] arg specify a Jsonnet external variable=<code>
  -A [ --tla-str ] arg  specify a Jsonnet top level arguments variable=<string>
  --tla-code arg        specify a Jsonnet top level arguments variable=<code>
  -P [ --path ] arg     add to JSON/Jsonnet search path
  -t [ --threads ] arg  limit number of threads used
  -v [ --version ]      print the compiled version to stdout

$ jsonnet -h
Jsonnet commandline interpreter v0.20.0

jsonnet {<option>} <filename>

Available options:
  -h / --help             This message
  -e / --exec             Treat filename as code
  -J / --jpath <dir>      Specify an additional library search dir (right-most wins)
  -o / --output-file <file> Write to the output file rather than stdout
  -m / --multi <dir>      Write multiple files to the directory, list files on stdout
  -y / --yaml-stream      Write output as a YAML stream of JSON documents
  -S / --string           Expect a string, manifest as plain text
  -s / --max-stack <n>    Number of allowed stack frames
  -t / --max-trace <n>    Max length of stack trace before cropping
  --gc-min-objects <n>    Do not run garbage collector until this many
  --gc-growth-trigger <n> Run garbage collector after this amount of object growth
  --version               Print version
Available options for specifying values of 'external' variables:
Provide the value as a string:
  -V / --ext-str <var>[=<val>]     If <val> is omitted, get from environment var <var>
       --ext-str-file <var>=<file> Read the string from the file
Provide a value as Jsonnet code:
  --ext-code <var>[=<code>]    If <code> is omitted, get from environment var <var>
  --ext-code-file <var>=<file> Read the code from the file
Available options for specifying values of 'top-level arguments':
Provide the value as a string:
  -A / --tla-str <var>[=<val>]     If <val> is omitted, get from environment var <var>
       --tla-str-file <var>=<file> Read the string from the file
Provide a value as Jsonnet code:
  --tla-code <var>[=<code>]    If <code> is omitted, get from environment var <var>
  --tla-code-file <var>=<file> Read the code from the file
Environment variables:
JSONNET_PATH is a colon (semicolon on Windows) separated list of directories added
in reverse order before the paths specified by --jpath (i.e. left-most wins)
E.g. JSONNET_PATH=a:b jsonnet -J c -J d is equivalent to:
JSONNET_PATH=d:c:a:b jsonnet
jsonnet -J b -J a -J c -J d

[knoepfel]

@brettviren, thanks for your thoughts (https://github.com/orgs/Framework-R-D/discussions/506#discussioncomment-16658615, https://github.com/orgs/Framework-R-D/discussions/506#discussioncomment-16700045). They will serve as ingredients to the broader discussion regarding the configuration subsystem. It makes sense to me for Phlex to support what the underlying configuration languages are intended to support. But we'll have to make sure we get agreement among the relevant parties/stakeholders.

I will open a dedicated discussion re. the configuration subsystem and include the links to your comments.

[knoepfel]

@beojan, @marcpaterno, and @knoepfel met May 5, 2026 continuing the discussion mentioned above. Additional findings:

7. The data layers of the data products processed by an algorithm do not necessarily need to be known before forming the data-flow graph.
   1. So long as the fields provided in the product_query are sufficient to unambiguously connect one node to another, the data layer does not need to be known ahead of time.
   2. The user must specify a product's data layer (or equivalent information using the other product-query fields) to guarantee that data products associated with only that data layer will be processed by the algorithm.