Create walking skeleton for hsd_imaging

[jrhosk]

A sub-issue focusing specifically on the hsd_imaging task.

[jrhosk]

Skeleton code here.

[jrhosk]

Overview

With the latest commits, this is finished for now. The notebook demonstrates, given the current understanding of the graphviper workflow, how the hsd_imaging() task might be implemented. Within the notebook there is a section that provides a simple demonstration of building a graph with the provisional graph-building tool in toolviper. The main task is implemented at the end of the notebook and shows how the internal imaging functions would potentially be executed along the spw, field, and antenna axes. The internal functions are designed to simply have a sleep function, and for simulation and demonstration purposes, two additional features are added: first, a delay parameter can be passed to simulate a given field name being intrinsically slower than the rest. The goal here isn't so much the case of a field or antenna being slower but to show what might happen. were one of the nodes to be significantly slower than the others. Unfortunately, this could be fully explored because of the inability to make connected graphs and time. The second is a demonstration that was added to show how a file might be written when multiple nodes are run concurrently.

Questions and Addressable Issues

• The single-dish group would like to be able to make larger, more complicated connected graphs. For instance, a graph with axes as follows: [spw, field] ==> [spw, field, antenna] ==> [spw, field] with the final step being to gather the writes concurrently to the weblog. Part of the interest in the delayed response mentioned before is the case where several nodes are slower than others. A single gather means all of the code waits and waits for that single node to finish before proceeding. Whereas a more complicated graph can continue to process and write results while the offending node works. It is true that the Dask graph and execution can be done without the gather, in general, giving the ability to generate this kind of output, but to my knowledge, this can only be done after a single map. The point being, there is much more available that can be done with a Dash graph that we would like to explore.
• To our knowledge, there is no way to process multiple MSv4 concurrently in a single batch on a single graph. It is my understanding that this is something that will need to be done. Therefore, we would like to look into the direction of future development in regard to this. As a note, this doesn't imply running multiple MSv4s on different cluster nodes and gathering. The goal is to process multiple MSv4 as a single unit (an EB).