Nextflow Tower

[bentsherman]

The developers of nextflow have also been working on a monitoring system called nextflow tower:

https://github.com/seqeralabs/nf-tower

I think there is some overlap with our nextflow api but I'm not sure the extent of it yet. We should look into this project and clarify how we are different. Don't want to be duplicating the wheel.

[bentsherman]

I got beta access to nextflow tower so I was able to test it myself. Basically, you get an API token which gives you access to Seqera Lab's main instance of nf-tower, then when you run nextflow with the -with-tower option, nextflow will stream information about the workflow run to said nf-tower instance. From there you can monitor the workflow run from the web browser, and all of this information is persistent so you can refer to it even after the workflow has finished.

The web browser interface is largely analogous to what we have for nextflow-api, but it is much more detailed. Whereas we simply show the log in real time, nf-tower displays the same information but as a dashboard, and it also shows performance and utilization stats. In this respect nf-tower is much nicer as it is able to extract a wider variety of data directly from nextflow. We could display the trace data but probably only after the fact.

However, the key difference between nextflow-api and nf-tower is that nf-tower does not allow you to launch workflows, it is only a monitoring dashboard. It's nice because it can connect to nextflow anywhere, be it local or hpc or cloud, etc. But the user still has to set up the infrastructure required to run nextflow wherever they're going to run it. So there may actually be a way to synergize here.

The core of nextflow-api is still the management of workflow runs, which includes:

• uploading input data and configuration
• launching, resuming, (eventually) cancelling
• downloading output data

Currently the only piece of overlap with nf-tower is the fact that we stream the log in real time. So it might make sense to simply add an option to enable nf-tower and then users could open the monitoring dashboard in a separate window. The log file provides the url.

If that is the case, then we can probably move forward with implementing a database to store user credentials and performance data. I will reach out to the nf-tower folks to see if I can get more information about planned features.

[cbmckni]

Just saw this, but as of a few weeks ago I can confirm that Nextflow Tower works with Nextflow-API!

(Nextflow Tower now works with the K8s executor, which was not the case when I initially tested Tower)