dependencies: do job and resource counts need to be incremented in SCHED state instead of RUN state?

[cmoussa1]

@ryanday36 and I were discussing offline this morning that the singleton dependency may or may not have allowed a user to inadvertently bypass certain per-queue resource and running job limits and take up all the resources on the system. The scenario looked something like the following:

• the queue had a per-association max nodes limit of X
• the user submitted a number of jobs (with a total node count > X), each with the singleton dependency, which means that the job is satisfied only when there are no other active jobs of the same userid and job name which are not already held with a singleton dependency.

If the singleton dependency jobs were submitted before any of them proceeded to RUN state (and thus, incremented running job and resource counts for the user in the priority plugin), then I think those jobs could potentially not have any flux-accounting dependencies applied to them, since none of the jobs had been running yet.

I wonder if perhaps incrementing the resource counts for a job as it enters SCHED state instead of RUN state would prevent this from happening in the future.

[garlick]

That makes sense! For that purpose it seems like you'd want to count a job against the runtime total as long as it's in SCHED, RUN, or CLEANUP. So add to the user's total on entry to SCHED, subtract on entry to INACTIVE.

Edit: another case: what if you remove the accounting dependency on a job but it remains in DEPEND state due to another dependency?

[cmoussa1]

Yeah, actually, after discussing with @ryanday36 about the use case specifically, it seems that incrementing these counts in SCHED instead of RUN might not actually solve the problem at hand. Here is a full description:

A max-nodes limit of 2 is set on the user. A number of jobs are submitted with the same job name and singleton dependencies so that only one runs at a time:

$ flux batch -N1 --dependency=singleton --job-name=job1 --wrap "sleep 5m"
f3JMFETdiKL3
$ flux batch -N1 --dependency=singleton --job-name=job1 --wrap "sleep 5m"
f3JMFF282aWK
$ flux batch -N1 --dependency=singleton --job-name=job1 --wrap "sleep 5m"
f3JMFFXZvK9Z
$ flux jobs
       JOBID QUEUE    USER     NAME       ST NTASKS NNODES     TIME INFO
f3JMFF282aWK pdev     day36    job1        D      1      1       4h depends:singleton
f3JMFFXZvK9Z pdev     day36    job1        D      1      1       4h depends:singleton
f3JMFETdiKL3 pdev     day36    job1        R      1      1   5.975s cluster1076

So, the second and third job1 jobs clear all of the flux-accounting dependencies (since the user is under the max-nodes limit when the second and third job1 jobs are submitted) and instead are just waiting under the singleton dependency.

A second set of jobs (under the name job2) are submitted, also with the singleton dependency. The first one starts, but the rest get both the singleton and the max-resources limit since the first job2 job took up the last available node for the user:

$ flux batch -N1 --dependency=singleton --job-name=job2 --wrap "sleep 5m"
f3JMFL1dLLFq
$ flux batch -N1 --dependency=singleton --job-name=job2 --wrap "sleep 5m"
f3JMFLaLzUyD
$ flux batch -N1 --dependency=singleton --job-name=job2 --wrap "sleep 5m"
f3JMFM1jABod
$ flux jobs
       JOBID QUEUE    USER     NAME       ST NTASKS NNODES     TIME INFO
f3JMFF282aWK pdev     day36    job1        D      1      1       4h depends:singleton
f3JMFFXZvK9Z pdev     day36    job1        D      1      1       4h depends:singleton
f3JMFLaLzUyD pdev     day36    job2        D      1      1       4h depends:singleton,max-resources-user-limit
f3JMFM1jABod pdev     day36    job2        D      1      1       4h depends:singleton,max-resources-user-limit
f3JMFL1dLLFq pdev     day36    job2        R      1      1   3.952s rzadams1078
f3JMFETdiKL3 pdev     day36    job1        R      1      1   16.54s rzadams1076

So, in this case, since the old job1 jobs had cleared all of the flux-accounting dependency checks when they were first submitted, the only dependency they are waiting on is the singleton dependency of the first job1 job, which means they could bypass any max-resources limit set for the user. And I don't think any flux-accounting dependencies can be retroactively placed on the job once it leaves DEPEND state, yes?

I think, ultimately, there needs to be a way for the flux-accounting jobtap plugin to update these singleton jobs with a flux-accounting dependency if they are sitting in DEPEND and flux-accounting limits are reached by other jobs that get submitted and can run immediately.