Update validator API to get outcome

[Stevendeo]

This is a "for later" problem. No need to change that API until then.

Originally posted by @nberth in #22 (comment)

We could benefit from getting the actual results of the test validation (at the tool level) instead of relying on tools', which sometimes are incomplete.

For example, when cbmc (assert mode) returns a test covering label 1, it will only state the test covers the first label.

[nberth]

Note: it will be clearer if the outcome of a test is retrieved from the corpus instead of the validator.

[Stevendeo]

I'm not sure to understand, isn't the validator part of the corpus?

[nberth]

I'm not sure to understand, isn't the validator part of the corpus?

No it's designed as a separate component as it is essential to the consistency of both the corpus and the store. For now it's a module of seacoral-corpus (for historical reasons), but for clarity it could/should be in its own library (or maybe better seacoral-store could be merged into seacoral-corpus, and that latter library be renamed into something like seacoral-generation (or -gen maybe, to stay concise — seacoral-core is already used but that could be a good name for that lib).

[nberth]

For the short term if you want the outcome back from validation, I think it's okay-ish to provide the validation routines with an optional callback (outcome -> unit Lwt.t) so it does not interfere with existing uses of the validator, and these routines remain unit Lwt.t (to clearly denote they have side-effects).

[Stevendeo]

For the short term if you want the outcome back from validation, I think it's okay-ish to provide the validation routines with an optional callback (outcome -> unit Lwt.t) so it does not interfere with existing uses of the validator, and these routines remain unit Lwt.t (to clearly denote they have side-effects).

What's the difference between the use of a callback in this case and its use in my previous PR where working on parsed data through a continuation was bad? In this case I really don't want a callback that returns unit Lwt.t, the goal is to use each outcome as information to treat the next tests, and I'd rather not add mutexes when I already can treat the outcomes as they come.

Now I think of it, I could however push the outcome in a stream in the continuation... to which I would say, it would be way clearer without continuation.

[nberth]

Using an outcome returned from the validator to decide on next actions may be misleading for several reasons:

• at the moment you receive it many other tests have been validated (several validations occur in parallel, with tests provided by any other analysis tool) ;
• the outcome only gives you feedback on past history and the last test submitted by the analysis tool; if the next test output by the tool comes much later you may not have up-to-date information on current coverage.

Instead, what would make more sense is to retrieve current coverage information (possibly extended with tool-related metadata) from the store (and/or maybe other info from the corpus) at the moment you need to take decisions on a new test produced by the analysis tool.