Inconsistencies along consecutive runs

[mmatera]

Running several times the same benchmark seems to produce quite different results. Here is an example of what happens if I run elementary.yaml
but with 1000 iterations (100 iterations seems to be worst):

I put this here, but maybe this is related to Mathics-core, where something similar happens when we call many times

mathics -e 'Timing[Do[<test expression>,{1000}]][[1]]' 

[mmatera]

Looking for a way to reduce the fluctuations, I found this: https://pythonspeed.com/articles/consistent-benchmarking-in-ci/

Here are the results when I fix the PYTHONHASHSEED and disable Linux ASLR:

In any case, to get more accurate benchmarks, we should run the benchmarks many times for each branch, and store the average time obtained from these many runs.

[TiagoCavalcante]

@mmatera I have seen that running the benchmarks in a container and not using your machine (or apt updateing it, ...) reduces the noise.

[mmatera]

@TiagoCavalcante, OK, but since I was not running anything else during the benchmark, this would imply that there is some kind of memory effect between runs in a physical machine, that does not happen in a container. Maybe this is also related to #44, isn't it?