Tests failing depending on order they are run

[elichad]

We have drafted a profile using the must/ should/ may/ folder structure. However, we now want to "flatten" that structure, and collect related checks in top-level files (e.g. must/1_requesting_agent.ttl and should/1_requesting_agent.ttl get merged into a top-level 1_requesting_agent.ttl file, and so on).

However, when we do this, some of our tests start to fail in the CI. We can't figure out why...

For example, in this PR: eScienceLab#89
we have only merged files with the name 4_sign_off.ttl, but we see failures appear from tests which relate to the 1_root_data_entity_metadata.ttl 2_requesting_agent.ttl files, when neither those files nor the tests were changed.

Even stranger, the failures seem dependent on order in which the tests are run (and/or exactly which command is used to run them and/or the number of cores).

When I run locally:
pytest: 5 failures
pytest -k five-safes-crate (targeting just our profile): no failures
pytest -n 1 -k five-safes-crate (one worker only): no failures
pytest -v -n 1 tests/integration/profiles/five-safes-crate/test_5src_2_requesting_agent.py tests/integration/profiles/five-safes-crate/test_5src_4_signoff_phase.py (two specific test files): 3 failures from test_5src_2_requesting_agent.
pytest -v -n 1 tests/integration/profiles/five-safes-crate/test_5src_4_signoff_phase.py tests/integration/profiles/five-safes-crate/test_5src_2_requesting_agent.py (the same files in opposite order): no failures

and the CI for the PR has only 4 failures for pytest instead of 5.

@kikkomep I know you're not familiar with our profile but any help debugging this would be very welcome - we are completely stumped. If it's at all helpful, I also made a test branch where I "flattened" the workflow-ro-crate profile - https://github.com/eScienceLab/rocrate-validator/tree/upstream_merging_expt - and this also generated some inconsistent test errors (I got different errors with pytest vs pytest -k workflow-ro-crate).

I don't know if it's something to do with the mixed file structure, or something odd in how we have written certain checks or tests. As far as I know there were no such issues with the ISA profile (I helped develop it), which was flattened to begin with.