Skip to content

Use ctest to Run SystemTests on the CI#40356

Open
cailafinn wants to merge 12 commits intomantidproject:mainfrom
cailafinn:34850-ci-using-ctest
Open

Use ctest to Run SystemTests on the CI#40356
cailafinn wants to merge 12 commits intomantidproject:mainfrom
cailafinn:34850-ci-using-ctest

Conversation

@cailafinn
Copy link
Contributor

@cailafinn cailafinn commented Nov 18, 2025 •
edited
Loading

Description of work

Following on from #40296 - now is the step where we replace the old system test runner with ctest in the CI scripts.
This PR is also where we'll fix any failing tests.

Closes #34850

To test:

  1. Check all tests run on the CI.
  2. Check all tests pass on the CI.

Build from Branch:
Build Status

Reviewer

Your comments will be used as part of the gatekeeper process. Comment clearly on what you have checked and tested during your review. Provide an audit trail for any changes requested.

As per the review guidelines:

  • Is the code of an acceptable quality? (Code standards/GUI standards)
  • Has a thorough functional test been performed? Do the changes handle unexpected input/situations?
  • Are appropriately scoped unit and/or system tests provided?
  • Do the release notes conform to the guidelines and describe the changes appropriately?
  • Has the relevant (user and developer) documentation been added/updated?
  • If the PR author isn’t in the mantid-developers or mantid-contributors teams, add a review comment rerun ci to authorize/rerun the CI

Gatekeeper

As per the gatekeeping guidelines:

  • Has a thorough first line review been conducted, including functional testing?
  • At a high-level, is the code quality sufficient?
  • Are the base, milestone and labels correct?

@cailafinn cailafinn added the Maintenance Unassigned issues to be addressed in the next maintenance period. label Nov 18, 2025
@cailafinn cailafinn added this to the Release 6.15 milestone Nov 18, 2025
@github-actions github-actions bot added the Has Conflicts Used by the bot to label pull requests that have conflicts label Nov 25, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Nov 25, 2025

👋 Hi, @cailafinn,

Conflicts have been detected against the base branch. Please rebase your branch against the base branch.

@cailafinn cailafinn force-pushed the 34850-ci-using-ctest branch from 8b22bb1 to 49112d8 Compare December 19, 2025 11:17
@github-actions github-actions bot removed the Has Conflicts Used by the bot to label pull requests that have conflicts label Dec 19, 2025
@sf1919
Copy link
Contributor

sf1919 commented Jan 22, 2026

What is happening with this one? It's still in draft - do we need to change the milestone?

@cailafinn cailafinn modified the milestones: Release 6.15, Release 6.16 Jan 22, 2026
cailafinn and others added 9 commits February 6, 2026 10:05
@cailafinn
Use ctest for systemtests in CI.
5f9f9d3 
RE mantidproject#34850
@cailafinn
Fix odd pathing on SANS tests
0a061d0 
RE mantidproject#34850
@cailafinn @perenon
Fix DrILLProcessTest
9f7ea39 
RE mantidproject#34850

Co-authored-by: Remi Perenon <perenon@users.noreply.github.com>
@cailafinn
Fix incorrect initalisation of some tests
ee2faea 
RE mantidproject#34850
@cailafinn
Fix SaveNexusTest
33acee8 
RE mantidproject#34850
@cailafinn
Try not extending sys.path all the time
d426956 
RE mantidproject#34850
@cailafinn
Avoid adding to the data search dirs too much
f9215f9 
It seems that the setup step can end up being run multiple times
between each cleanup, resulting in the size of the dict doubling
each time and taking exponentially longer to append to.

RE mantidproject#34850
@cailafinn
Clean the reduction singleton between test runs
bcc1cf1 
The ctest implementation results in the ReductionSingleton being
carried over between tests within a single test file. The singleton
has a Clean() function that needs to be called at the start of each
test - and probably should have always been anyway.

RE mantidproject#34850
@cailafinn
Add some more data search dirs
744c7c8 
RE mantidproject#34850
@cailafinn cailafinn force-pushed the 34850-ci-using-ctest branch from 0e88419 to 744c7c8 Compare February 6, 2026 10:05
@cailafinn cailafinn marked this pull request as ready for review February 6, 2026 13:28
@cailafinn cailafinn force-pushed the 34850-ci-using-ctest branch from c215061 to 3de1b7f Compare February 6, 2026 14:18
@cailafinn cailafinn force-pushed the 34850-ci-using-ctest branch from 3de1b7f to b86085e Compare February 6, 2026 15:52
@MialLewis MialLewis requested a review from jhaigh0 February 9, 2026 10:07
jhaigh0
jhaigh0 reviewed Feb 9, 2026
View reviewed changes
@jhaigh0
Copy link
Contributor

jhaigh0 commented Feb 9, 2026

Could you launch a jenkins build_branch job where you run the system tests on all platforms? (maybe you have already done this while testing?)

@cailafinn cailafinn force-pushed the 34850-ci-using-ctest branch from cc9d123 to a3a619f Compare February 11, 2026 14:16
@jhaigh0 jhaigh0 self-assigned this Feb 11, 2026
@jhaigh0
Copy link
Contributor

jhaigh0 commented Feb 12, 2026 •
edited
Loading

Looks like the test failures in the build_branch job are normally time-outs (lasting exactly 20mins). Tom suggests setting the time-out limit for system tests to be higher. It also seems like only 292 system tests are being run, I thought there were more (~700?).

@cailafinn
Copy link
Contributor Author

cailafinn commented Feb 13, 2026

Looks like the test failures in the build_branch job are normally time-outs (lasting exactly 20mins). Tom suggests setting the time-out limit for system tests to be higher. It also seems like only 292 system tests are being run, I thought there were more (~700?).

Yes, there are - but as mentioned in this comment, the system tests are now run on a file-basis rather than a class-basis. Since most system test files contain multiple classes the apparent number of system tests drops at the high level.

@jhaigh0
Copy link
Contributor

jhaigh0 commented Feb 13, 2026

Looks like the test failures in the build_branch job are normally time-outs (lasting exactly 20mins). Tom suggests setting the time-out limit for system tests to be higher. It also seems like only 292 system tests are being run, I thought there were more (~700?).

Yes, there are - but as mentioned in this comment, the system tests are now run on a file-basis rather than a class-basis. Since most system test files contain multiple classes the apparent number of system tests drops at the high level.

Ah got it, thanks. That might further the case for upping the time-out limit then.

@cailafinn
Increase timeout to 30 mins
9434e34 
The systemtests are longer now. Since they're run on a per-file basis
rather than a per-class basis some of the longer-running tests have
more of a tendancy to time out before completing.

RE mantidproject#34850
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jhaigh0 jhaigh0 jhaigh0 left review comments

At least 1 approving review is required to merge this pull request.

Assignees

@jhaigh0 jhaigh0

Labels

Maintenance Unassigned issues to be addressed in the next maintenance period.

Projects

None yet

Milestone

Release 6.16

Development

Successfully merging this pull request may close these issues.

Investigate using ctest to run system tests

3 participants

@cailafinn @sf1919 @jhaigh0