Exceptions in student code accessed by fixtures are not reported to students

[charmoniumQ]

Defining the correct answer as a fixture is a useful pattern if it one needs to check the answer in multiple pytest tests. For example,

@pytest.fixture
def ans(sandbox: StudentFixture) -> ...:
    s = sandbox.query_setup("s")
    ...
    return # correct answer, could be a tuple of multiple parts

@pytest.mark.grading_data(name="Sigma", points=2)
def test_Sigma(sandbox: StudentFixture, ans: ...) -> None:
    np.testing.assert_allclose(sandbox.query("Sigma"), ans[0])

@pytest.mark.grading_data(name="X", points=2)
def test_X(sandbox: StudentFixture, ans: ...) -> None:
    np.testing.assert_allclose(sandbox.query("X"), ans[1])

# ... other similar tests follow

This generally works, but if the student code raises an Exception, students see,

Student code grading failed with an exception: Failed
Exception Message: Student code execution failed with an exception: TypeError

The backend stdout is:

info: container> _____________________ ERROR at setup of test_U[user_code] ______________________
info: container> Student code execution failed with an exception: TypeError
info: container> Exception Message: hello

Presumably because it fails in a fixture, in the setup, the backtrace doesn't get reported to the user?

Is accessing student code in a fixture a supported feature or not? It may be useful for questions in which there are multiple tests checking different aspects of essentially the same answer.

[charmoniumQ]

Unless that's how it's always supposed to be?

[eliotwrobson]

I'll play with this example more locally, but for this example, you should use the module_sandbox parameter name instead of the sandbox. This will prevent it from being restarted between the different test functions. Also, the stack trace default was changed to only show it in full if the marker is set to traceback: #13 (comment)

This does highlight that more information about the failure should be given to the output of the container however, so I'll make a change at some point that gives more debugging information to the console so that this is less confusing.

[charmoniumQ]

Also, the stack trace default was changed to only show it in full if the marker is set to traceback: #13 (comment)

Is this relating to stack traces in student code or in the pytest code?

[eliotwrobson]

Also, the stack trace default was changed to only show it in full if the marker is set to traceback: #13 (comment)

Is this relating to stack traces in student code or in the pytest code?

Stack traces from student code shown to the student