Test against (bad) custom data sources

[corranwebster]

What is the problem or limitation you are having?

Currently the testbed tests for the widgets which use data sources are all using the default list and tree sources, which are fairly robust. But they are intended to be used with user-supplied custom data sources as well. While a well-written data source should perform properly, it is likely that developers writing their own data sources will make mistakes. In these cases good feedback about the problem is preferable to a hard crash which may be difficult to interpret.

Describe the solution you'd like

If there were tests of data sources which are bad in some ways (eg. failing to notify when the data changes, or raising an exception when asked for data) that verify that these are handled gracefully.

Exactly what "gracefully" means isn't entirely clear to me - crashing is OK, for example, as long as there is information provided as part of a traceback or log that can help pin-point what went wrong. The complexity is that a lot of this code is called as callbacks from C/C++, and worst case is that you get a crash at that layer with no diagnostics.

Describe alternatives you've considered

Not doing anything is an option. It may be difficult to get a consistent behaviour for all backends that is testable.

Additional context

This came up from discussion in #4081.

[freakboy3742]

Agreed this would be a good idea. If were in a position to provide a suite of tests that third-parties could use to verify that their data sources was compliant, that would be even better - but defining a suite at that level would be even more difficult as it needs to be adaptive to a wider range of possible inputs.