Test skipping works in a less then optimal way

[thibaultduponchelle]

It's possible to configure some tests to be skipped when smoking.

Depending on which test is to be skipped it uses a different implementation:

• for tests in t/ it alters the test file
• for tests in lib/, ext/, cpan/, dist/ it doesn't change the test file but instead it modifies the MANIFEST file that is in the root of the build directory.

This second method is less optimal, on my smoker I skipped a test in dist/ (because it fails on the system for unrelated reason);
The resulting smoke report:

Tests skipped on user request:
    dist/Time-HiRes/t/utime.t
Failures: (common-args) none
[default]
../t/porting/manifest.t.....................................FAILED
    10388, 10390-10391

What happened:

Test::Smoke::Smoker::set_skip_tests modified the MANIFEST file
t/porting/manifest.t checks if the MANIFEST contains all files; that is no longer the case since it was modified to remove some files from it.

I can think of several solutions (in no particular order)

1. Do nothing and leave it up to the user to also skip t/porting/manifest.t when skipping tests
2. Automatically skip t/porting/manifest.t when a test is being skipped
3. Alter Test::Smoke::Smoker::set_skip_tests to always modify the test file (i.e. remove the special case for lib|ext|cpan|dist)
4. t/harness which is used when running make test_harness does have some options to skip tests; but that might not be exposed via make test_harness (and isn't a solution for make test)
5. ...?