[feature request] Support runtime test skipping for platform-dependent tests (xUnit v3 upgrade)

[sandy2008]

Package

OpenTelemetry

Is your feature request related to a problem?

Yes. The test suite has several categories of tests that need conditional skipping, but not all can be properly handled with xUnit 2.x's discovery-time Skip mechanism.

Current Test Skipping Patterns in the Codebase

1. Discovery-time skipping (working correctly):

• [Fact(Skip = "Known issue.")] - Static skip for known issues (e.g., MetricApiTests.cs:880)
• [SkipUnlessEnvVarFoundFact] / [SkipUnlessEnvVarFoundTheory] - Custom attributes that set Skip property at discovery time based on environment variables

2. Compile-time platform targeting (working correctly):

• #if NET / #if NETFRAMEWORK preprocessor directives exclude code at compile time

3. Runtime platform detection (problematic):

These tests need to detect platform capabilities at runtime, which xUnit 2.x doesn't properly support:

Location	Issue
OtlpTlsOptionsTests.cs (2 tests)	TLS/mTLS certificate operations may throw PlatformNotSupportedException or CryptographicException on some platforms. Currently uses SkipTestIfCryptoNotSupported() which catches exceptions and prints to console, but test shows as passed instead of skipped.
OtlpSecureHttpClientFactoryTests.cs (3 tests)	Same pattern for secure HTTP client tests.
StressTestNativeMethods.cs	Windows-only QueryProcessCycleTime P/Invoke. Returns 0 on non-Windows instead of skipping.
ExceptionProcessorTests.cs	Different assertions based on Environment.Is64BitProcess, but test doesn't skip on 32-bit.

Example of the problematic pattern:

private static void SkipTestIfCryptoNotSupported(Action testBody)
{
    try
    {
        testBody();
    }
    catch (PlatformNotSupportedException ex)
    {
        // Test appears as PASSED, not SKIPPED!
        Console.WriteLine($"[SKIPPED] Test skipped: {ex.Message}");
    }
}

Problems:

1. Test results show "passed" instead of "skipped" - misrepresents actual test coverage
2. CI dashboards cannot accurately report which platforms fully tested vs. partially tested
3. Developers may incorrectly assume TLS/crypto functionality is verified on all platforms

What is the expected behavior?

Tests that cannot run due to runtime-detected platform limitations should report as "skipped" in test results with a clear reason.

xUnit v3 introduces Assert.Skip("reason") which enables proper runtime skipping:

[Fact]
public void CreateHttpClient_ConfiguresServerCertificateValidation()
{
    if (!TryCreateTestCertificate(out var cert, out var skipReason))
    {
        Assert.Skip(skipReason); // Properly reported as skipped
    }
    
    // Actual test code using cert
}

Which alternative solutions or features have you considered?

Alternative	Pros	Cons
Upgrade to xUnit v3	Native Assert.Skip() support, proper test reporting	Breaking changes, migration effort required
Add Xunit.SkippableFact package	Works with xUnit 2.x, provides Skip.If()	External dependency, not as clean as native support
Environment variable detection	Works with existing SkipUnlessEnvVarFoundFact	Requires manual CI configuration per platform, doesn't auto-detect
Keep current workaround	No changes needed	Tests misleadingly show as "passed"
Split into separate test assemblies per platform	Clean separation	Significant restructuring, harder to maintain

Additional context

Affected tests by category:

1. TLS/mTLS Certificate Tests (5 tests)

   • OtlpTlsOptionsTests.OtlpSecureHttpClientFactory_CreatesClient_WithCaCertificateOnly
   • OtlpTlsOptionsTests.OtlpSecureHttpClientFactory_CreatesClient_WithMtlsClientCertificate
   • OtlpSecureHttpClientFactoryTests.CreateHttpClient_ConfiguresServerCertificateValidation_WhenCaCertificatesProvided
   • OtlpSecureHttpClientFactoryTests.CreateHttpClient_ConfiguresServerValidation_WithCaOnly
   • OtlpSecureHttpClientFactoryTests.CreateHttpClient_InvokesServerValidationCallbackAfterFactoryReturns

2. Windows-only Stress Test Metrics (1 test area)

   • CPU cycle measurement using QueryProcessCycleTime - silently returns 0 on non-Windows

3. Architecture-dependent Tests (conditional assertions)

   • ExceptionProcessorTests has different expectations for 32-bit vs 64-bit processes

Current xUnit version: 2.9.3
xUnit v3 status: Stable release available

References:

• xUnit v3 Skip Assertions
• xUnit v3 Migration Guide
• Related: OpenTelemetry Java uses JUnit 5's Assumptions.assumeTrue() for similar scenarios

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[martincostello]

Xunit.SkippableFact can be used to achieve similar results without upgrading to xunit v3.

We can't upgrade to xunit v3 until we drop support for .NET Framework 4.6.2 as xunit v3 requires a minimum target of 4.7.2.

[martincostello]

Example usage here.