Intermittent NetConnectNotAllowedError because stub_requests unexpectedly empty

[dan-jensen]

Problem

My Rspec test suite occasionally encounters WebMock::NetConnectNotAllowedError because StubRegistry cannot find any stubs (when multiple were declared before(:each).

Details

We're stubbing external services as described by Thoughtbot. In other words, we're doing this:

RSpec.configure do |config|
  config.before(:each) do
    stub_request(:any, /api.github.com/).to_rack(FakeGitHub)
  end
end

However, we see intermittent test failures randomly in our test suite because of WebMock::NetConnectNotAllowedError. After digging, we've found the problem is because in StubRegistry#request_stub_for, there are no stubs (both global_stubs and request_stubs are empty).

Some things we've learned that support the theory of a race condition:

• We monkey-patched StubRegistry#response_for_request to re-try finding a stub (by calling request_stub_for a second time if there's no stub the first time). Sometimes it succeeds in finding a stub on the second attempt.
• Using config.prepend_before(:each) seems to mitigate the problem.

Question

Given that we're calling stub_request in a before(:each) block, how is it possible for request_stubs to ever be empty? Or from another perspective, what could possibly be causing a race condition?

[bblimke]

@dan-jensen have you investigated that further? Do you have any more details?

[dan-jensen]

@bblimke we're overriding StubRegistry#response_for_request to automatically re-try the search for a stub. This has proven durable, with no more failures across thousands of invocations. It's ugly though:

module WebMock
  class StubRegistry

    def response_for_request(request_signature)
      stub = request_stub_for(request_signature)
      if !stub && request_signature.uri.host != '127.0.0.1' # override
        stub = request_stub_for(request_signature)
        puts "FOUND stub on second attempt" if stub
      end
      stub ? evaluate_response_for_request(stub.response, request_signature) : nil
    end

    private

    def request_stub_for(request_signature)
      stub = (global_stubs[:before_local_stubs] + request_stubs + global_stubs[:after_local_stubs])
        .detect { |registered_request_stub|
          registered_request_stub.request_pattern.matches?(request_signature)
        }
      if !stub && request_stubs.empty? && request_signature.uri.host != '127.0.0.1' # override
        puts "WARN: request_stubs unexpectedly empty due to apparent race condition with test configuration (#{request_signature.uri.to_s})"
      end
      return stub
    end

  end
end

[acant]

Hi @dan-jensen and @bblimke,

I ran into a similar looking problem in a Cucumber/Capybara/Selenium/Puma based test suite. I was working around it by having Cucumber re-try failures, which mostly made the test suite pass.

After some playing around, I discovered that the problem was that WebMock.reset! was getting called too early, and clearing out the registry before all the steps had finished running.

I inlined require **'webmock/cucumber'(webmock/cucumber.rb) into features/support/env.rb, and changed the After handler:

After { WebMock.reset! }
to
Before { WebMock.reset! }

I have run the test suite multiple times since, and this has entirely eliminated the failures. 🎉

This seems like it was a different version of the problem then what @dan-jensen encountered. But hopefully this is useful to someone.