max_job_runtime processing the item twice #11
Description
Looks like this happens to all enumerators but reproducing with array_enumerator is fairly easy:
class DummyWorker
include Sidekiq::Worker
include SidekiqIteration::Iteration
self.max_job_runtime = 5.seconds
def build_enumerator(cursor:)
array_enumerator(%w[a b c d], cursor: cursor)
end
def each_iteration(item)
Sidekiq.logger.info "Processing: #{item}"
sleep 3.seconds
end
endThe log happens like this:
INFO: start
INFO: Processing: a
INFO: Processing: b
INFO: [SidekiqIteration::Iteration] Interrupting and re-enqueueing the job cursor_position=1
INFO: done
INFO: start
INFO: processing b
INFO: processing c
INFO: [SidekiqIteration::Iteration] Interrupting and re-enqueueing the job cursor_position=2
INFO: done
INFO: start
INFO: processing c
INFO: processing d
INFO: [SidekiqIteration::Iteration] Interrupting and re-enqueueing the job cursor_position=3
INFO: done
INFO: start
INFO: processing d
INFO: [SidekiqIteration::Iteration] Completed iterating. times_interrupted=3
INFO: done
a: 1 time
b: 2 times
c: 2 times
d: 2 times
Looks like every time the iteration is interrupted, the last item is processed again even though there are no errors.