Workers crash when model does not converge, IES

[acmoody]

I'm trying to run a prior Monte Carlo analysis with PEST++ IES. When a certain combination of parameters causes MODFLOW to not converge, that worker will crash instead of becoming a failed run. This occurs both in serial and parallel runs. Is this the expected behavior or should IES be able to continue in the case of non-convergence?

[jtwhite79]

All of the pest++ codes will tolerate forward run failures and deal accordingly. If both the forward run and worker are crashing, maybe that means the host machine is running out of resources? Can you post your model+pest files that are causing this?

[acmoody]

Hm, posting the files ended up being harder than anticipated. The files are too large to post here, and I'm unable to make my repo public because it's forked from a private repo. I placed the files at my agency's site , under Test Files. We have a water budget calculator that takes a few parameters that I commented out so you don't have to deal with installing that package. The posted files are a case where MODFLOW does not converge.

I've been testing it with just a few workers on a machine with 16 processors. I haven't had resource issues with this model before during GLM runs.

[jtwhite79]

Got the files - thanks for that. I tested this (using v5.2.24) by commenting more stuff out in that forward_run.py and changing the convergence conditional check to True on line 83 so that it will always think mf6 failed to converge and pestpp-ies had no issues dealing with the apparent failed run(s):

What version are you using? Can you post a screen shot of what happens when the worker (or serial run mgr) dies?

[acmoody]

I was running the 5.0.0 from the USGS data release and hadn't thought to make sure I was running the latest version. Using v5.2.24 seems to have resolved this issue. Thanks for your help!