PestPP IES executable (may 2025 release) hogs memory resources #356
Replies: 7 comments 34 replies
-
|
How are you remoting into the master? ssh? Teamviewer? Is this windows? is the master by any chance AMD Ryzen or similar? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @p-ortega, I’m using windows desktop setup below: I am remoting into the computer using Windows remote desktop |
Beta Was this translation helpful? Give feedback.
-
|
Just to confirm this is 5.2.19? |
Beta Was this translation helpful? Give feedback.
-
I can't 100% confirm this, but having done 5 or 6 tests with noptmax -1, I think your recc's re: "panther_master_echo_interval_milliseconds and panther_master_timeout_milliseconds to 500" solve the issue, for me at least. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks C. I'll make some mods to the default behavior asap and get a release going asap. |
Beta Was this translation helpful? Give feedback.
-
@CooperGarth did setting those args help with your issue? |
Beta Was this translation helpful? Give feedback.
-
|
@jtwhite79 Next week I'll give the settings a try - just been working remotely for the last week so not too keen to crash my master without being able to physically restart my computer! |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jtwhite79 i've downloaded the latest release - using 5.2.20. |
Beta Was this translation helpful? Give feedback.
-
|
@jtwhite79 I changed the settings to 1000 ms - model still crashed. I've run a perfomance log during the run It appears the ram usage is linearly over 30 minutes increasing to 100% which is stopping me from remoting in. This only happens with the new version of Pest-IES. Is there a memory leak in the new Pest++ IES? Thanks, |
Beta Was this translation helpful? Give feedback.
-
|
And the .rec file too if you have it |
Beta Was this translation helpful? Give feedback.
-
|
Anf the .rmr file... |
Beta Was this translation helpful? Give feedback.
-
I will need to run the master again, I'll do that later tonight. It just takes a while for the error to occur (about 30 to 45 minutes). I'll attach the files once I have it. I'll also do a comparison with an older version - I'll see what the RAM usage is like. |
Beta Was this translation helpful? Give feedback.
-
|
@CooperGarth any news on this front? |
Beta Was this translation helpful? Give feedback.
-
sorry I bit slow on my end - next week I'll get some time to test it agian. |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
Bit of a loose update at my end on this memory usage issue: my system was choking on pestpp-mou 5.2.19 within about 5 mins of kicking off a run with just 60 workers. I reverted to 5.2.17 and I do not experience the issue. I can see about a 10GB difference in memory usage within that first 5 minutes....I reckon something is definitely up in 5.2.19. |
Beta Was this translation helpful? Give feedback.
-
|
I'll get back to you on hat later today (sorry didn't think to check that, just killed it and reverted to 5.2.17). You know, one thing that comes to mind here is that stuff you did for me on IES covariance reinflation restarts having the full ies_parameter_ensemble available for mean shifting (to 5.2.17, which ended up in 5.2.18). I.e. rather than a trimmed down version (dropping failed/bad phi realz). From memory you mentioned the reason for the trimming/reordering of the original par en was memory saving. Wonder if there's something going on there like a memory leak given that persistent linear rise garth pointed out. Just a thought,. not based on any real logic. I'll test 5.2.18 / one of your test exes on that issue and report back. |
Beta Was this translation helpful? Give feedback.
-
|
thats a good point Chris. Ive been running a few leak detectors today and nothing serious is showing up but I'll try a run with the reinf active now |
Beta Was this translation helpful? Give feedback.
-
fyi - I re-ran the same mou case with 5.2.19 (rather than 5.2.17) with these settings as you previously suggested: ...and the system bog-down issue goes away. This tallies with my previous tests with a bunch of noptmax -1 runs. Memory usage is back to the level of 5.2.17 too (10gb lower)... From memory, you implemented something that internally adjusts these echo / timeout variables based on live run times, but I still saw a problem (in the first 5 mins at least) - maybe it started off at too-short an interval in my case, and hadn't yet had time to adjust as no runs had completed by the time I killed it (because the system was almost unusable). Maybe this just comes down to the user making sure they specify these options with decent params for their use case. I don't see how this could cause the OP's issue re: linearly increasing memory usage though; could be something else (either in pest, or outside of it I guess). |
Beta Was this translation helpful? Give feedback.
-
|
Ok good to know. I've chased down a few small mem issues that I found with massif and memcheck, but nothing major...I'll probably increase those default timeout values further to avoid this thrashing... I've been running some large ish problems for multiple days and haven't seen that monotonic memory issue either but I'm on Linux or Mac...maybe I'll fire up a windows machine and relive the glory days haha. @CooperGarth any more info on this front? |
Beta Was this translation helpful? Give feedback.
-
Apologies for the delay - still using the old version to run IES; I'll try to get onto it today or tomorrow with the rec and rmr files. |
Beta Was this translation helpful? Give feedback.
-
|
TS_20_sal_IES.zip |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for those files @CooperGarth . I'll have a test run on a windows machine first thing in the AM. Can you do one more really quick test - same setup as the files you just posted, but with |
Beta Was this translation helpful? Give feedback.
-
|
I'll try that now. I'll ask see if I can get a snapshot of the processes and network usage it's currently using and see if there are lots of background processes that are being left open. |
Beta Was this translation helpful? Give feedback.
-
|
@jtwhite79 is this the command you are refeering to? |
Beta Was this translation helpful? Give feedback.
-
|
That's the one |
Beta Was this translation helpful? Give feedback.
-
|
Hey @jtwhite79 I've run it with the command - still crashes. I tried to plot the processes using ram at the time using manual dumps of current process- Memory compression seesm to be suffering. [TS_20_sal_IES_1000.zip](https://github.com/user-attachments/files/21028608/TS_20_sal_IES_1000.zip
|
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi,
I'm using the latest release of PestPP-IES and the computer I run the Manager on becomes impossible to remote into.
On the 2023 offical release (not the pre-releases) I don't seem to have this problem.
My only work around is to have a computer dedicated to running the Manager which reduces the number agents you can have.
Has anyone has this issue before or is there any diagnosis I can do to figure out what's going on?
Thanks!
Beta Was this translation helpful? Give feedback.
All reactions