Memory Leak / Bleeding

[Jacob J. Banks]

We have recently gained access to our School's Linux based cluster, and we have gotten OpenSim (4.3) to work with Matlab (by 'we', I mean an outstanding IT person). While things 'work', we have noticed that there is a pretty significant memory leak when we use the Analysis Tool (and probably other tools, but we typically can/will run those on our own PCs in advance). Specifically, we run SOpt/JRA on a frame-by-frame basis so we can control box mass and some known assist actuator forces. We have tried to diagnosis/isolate this issue and/or variable, but it appears that it is within OpenSim (i.e., Matlab doesn't seem to see/have access to a variable that is noticeably leaking memory).

Aside from just being annoying, this is costing us money. We are charged by the amount of memory we require to run our jobs.

So, we were wondering if there is any known solution to this? Guessing if there is, that it is environment dependent. Otherwise, is there a way to close and reopen OpenSim within Matlab? We have tried to close and delete all model and tool variables, but to no avail. Our last potential solution is/will be to write a script outside of Matlab that will open/close matlab after each SOpt/JRA run, and seemingly clearing memory.

Note: It appears that on our PCs this issue may be 'hidden' via Matlab's ability to SWAP memory.

I've seen similar posts (one very recently about the IK tool), to what I am asking, but have haven't seen a resolution. So I apologize if I should have just followed up on one of those.

Any insights would be appreciated. Thank you,
Jake Banks

[Pavlos Silvestros]

Hi Jake,

I had a similar issue a few years ago. Some differences there as I was running OpenSim 3.3 on Matlab R2019(?) on a windows PC. In any case it might be of use. I tried a lot of tweaking within Matlab to try and control memory but what worked for me was to bundle the majority of my code that looped through different conditions in a separate function that was called by Matlab after each iteration. I know it is not the most elegant solution but it got me out of trouble. From what I remember Matlab handles some features of memory internally when a function is called again - but I would have to read up on that again to be sure.

See this post from back then:

viewtopicPhpbb.php?f=91&t=12232&p=34741&start=0&view=

Hopefully it can give you some ideas. Let me know and I can find some more information on it.

Cheers,
Pavlos

[Ayman Habib]

Hi Jacob,

Thanks for reporting.

When using Matlab (or Python) one factor that comes into play is garbage collection. Basically when the Java JVM used by Matlab decides that some variable/memory is not needed any more then it gets released (garbage collected). Typically you can "suggest" to Matlab to clear memory but there's more to it that is internal to Matlab (e.g. https://www.mathworks.com/matlabcentral ... re-cleared)

One possible way to help Matlab release memory is to set to null the variables you're not using anymore (e.g. tables etc.) to see if that helps.

It would be great if you can provide a short program to reproduce the leak (ideally in C++ where other players/memory-management and communication between Java and C++ are not in play) and we'll try to help you out and save you money

All the best,
-Ayman

[John Davis]

Just wanted to chime in to say that I've noticed memory leaks as well in the Analysis Tool when running OpenSim with the MATLAB API--both when doing static optimization and when using Luca Modenese's muscle line of action plugin, which makes me think it's something in the Analysis Tool itself vs. the specific analysis being run.

Java eventually crashes (often along with MATLAB) if the memory fills up completely. The problem is more acute when using parfor, I think because each instance of MATLAB has less memory allotted to it. I'll see if I can code up an example later this week with demo data that reproduces the leak.

[Jacob J. Banks]

First off, thank you Pavlos, Ayman, and John for your replies. They were/are greatly appreciated. Second, sorry for my delayed response.

Thankfully, since last week, we have sort of have 'resolved'/found work arounds for memory bleeding.
Note: Pavlos, we already (previously) had several subfunctions (as you suggested... and that should seemingly work!), and that did not help.

What has 'helped' is this (and it is, admittedly, a bit 'crude'):
- we do not *.run the Static Optimization tool via our MATLAB API. Rather we now use the 'system' command within MATLAB to run opensim directly via the opensim operating system. This works well, as we have a correctly populated SOpt.xml setup file. This line of code looks like this (in Matlab):
system('opensim-cmd run-tool YOURSETUPFILE.xml')
We run this in a subfunction. We also do this for the JRA tool (exact same code, just a different setup file). We don't do this for IK, but suspect it would work similarly for those who have IK bleeding issues.

- After the above, we still had some memory bleeding due to making setup files though (for 200 frames, it would take ~20G total). To 'fix' this, we make 1 original setupfile via our API commands, and then adjust it via matlab text editing functions rather than our API code. Crude and perhaps prone to errors, but it works!

- Note: Tests revealed that creating/editing our model does not seem to bleed memory (thankfully, as that would be difficult to do via either of the above).

In all, the above allowed us to go from ~120 Gs of needed memory to ~3Gs (for 1 file with ~200 frames, that we run separately to control box mass and actuator forces). This amounts to a pretty substantial savings on our supercomputer costs (when running >500 files).

We have not yet tested whether any of the above helps our PCs run more efficiently (just the Linux based supercomputer), but we have a feeling it will help quite a bit, as this and other posts have highlighted that there are definitely some OpenSim to MATLAB API memory issues (but on some PCs matlab seems to be able to SWAP memory successfully to hide the worst of it).

Well, that is all I have for now. Would be happy answer any questions (if I can!). Thanks again for the replies etc.
Jake Banks

[Ayman Habib]

Thanks John & Jake for the update.

It's great that you were able to workaround these issues, Jake, however it would be great for the community if we can fix these issues on our side so that users don't need to spend their time on workarounds.

Truly would appreciate a simple example to reproduce the leak and we'll do our best to fix it at the earliest possible chance.

All the best,
-Ayman