Not all results converted

[Shannen Kizilski]

I'm using the Python API to run a series of simulations with svSolver. As part of this routine, I use the svpost command within python:

Code: Select all

sub.run(["/usr/local/sv/svsolver/2019-02-07/svpost", "-all", "-start", str(s1), "-stop", str(s2),
                     "-incr", str(sinc), "-indir", "./"+nprocDir+"/", "-outdir", "./", "-vtkcombo",
                     "-vtu", "all_results.vtu", "-vtp", "all_results.vtp"])

I was trying to compare the results of two simulations in ParaView, when I noticed that one simulation had all results for all converted timesteps (200-300), whereas the other simulation had some results for all timesteps, but other results (like rWSS_XXXX) shown below had a seemingly random assortment of timesteps converted. The only difference between the two simulations is a very slight change in geometry.

paraview_incomplete.jpg (107.03 KiB) Viewed 456 times

Does this mean that the simulation did not converge correctly for those times? Is there anything else this could mean? Is this a common event? I'm running a larger study where I don't directly look at the vtu/vtp files, but instead I extract summary variables from them. I'm worried now that some of those variables may be from incomplete data sets like this.

Thanks!

[David Parker]

Hi Shannen,

I am wondering if some of your simulations have incomplete results. This can happen if the simulation terminated because it did not converge or produced NaNs, or if it was killed by the Slurm Workload Manager.

You need to check the output of the svsolver command to see if something like this happened, might see something like

Code: Select all

--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------

You should also examine svpost output to see if there are any error messages. If a node goes down during a simulation then there will be missing results for sections of the mesh.

Cheers,
Dave

[Shannen Kizilski]

Hi Dave,

These simulations are currently running on my Ubuntu 18.04 laptop using 6 cores. The simulation runs to completion, and most variables (pressure, velocity, timeDeriv, etc.) convert successfully for all time steps.

It seems that maybe the results that take some post-processing computation (rWSS, average_speed, average_pressure) run into a problem and don't convert. This only happened for one simulation out of 5 that I ran this week, so it's possible it was just a one-time event. For the time steps I converted, the residuals look fine, and nothing in the histor.dat or echo.dat files mention any errors. If I re-run svpost, the same variables are still missing.

I looked at the svpost output for one of the steps (#203) where rWSS didn't convert, and I see that it started converting for that step but then didn't find that variable for one of the processors:

Code: Select all

Reducing (rwall shear stresses) results : ./fullResults/design-490/unconverted-results//restart.203.1
Done reading (rwall shear stresses) results : ./fullResults/design-490/unconverted-results//restart.203.1

Reducing (rwall shear stresses) results : ./fullResults/design-490/unconverted-results//restart.203.2
Done reading (rwall shear stresses) results : ./fullResults/design-490/unconverted-results//restart.203.2

Reducing (rwall shear stresses) results : ./fullResults/design-490/unconverted-results//restart.203.3
NOTE: No (rwall shear stresses) in ./fullResults/design-490/unconverted-results//restart.203.3

Reducing (ybar) results : ./fullResults/design-490/unconverted-results//restart.203.1

How is rTAWSS calculated? If there are rWSS values missing, will it still give me a result that ignores these missing values, or would I get an error?

Thanks!

[David Parker]

Hi Shannen,

The NOTE: No (rwall shear stresses) in ./fullResults/design-490/unconverted-results//restart.203.3 message means that the header for restart.203.3 did not contain rwall information. This typically means that the restart file was interrupted while writing simulation results. Secondary variables that depend on rwall will not be created, you should get a warning message stating this.

rTAWSS is calculated from vinplane_traction.

It is not clear why restart.203.3 was corrupted. Make sure that you are not writing to the same directory for different simulations. If you are rerunning a simulation make sure to remove any files created by the previous simulation (e.g. 6-procs_case0.

Cheers,
Dave