• Infrastructure
• HighPerformanceComputing

Cleveland Clinic

The Open Knee(s) team have access to a high performance computing cluster located at the Cleveland Clinic. This infrastructure is available Cleveland Clinic employees and within the Cleveland Clinic intranet.

Hardware Specifications

Usage for Open Knee(s)

Before using the cluster, one needs

• an cccount on Cleveland Clinic computing cluster
• an SSH client to connect to access command prompt of computing cluster
• an SFTP client to upload/download files from the computing cluster

Once an account is available, Open Knee(s) simulations can be run on this infrastructure using the following steps. erdemira is used as the user name for the examples below.

• Connect the computing cluster using SSH client:
  • ssh erdemira@hpc-login
• Change directory to 'lustre' folder:
  • cd lustre
• Create a new directory to keep simulation files (create folder for each simulation):

  • mkdir SampleSimulation

• Change directory to simulation folder
  • cd SampleSimulation/
• Connect the computing cluster using SFTP client to upload model:
  • Use gftp for a GUI based client, host: hpc-login username: erdemira

  • Change remote directory to simulation folder, e.g. SampleSimulation

  • Change local directory to where your model resides, e.g. /home/erdemira/OpenKnee-G1
  • Upload model to computing cluster, e.g. tf_joint_FEBio_v2.feb
  • attachment:uploading_files.png
• In the SSH client, list directory to see the model file:

  • [erdemira@cc-dclrilog52 SampleSimulation]$ ls

  • tf_joint_FEBio_v2.feb
• Create a file with a text editor to use as a batch script:

  • e.g., nano RunModel

• Insert variables of job submission to the file in following:

#SBATCH --mail-type=ALL 
#SBATCH --mail-user=erdemira@ccf.org 
#SBATCH --job-name=OKS-G1     
#SBATCH --nodes=1 
#SBATCH --ntasks=1 
#SBATCH --cpus-per-task=16 
#SBATCH --time=24:00:00 
#SBATCH --mem=56000 
#SBATCH --error=job.%J.err 
#SBATCH --output=job.%J.out 

export OMP_NUM_THREADS=16 

module load FEBio/2.1.2  
srun febio2.lnx64 -i tf_joint_FEBio_v2.feb

• Submit the simulation to the scheduler in the SSH client:

  • [erdemira@cc-dclrilog52 SampleSimulation]$ sbatch RunModel

  • Submitted batch job 10460
• You should have received an e-mail stating that A SLURM job has started. You will also get an e-mail when the job ends. You can exit the SSH client if you want.
• In SSH client, squeue provides a list of jobs running, e.g.

  • [erdemira@cc-dclrilog52 SampleSimulation]$ squeue

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON) 
             10460      defq   OKS-G1 erdemira  R       1:38      1 lri004 

• Once the simulation finishes you can download the results files (e.g. xplt, log) by SFTP client.