How to use the cube

Useful Links

How to prepare a Job for the queue

The howto from IT gives pretty usable instructions on how to submit a job. Please also read the following information
  • The submit hosts for the minicube are lxsub01 - lxsub04. The software on these hosts is identical to the software on the compute nodes.
  • Unlike lxsubxx the compute nodes only see the /hera/ lustre filesystem. For this reason all software you want to run and all output data you want to write should be located on /hera/bhs/.
  • There is a maximum runtime of default 24 hours per job. If you need more you need to specify your requirements according to the howto.
  • If you want to have your own non-bash startup script run on the compute nodes to start your mpi enabled software, you have to submit it as a "binary program" meaning you chmod it to 755 and have a #!/usr/bin/perl or similar header so the compute node knows what interpreter to run it with. You also have to pass the -b y parameter to qsub. Reminder: if you change the script you submitted with -b y before the job is started in the queue, the modified script will be run.
  • * If you run mpirun inside your script* please do not pass the number of cpus or your own machine file. Specify your number of processes in the qsub commandline instead -- either via -pe openmpi __ or via -pe smp __. The enviroment will be set such that a naked call to mpirun will do the right thing
  • * If you run mpirun inside your script* make sure to compile your code with the mpi libraries on the submit nodes and use the mpirun command. Specifically for patric this means to remove the include of mpich libraries from the make file and change the mpi-compiler to mpicxx.

Software on the submit nodes

  • ITD offers different compilers on the submit nodes. ls /cvmfs/it.gsi.de/ shows you the list of compilers. This directory exists, even if ls /cvmfs/ shows nothing. It is due to the nature of cvmfs.
  • We have installed the EPD academic python distribution on /hera/bhs/software/epd , you cann add it to your path and scripts if you want to use it.

Tricks and Tips

  • * If you want to use the same Makefile on lxir and lxsub you can set location specific variables as for example
HOST=`hostname | head -c 4`

ifeq (HOST,'lxir')
   CXX = /u/pgoergen/local/bin/mpicxx
   INCLUDES = -I./include/ -I../obflib/include -I/u/pgoergen/local/include
   INCLUDES = -I./include/ -I../obflib/include

Known problems

  • Sometimes a job will not start because of a problem in the submit process. In these cases resubmitting the job can fix the problem.

-- PaulG - 23 May 2012
Topic revision: r2 - 2012-05-30, PaulG
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding GSI Wiki? Send feedback
Imprint (in German)
Privacy Policy (in German)