CernVM-FS
The distribution of custom software for user groups on the HPC infrastructure is provided using the
Cern Virtual Machine File-System (CernVM-FS).
CernVM-FS is a read-only network file system based on HTTP and optimized to deliver software in a fast, scalable, and reliable way. Files and file meta-data are cached and downloaded on demand.
Cluster nodes, desktops and interactive machines mount software repositories in the directory
/cvmfs/
. Every sub-directory is called like the maintainers department or experiment name followed by the GSI domain name. The IT Department for example provides
/cvmfs/it.gsi.de
with common software available to all users. Experiment groups have repositories like
/cvmfs/hades.gsi.de/
for the HADES experiment or
/cvmfs/csee.gsi.de/
for the Common Systems and Experiment Electronics department.
For a list of software at GSI made available via CVMFS see SoftwareInCvmfs.
Usage
Sub-directories in /cvmfs/
will be automatically mounted on demand as soon as they are accessed. This means directories are visible when listing with the
ls command only if they are mounted already.
It is up to the maintainers how to structure the software inside a repository, and how to enable users to load required environment variables to use it. The HPC department recommends to use the
environment-modules package providing a convenient command called
module (→
EnvironmentModules).
Repositories
The following table contains a list of all software repositories available:
Note that not all repositories are supported on all Debian platforms.
Instructions for maintainers
Please contact hpc@gsi.de to commission a new software repository.
Each software repository is hosted on a dedicated server called
lxcvmfs. Write access to these nodes is restricted to the repository maintainer(s). In order to install or updated software on a repository server, maintainers have to deploy the software package on a build-server
lxbuild. From there a publishing script synchronizes the software repository with the CernVM-FS server.
Repository Access
We need a public SSH key from the responsible software maintainer. Create one with:
» mkdir key; ssh-keygen -q -t rsa -b 4096 -f key/id_rsa
Please send us only the id_rsa.pub
file containing the public part of your key.
Don't forget to specify this private key when logging in to our build hosts, e.g.:
» ssh -i key/id_rsa lxbuild02.gsi.de
Along with your public key send us the
name of the repository you want to become software maintainer for, and your
user account name.
Build Servers
The HPC Department operates completely independent CernVM-FS infrastructures for each Debian Version. (This avoids potential incompatibilities between binaries compiled for different Debian releases). Maintainers access the infrastructure for a particular Debian version using a dedicated build server.
Build Server |
Platform |
CernVM-FS server |
Repository |
lxbuild04.gsi.de |
Debian 7 (Wheezy) |
lxcvmfs11.gsi.de |
it.gsi.de |
lxcvmfs12.gsi.de |
fairroot.gsi.de |
lxcvmfs13.gsi.de |
alice.gsi.de |
lxcvmfs14.gsi.de |
hades.gsi.de |
lxcvmfs15.gsi.de |
csee.gsi.de |
lxcvmfs16.gsi.de |
theory.gsi.de |
lxcvmfs17.gsi.de |
epics.gsi.de |
lxcvmfs18.gsi.de |
labview.gsi.de |
lxcvmfs19.gsi.de |
gamma.gsi.de |
lxcvmfs20.gsi.de |
r3b.gsi.de |
lxbuild05.gsi.de |
Debian 8 (Jessie) |
lxcvmfs21.gsi.de |
it.gsi.de |
lxcvmfs22.gsi.de |
alice.gsi.de |
lxcvmfs23.gsi.de |
fairroot.gsi.de |
lxcvmfs24.gsi.de |
csee.gsi.de |
lxcvmfs25.gsi.de |
theory.gsi.de |
lxcvmfs27.gsi.de |
panda.gsi.de |
lxcvmfs28.gsi.de |
hades.gsi.de |
lxcvmfs29.gsi.de |
gamma.gsi.de |
lxcvmfs30.gsi.de |
r3b.gsi.de |
lxcvmfs31.gsi.de |
epics.gsi.de |
lxcvmfs32.gsi.de |
labview.gsi.de |
lxcvmfs33.gsi.de |
htit.gsi.de |
Note that the name of the software repositories are similar for all Debian versions, while the repository servers are tied to a particular Debian release.
Software Installation
There is no backup of software on the CernVM-FS facility, neither on the build servers nor the repository servers!
It is assumed that you maintain your software with a version control system and that you can reproduce your software stack anytime. Login to the build server for your target platform and follow these three basic steps:
- Install your software in the corresponding repository directory. (Don't forget to check permissions and executable bits.)
- Adjust the list of files which will NOT be synced to the repository server (if needed).
- Call the script to publish your software.
It is up to the software maintainers to structure the directories inside their software repository. We strongly encourage you to attempt the
smallest possible installation in terms of storage size. This means you should use out-of-source compilation or the install functionality of your build tool. Furthermore we recommend to
never change software already in use! If you
really need to overwrite already installed software please contact us beforehand.
Maintainers have the option to
exclude files and directories from the synchronization process by enlisting them in a file named like the repository but suffixed
*.gsi.de.exclude
. (Take a look to the Rsync manuals 'exclude pattern rules' section for more details.)
Common obstacles and their solutions:
- If you encounter missing library dependencies of standard software, please let us know. We will install them in the cluster for you.
- Make sure that the file and directory permissions are correct, for example with commands like:
find . -type f -exec chmod o+r {} \;
and/or find . -type d -exec chmod o+rx {} \;
- If you decide to build your software on another host, make sure the platform is compatible. (Be careful with static linking!)
- Test your software by executing it first on the build host, and after publishing on the cluster.
Publishing
Publish software with the cvmfs
command, respectively cvmfs-remote
on Debian Wheezy and Jessie.
» sudo cvmfs-remote publish REPO
The option
REPO needs to be replaced by the name of your repository, e.g.
alice.gsi.de. Depending on the amount of files newly added to the software repository synchronization and publishing can take several minutes.
Resigning
Repositories need to be resigned at least every 30 days!
» sudo cvmfs-remote resign REPO
Otherwise the software will expire and the repository will become unavailable. This mechanism is in place to prevent clients from using outdated software.
You may want to deploy a Cronjob to automatically resign your repository.
MAILTO=j.doe@gsi.de
0 0 1,20 * * sudo cvmfs-remote resign REPO
Add a Cronjob to your user account with the command
crontab -e
. Adjust the mail address and the repository name like it is illustrated in the example above.
Best Practice
You should provide a README file in your repository pointing users to information resources, documentation and support.
This file should contain:
- Name and purpose of the software.
- Small how-to for beginners.
- A description of the logical structure of the repository.
- Links to documentation and support pages.
- Your contact information.
You should provide a script to enable users to add your software into their shell environment.
The HPC department recommends the use of Environment Modules like described above in the section CvmFs.