IT-Service: Lustre file system

Overview

HPC file system for mass data
  • output of simulations
  • experiment data, output of analyses

  • Used in connection with GSI batch farm
  • Accessible for GSI Linux users
  • User directories upon request

Lustre file system

  • Lustre is the general large scale storage for experiment and simulation data
    • Data on Lustre is mainly used = analyzed or produced by batch farm calculations
    • Lustre is not an archive storage for experiment data
    • Lustre does not have any backup
  • To use Lustre, a Linux account is necessary
  • The computing-coordinator of the users group or an administrator has to create a suitable directory on Lustre
  • Access to Lustre is via the interactive lustre.hpc.gsi.de nodes
  • Outside the cluster, Lustre is only accessible through sshfs.
  • Limits:
    • Size-Quotas for groups, managed by computing coordinators
    • Inode-Quotas for users, managed by administrators
  • Best practices:
    • avoid many small files ( O(10^6) )
    • avoid compilation/installations on Lustre -> see also CvmFs, VirgoContainers
    • keep number of parallel file-open and file-lock operations small
    • avoid massively parallel metadata operations
    • do not store too many files in a directory ( O(10^4) )
    • do not write into one file from many processes
    • keep files open and buffer data during job run time
  • Batch Farm
  • Lustre details

Availability, Support

-- ThomasRoth - 2020-07-16
Topic revision: r4 - 2021-01-07, ChristopherHuhn - This page was cached on 2021-03-04 - 20:03.

This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding GSI Wiki? Send feedback | Imprint (german) | Privacy Policy (german)