ALICE Data Transfer

Howto:

according to the requests in the Transfer Request Wiki (https://alice-wiki.gsi.de/Data/TransferRequests)

config files requesting data transfers should be placed in the transferinput Directory (see below).

See example config files in the transferprocessed Directory (see below)

A config file should have the following naming convention:
  • config_<Name of Person writing the config>_<data set>_<Name of Person requesting the Transfer>_<sim/data>_<esd/aod>_<ev. reconstruction pass>_<collision particles>_<energy>
    • e.g.: config_ks_LHC10e_sweber_data_esd_pass4_pp_7tev
A config file should have the following structure:

note: one configuration parameter per line, it needs to be 13 configuration lines in total before the run numbers start
  • top level directory in AliEn <e.g. /alice/data/2010/LHC10e>
  • name of text file which contains the list of files which could not be copied
    • naming convention: should be the same name as the config file but starting with "errorfiles"
      • e.g. errorfiles_ks_LHC10e_sweber_data_esd_pass4_pp_7tev
  • number of parallel transfers <e.g. 10>
  • top level target directory on Lustre <e.g. /lustre/nyx/alice/alien>
  • string for AliEn find command <e.g. /pass4/*.*/root_archive.zip>
  • TTL of a copy process in seconds <e.g. 4000>
  • list of black listed SEs (should be 4 entries in the config files)
    • e.g.:
      • ALICE::NECTEC::SE
      • ALICE::LBL::SE
      • ALICE::KISTI_GSDC::SE
      • ALICE::Hiroshima::SE
  • copy mode (e.g. standard)
    • alternative copy modes:
      • listmode (copies from a list of files instead of creating ist own list via alien_find)
        • in listmode the errorfile (position two of config file) takes the role of specifying the list of files to be copied
          • in this mode obsolete Parameters are:
            • top Level Directory in AliEn
            • string for AliEn find command
            • list of runs to be copied
      • random (copies randomly a specified percentage of a run)
        • Parameter "additional Parameter" (see below) Needs to contain the percentage in number (e.g. 50)
  • additional parameters for copy mode
    • e.g. (NA for Standard mode)
  • check size of data set (enabled by a value of 1, since this is very time consuming a value of 0 should be the default)
  • list of successfully copied files (file name to be given)
  • collision system (pp, pPb, PbPb)
  • collision energy (2.76TeV, 7TeV, 8TeV, 2.76ATeV, 5.023ATeV, ...)
  • base dir for run lists for train Analysis
    • /hera/alice/train/lists/transfer
    • /lustre/nyx/alice/train/lists/transfer
  • external script for creating train lists
  • list of runs to be copied
    • e.g.
      • 000130850
      • 000130848
      • 000130847
Current Data Transfer Boxes:
  • lxalitransfer5: Transfer to /hera/alice/alien
  • lxalitransfer6: Transfer to /lustre/nyx/alice/alien
Access via ssh key.

Precondition: member in the alidata group

Directory Structure:

  • /home/ali-data
    • alien (link to AliEn Installation)
    • bin (contains executables)
    • scripts (contains setup script)
    • Transfers
      • tmp (contains list of transfers to be processed)
      • transferinput (here the config files for data transfers should be placed)
      • transferlogs (contains log files of data transfers)
      • transferprocessed (contains processed data transfer config files and list of files which could not be copied)
      • transferworkdir (workdir of the transfer script: contains temporary files)

Infrastructure:

  • 2 cron jobs (crontab -l)
    • proxy-renewal (renews Grid Proxy, runs once per day at 5:35 am)
    • transfermaster (checks content of transferinput and starts transfers if config files are present, runs every 2 hours)

-- KilianSchwarz - 13 Nov 2015
Topic revision: r5 - 2015-11-27, KilianSchwarz - This page was cached on 2021-07-30 - 00:31.

This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding GSI Wiki? Send feedback | Imprint (german) | Privacy Policy (german)