ALICE Data Transfer

Howto:

according to the requests in the Transfer Request Wiki (https://alice-wiki.gsi.de/Data/TransferRequests)

config files requesting data transfers should be placed in the transferinput Directory (see below).

See example config files in the transferprocessed Directory (see below)

A config file should have the following naming convention:
  • config_<Name of Person writing the config>_<data set>_<Name of Person requesting the Transfer>_<sim/data>_<esd/aod>_<ev. reconstruction pass>_<collision particles>_<energy>
    • e.g.: config_ks_LHC10e_sweber_data_esd_pass4_pp_7tev
A config file should have the following structure:

note: one configuration parameter per line, it needs to be 13 configuration lines in total before the run numbers start
  • top level directory in AliEn <e.g. /alice/data/2010/LHC10e>
  • name of text file which contains the list of files which could not be copied
    • naming convention: should be the same name as the config file but starting with "errorfiles"
      • e.g. errorfiles_ks_LHC10e_sweber_data_esd_pass4_pp_7tev
  • number of parallel transfers <e.g. 10>
  • top level target directory on Lustre <e.g. /lustre/nyx/alice/alien>
  • string for AliEn find command <e.g. /pass4/*.*/root_archive.zip>
  • TTL of a copy process in seconds <e.g. 4000>
  • list of black listed SEs (should be 4 entries in the config files)
    • e.g.:
      • ALICE::NECTEC::SE
      • ALICE::LBL::SE
      • ALICE::KISTI_GSDC::SE
      • ALICE::Hiroshima::SE
  • copy mode (e.g. standard)
    • alternative copy modes:
      • listmode (copies from a list of files instead of creating ist own list via alien_find)
        • in listmode the errorfile (position two of config file) takes the role of specifying the list of files to be copied
          • in this mode obsolete Parameters are:
            • top Level Directory in AliEn
            • string for AliEn find command
            • list of runs to be copied
      • random (copies randomly a specified percentage of a run)
        • Parameter "additional Parameter" (see below) Needs to contain the percentage in number (e.g. 50)
  • additional parameters for copy mode
    • e.g. (NA for Standard mode)
  • check size of data set (enabled by a value of 1, since this is very time consuming a value of 0 should be the default)
  • list of successfully copied files (file name to be given)
  • collision system (pp, pPb, PbPb)
  • collision energy (2.76TeV, 7TeV, 8TeV, 2.76ATeV, 5.023ATeV, ...)
  • base dir for run lists for train Analysis
    • /hera/alice/train/lists/transfer
    • /lustre/nyx/alice/train/lists/transfer
  • external script for creating train lists
  • list of runs to be copied
    • e.g.
      • 000130850
      • 000130848
      • 000130847
Current Data Transfer Boxes:
  • lxalitransfer5: Transfer to /hera/alice/alien
  • lxalitransfer6: Transfer to /lustre/nyx/alice/alien
Access via ssh key.

Precondition: member in the alidata group

Directory Structure:

  • /home/ali-data
    • alien (link to AliEn Installation)
    • bin (contains executables)
    • scripts (contains setup script)
    • Transfers
      • tmp (contains list of transfers to be processed)
      • transferinput (here the config files for data transfers should be placed)
      • transferlogs (contains log files of data transfers)
      • transferprocessed (contains processed data transfer config files and list of files which could not be copied)
      • transferworkdir (workdir of the transfer script: contains temporary files)

Infrastructure:

  • 2 cron jobs (crontab -l)
    • proxy-renewal (renews Grid Proxy, runs once per day at 5:35 am)
    • transfermaster (checks content of transferinput and starts transfers if config files are present, runs every 2 hours)

-- KilianSchwarz - 13 Nov 2015
Topic revision: r5 - 2015-11-27, KilianSchwarz
 
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding GSI Wiki? Send feedback
Imprint (in German)
Privacy Policy (in German)