ALICE Data Transfer

Howto:

according to the requests in the Transfer Request Wiki (https://alice-wiki.gsi.de/Data/TransferRequests)

config files requesting data transfers should be placed in the transferinput Directory (see below).

See example config files in the transferprocessed Directory (see below)

A config file should have the following naming convention:

• config_<Name of Person writing the config>_<data set>_<Name of Person requesting the Transfer>_<sim/data>_<esd/aod>_<ev. reconstruction pass>_<collision particles>_<energy>
  • e.g.: config_ks_LHC10e_sweber_data_esd_pass4_pp_7tev

A config file should have the following structure:

note: one configuration parameter per line, it needs to be 13 configuration lines in total before the run numbers start

• top level directory in AliEn <e.g. /alice/data/2010/LHC10e>
• name of text file which contains the list of files which could not be copied
  • naming convention: should be the same name as the config file but starting with "errorfiles"
    • e.g. errorfiles_ks_LHC10e_sweber_data_esd_pass4_pp_7tev
• number of parallel transfers <e.g. 10>
• top level target directory on Lustre <e.g. /lustre/nyx/alice/alien>
• string for AliEn find command <e.g. /pass4/*.*/root_archive.zip>
• TTL of a copy process in seconds <e.g. 4000>
• list of black listed SEs (should be 4 entries in the config files)
  • e.g.:
    • ALICE::NECTEC::SE
    • ALICE::LBL::SE
    • ALICE::KISTI_GSDC::SE
    • ALICE::Hiroshima::SE
• copy mode (e.g. standard)
  • alternative copy modes:
    • listmode (copies from a list of files instead of creating ist own list via alien_find)
      • in listmode the errorfile (position two of config file) takes the role of specifying the list of files to be copied
        • in this mode obsolete Parameters are:
          • top Level Directory in AliEn
          • string for AliEn find command
          • list of runs to be copied
    • random (copies randomly a specified percentage of a run)
      • Parameter "additional Parameter" (see below) Needs to contain the percentage in number (e.g. 50)
• additional parameters for copy mode
  • e.g. (NA for Standard mode)
• check size of data set (enabled by a value of 1, since this is very time consuming a value of 0 should be the default)
• list of successfully copied files (file name to be given)
• collision system (pp, pPb, PbPb)
• collision energy (2.76TeV, 7TeV, 8TeV, 2.76ATeV, 5.023ATeV, ...)
• base dir for run lists for train Analysis
  • /hera/alice/train/lists/transfer
  • /lustre/nyx/alice/train/lists/transfer
• external script for creating train lists
• list of runs to be copied
  • e.g.
    • 000130850
    • 000130848
    • 000130847

Current Data Transfer Boxes:

• lxalitransfer5: Transfer to /hera/alice/alien
• lxalitransfer6: Transfer to /lustre/nyx/alice/alien

Access via ssh key.

Precondition: member in the alidata group

Directory Structure:

• /home/ali-data
  • alien (link to AliEn Installation)
  • bin (contains executables)
  • scripts (contains setup script)
  • Transfers
    • tmp (contains list of transfers to be processed)
    • transferinput (here the config files for data transfers should be placed)
    • transferlogs (contains log files of data transfers)
    • transferprocessed (contains processed data transfer config files and list of files which could not be copied)
    • transferworkdir (workdir of the transfer script: contains temporary files)

Infrastructure:

• 2 cron jobs (crontab -l)
  • proxy-renewal (renews Grid Proxy, runs once per day at 5:35 am)
  • transfermaster (checks content of transferinput and starts transfers if config files are present, runs every 2 hours)

-- KilianSchwarz - 13 Nov 2015