FTS User Guide

This document describes the user interface to FTS, in other words how to submit and manage transfers. Longer, detailed guide can be found in  gLite User Manual.

Channel management and service configuration are described in the admin guide.

Overview

FTS transfers are managed via glite-transfer-* commands.

The main commands are:

glite-transfer-submit
Submits a transfer job
glite-transfer-status
Displays the status of an ongoing transfer job
glite-transfer-list
Lists all submitted transfer jobs owned by the user
glite-transfer-cancel
Cancels a transfer job

Proxies

You will need a valid proxy in order to use these commands.

On first use, the glite-transfer-submit command will automatically delegate your proxy to the FTS server (default lifetime is 12 hours).

When the remaining lifetime of the stored proxy passes under 4 hours, glite-transfer-submit will automatically delegate a new one.

Currently the _only_ way a Myproxy server can be used is via the legacy "-p" option. Proxies delegated to the FTS have to be renewed by the _user_, because the FTS renewal code does not work for VOMS attributes.

Submitting a transfer

The user can submit a transfer either by specifying the source-destination pair in the command line:

$ glite-transfer-submit -s https://fts.cnaf.infn.it:8443/sc3/glite-data-transfer-fts/services/FileTransfer \
srm://srm.sara.nl/pnfs/srm.sara.nl/data/lhcb/doe/zz_zz.f \
srm://srm.cnaf.infn.it/castor/cnaf.infn.it/grid/lcg/lhcb/test/SARA_1.25354

c2e2cdb1-a145-11da-954d-944f2354a08b

or by specifying all source-destination pairs in an input file (bulk submission).

The -s option specifies the FTS service endpoint to be contacted. If the service starts with http://, https:// or httpg:// it is taken as a direct service endpoint URL; otherwise is taken as a service instance name and Service Discovery is invoked to look up the endpoints. If not specified the first available transfer service from the Service Discovery will be used. This is true for all subsequent examples.

 $ glite-transfer-submit -s https://fts.cr.cnaf.infn.it:8443/sc3/glite-data-transfer-fts/services/FileTransfer \
SARA-CNAF.in -p $passwd

where the input file SARA-CNAF.in looks like:

$ cat SARA-CNAF.in
srm://srm.grid.sara.nl/pnfs/grid.sara.nl/data/lhcb/test/doe/zz_zz.f \
srm://sc.cr.cnaf.infn.it/castor/cnaf.infn.it/grid/lcg/lhcb/test/doe/SARA_1.25354
srm://srm.grid.sara.nl/pnfs/grid.sara.nl/data/lhcb/test/doe/zz_zz.f \
srm://sc.cr.cnaf.infn.it/castor/cnaf.infn.it/grid/lcg/lhcb/test/doe/SARA_2.25354
srm://srm.grid.sara.nl/pnfs/grid.sara.nl/data/lhcb/test/doe/zz_zz.f \
srm://sc.cr.cnaf.infn.it/castor/cnaf.infn.it/grid/lcg/lhcb/test/doe/SARA_3.25354
.....

Attention! The transfers handled by FTS within a single job bulk submission must be all assigned to the same channel, otherwise FTS will not process such transfers and will return the message: Inconsistent channel.

Querying a transfer

The following example shows a query to FTS to infer information about the state of a transfer job:

$ glite-transfer-status \
-s https://fts.cnaf.infn.it:8443/sc3/glite-data-transfer-fts/services/FileTransfer \
-l c2e2cdb1-a145-11da-954d-944f2354a08b
Pending
Source:
Destination:
SARA_1.25354
State:
Retries:
Reason:
Duration:
srm://srm.grid.sara.nl/pnfs/grid.sara.nl/data/lhcb/test/doe/zz_zz.f
srm://sc.cr.cnaf.infn.it/castor/cnaf.infn.it/grid/lcg/lhcb/test/doe/
Pending
0
(null)
0

Attention! The verbosity level of the status of a given job is set with the -v option; the status of individual files is however available through the option -l.

The following example allows to query all ongoing data transfers in the specified (intermediate) state in a defined FTS service. In order to list only the transfer jobs relative to a channel, this must be specified with the -c option.

$ glite-transfer-list \
-s https://fts.cnaf.infn.it:8443/sc3/glite-data-transfer-fts/services/FileTransfer Pending
...
c2e2cdb1-a145-11da-954d-944f2354a08b Pending
...

Cancelling a transfer

An example of cancellation of a previously submitted data transfer job is shown here:

$ glite-transfer-cancel -s https://fts.cnaf.infn.it:8443/sc3/glite-data-transfer-fts/services/FileTransfer c2e2cdb1-a145-11da-954d-944f2354a08b

Architecture

The FTS service comprises an FTS server used for configuration and user-level transfer management, and two types of agent which execute the transfers.

FTS Web service ("FTS")

This component allows users to submit FTS jobs and query their status. It is the only component that users interact with. It runs as a Tomcat web-application (Java based). The node also has a local BDII with a GIP publishing the necessary information about this FTS server (the site BDII should configured to pull this information).

Referred to throughout a node type FTS.

FTS agents ("FTA")

These are the back-end agents that do the work of the service. Each agent runs as a distinct daemon, and you may have as many agents daemons running on a node as it can support. There are two main type (channel and VO agent) which may be mixed across nodes as necessary.

Referred to throughout a node type FTA.

FTS agents: channel agent

Each network channel, (e.g. "transfers from CERN to RAL") has a distinct daemon running transfers for it. The daemon is responsible for starting and controlling transfers on the associated network link. When a transfer is started, a controlling process for that transfer is (double) forked from the controlling agent.

There should be one agent daemon for every channel that the FTS has defined, each managing a different channel.

Since they produce a large number of forked processes, the channel agent daemons are generally spread over a number of agent nodes ~equally.

FTS agents: VO agents

Each VO served by the FTS service has a distinct VO agent daemon running for it. This performs house-keeping tasks for that VO. There should be one VO agent daemon for every VO that will use the FTS service.

The VO agents consume very little resources and can be put freely on any agent node in the cluster.

Credit

This document is based on the gLite User Guide.

Attachments