Dpm/User/Tutorial

DPM (Disk Pool Manager)

The Disk Pool Manager (DPM) is a lightweight solution for disk storage management. It is part of LCGDM ( https://svnweb.cern.ch/trac/lcgdm )


Tutorial

This is a tutorial for DPM users (clients, not administrators).

Do man <command-name> for details on usage (export MANPATH=/opt/lcg/share/man in a default installation).

  1. Setting the environment
  2. Interacting with the namespace
  3. Getting and putting files
  4. Accessing the disk node via rfio
  5. Interacting using the SRM interface
  6. Interacting via HTTP

Setting the environment

# export DPNS_HOST=<dpm-headnode-host>
# export DPM_HOST=<dpm-headnode-host>

Creating your proxy, and specifying which VO you belong to (in this case, dteam).

# voms-proxy-init -voms dteam

Interacting with the namespace

You can easily access and modify the DPM namespace, listing files, creating directories, etc. Commands start with dpns-, you should find all expected unix like commands for interacting with the filesystem namespace.

# dpns-mkdir /dpm/cern.ch/home/dteam/test-dir
# dpns-ln /dpm/cern.ch/home/dteam/link-to-test-dir /dpm/cern.ch/home/dteam/test-dir
# dpns-ls -l /dpm/cern.ch/home/dteam
lrwxrwxrwx   1 102      103                       0 Jun 03 18:53 link-to-test-dir -> /dpm/cern.ch/home/dteam/test-dir
drwxrwxr-x   0 102      103                       0 Jun 03 18:52 test-dir

You can also manage permissions of course, including acl management.

# dpns-chmod 700 /dpm/cern.ch/home/dteam/test-dir
# dpns-getacl /dpm/cern.ch/home/dteam/test-dir

Getting and putting files

Usually you'll be doing this using the lcg-utils tools.

Let's start with uploading a file from your local host to the remote DPM. We pass the full SRM endpoint and the SRM type (-U option) as in this example we are not using the BDII.

# echo "this is a lcgcp test file" > test-lcgcp-file
# lcg-cp -b -D srmv2 test-lcgcp-file srm://$DPNS_HOST:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/test-lcgcp-file
# dpns-ls -l /dpm/cern.ch/home/dteam/test-lcgcp-file
-rw-rw-r--   1 102      103                      21 Jun 03 19:13 /dpm/cern.ch/home/dteam/test-lcgcp-file

Then we can copy it back to our local host, with a different name.

# lcg-cp -b -D srmv2 srm://$DPNS_HOST:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/test-lcgcp-file test-lcgcp-file-copy

And finally we'll delete it from the DPM (-l disables the attempt to remove it from the LFC catalog).

# lcg-del -b -l -D srmv2 srm://$DPNS_HOST:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/test-lcgcp-file

You can also use one of the lcg-utils commands to list the contents of a directory (similar to the direct namespace interaction mentioned above).

# lcg-ls -b -D srmv2 srm://$DPNS_HOST:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/
/dpm/cern.ch/home/dteam//link-to-test-dir
/dpm/cern.ch/home/dteam//test-dir

Accessing the disk node via rfio

If you node the direct location of a file in the disk node you can access it via rfio (it won't be more practical than use the tools above, but it is possible anyway).

Let's say you got a TURL (rfio) for a file using lcg-gt.

# lcg-gt -b -D srmv2 srm://$DPNS_HOST:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/test-lcgcp-file rfio
rfio://lxb7606v2.cern.ch//dpmfs/fs1/dteam/2010-06-03/test-lcgcp-file.9.0
031989c7-39ff-4226-9b03-6b2240bb260d

You can now use this TURL to directly access the file.

# rfcp rfio://lxb7606v2.cern.ch//dpmfs/fs1/dteam/2010-06-03/test-lcgcp-file.9.0 test-lcgcp-file-rfio
21 bytes in 0 seconds through eth0 (in) and local (out)

Most other rfio commands will not work as files are managed by DPM and you can't list/delete replicas directly.

Interacting using the SRM interface

You can achieve the same results as described above using the SRM tools directly.

# echo "this is a srm test file" >  test-srm-file
# srmcp -2 file:////afs/cern.ch/user/r/rocha/test-srm-file srm://$DPNS_HOST:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/test-srm-file

Same is true regarding directories.

# srmmkdir srm://$DPNS_HOST:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/test-srm-dir

And other namespace functionality.

# srmmv srm://$DPNS_HOST:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/test-srm-file srm://$DPNS_HOST:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/test-srm-dir
# srmls srm://$DPNS_HOST:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/test-srm-dir
0 /dpm/cern.ch/home/dteam/test-srm-dir/
24 /dpm/cern.ch/home/dteam/test-srm-dir/test-srm-file

Interacting via HTTP

Again, the same functionality as above, but using an HTTP interface.

Simply import your certificate into your browser (procedure is browser specific, so search for 'import certificate <your-browser-name>'), and visit:

https://<dpm-head-node>/dpm

You'll see a visual interface that allows file download / upload and navigation, in addition to the usual filesystem operations (rm, chmod, stat, ...).

You can also use our http transfer command line tool:

# echo 'my http test file' > my-http-test-file
# dpm-httpd-cp my-http-test-file https://$DPNS_HOST/dpm/cern.ch/home/dteam

Or do the usual namespace operations using:

# dpm-httpd-cmd ls https://$DPNS_HOST/dpm/cern.ch/home/dteam
# dpm-httpd-cmd mkdir https://$DPNS_HOST/dpm/cern.ch/home/dteam/new-http-dir