Introduction

The SCIGNE platform <https://scigne.fr/>`_ offers a high-throughput computing (HTC) service based on technologies developed for computing grids. Data exchanges between the various computing centres that make up these grids are carried out using specific protocols (HTTPS and XROOTD).

This document details the use of the GFAL2 tool to interact with these protocols to manage your data from between your workstation and the SCIGNE platform.

HTC storage system

Several storage systems (dCache, Storm, etc) are used by the computing grid sites. In order to ensure that the systems are interoperable, they communicate using identical protocols.

The SCIGNE platform uses the dCache system to make HTC storage available. Although several protocols are available (HTTPS, XROOTD), this documentation focuses on the use of HTTPS.

The dCache storage system consists of a front-end server and a set of file servers. The front-end server centralises file access requests. The file servers physically store the files. Files are accessed by querying the front-end server using the commands detailed in this document.

Access to data is controlled by a certificate. It is therefore necessary to have a valid proxy when using this type of storage.

Data management with dCache

This section describes how to manage data stored on a dCache server.

Foreword

The following URL is used throughout this section. It will be abbreviated to ${BASE_URL}:

$ export BASE_URL="https://sbgdcache.in2p3.fr/vo.scigne.fr"

Client installation

Currently, packages are only available for the RedHat distribution and its derivatives. The packages are available in the EPEL repository. Once this repository is available on your system, the gfal2 tools can be quickly installed with :

$ dnf install gfal2-util gfal2-all

Reading the contents of a directory

The gfal-ls command lists the contents of a directory. The result of this command is similar to that of the ls command. The example below illustrates how this command works:

$ gfal-ls ${BASE_URL}
EOST
IBMC
IBMP
ICS
ICUBE
IPHC

$ gfal-ls -l ${BASE_URL}
drwxrwxrwx   0 0     0             0 Aug 22 07:42 EOST
drwxrwxrwx   0 0     0             0 Aug 22 07:42 IBMC
drwxrwxrwx   0 0     0             0 Aug 22 07:42 IBMP
drwxrwxrwx   0 0     0             0 Aug 22 07:42 ICS
drwxrwxrwx   0 0     0             0 Aug 22 07:42 ICUBE
drwxrwxrwx   0 0     0             0 Aug 22 07:43 IPHC

Note: As the contents of this directory change over time, the result displayed by the previous commands differs from the examples above.

Creation a directory

The gfal-mkdir command permits to create a directory:

$ gfal-mkdir ${BASE_URL}/lab/user/data

Note that the value /lab/user/data is given as an example and must be adapted to your specific use case. By convention:

  • lab is the acronym of the laboratory in uppercase ;

  • user is the concatenation of the first letter of the first name and the surname.

Copying a file

The gfal-copy command is used to copy data from or to the dCache server:

$ gfal-copy file://${PWD}/my_dataset.tgz ${BASE_URL}/lab/user/data/my_dataset.tgz
Copying file://${PWD}/my_dataset.tgz   [DONE]  after 2s

Deleting a file

The gfal-rm command permits to delete a file on the dCache server:

$ gfal-rm ${BASE_URL}/lab/user/data/my_dataset.tar.gz
https://sbgdcache.in2p3.fr/vo.scigne.fr/lab/user/my_dataset.tgz      DELETED

Deleting a directory

The gfal-rm -r command is used to delete a directory (and its contents):

$ gfal-rm -r ${BASE_URL}/lab/user/data

Additional documentation

Ces liens proposent des références complémentaires pour approfondir la gestion des données sur la plateforme SCIGNE :