Introduction
============

The SCIGNE platform <https://scigne.fr/>`_ offers a high-throughput
computing (HTC) service based on technologies developed for computing
grids. Data exchanges between the various computing centres that make
up these grids are carried out using specific protocols (HTTPS and
XROOTD).

This document details the use of the GFAL2 tool to interact with
these protocols to manage your data from between your workstation and
the SCIGNE platform.


HTC storage system
==================

Several storage systems (dCache, Storm, etc) are used by the computing
grid sites. In order to ensure that the systems are interoperable,
they communicate using identical protocols.

The SCIGNE platform uses the dCache system to make HTC storage
available. Although several protocols are available (HTTPS, XROOTD),
this documentation focuses on the use of HTTPS.

The dCache storage system consists of a front-end server and a set of
file servers. The front-end server centralises file access requests.
The file servers physically store the files. Files are accessed by
querying the front-end server using the commands detailed in this
document.

Access to data is controlled by a certificate. It is therefore
necessary to have a valid *proxy* when using this type of storage.


Data management with dCache
===========================

This section describes how to manage data stored on a
dCache server.


Foreword
--------

The following URL is used throughout this section. It will be
abbreviated to ``${BASE_URL}``:

.. code-block:: console

   $ export BASE_URL="https://sbgdcache.in2p3.fr/vo.scigne.fr"


Client installation
-------------------

Currently, packages are only available for the RedHat distribution and
its derivatives. The packages are available in the `EPEL repository <https://fedoraproject.org/wiki/EPEL>`_.
Once this repository is available on your system, the ``gfal2`` tools
can be quickly installed with :

.. code-block:: console

   $ dnf install gfal2-util gfal2-all


Reading the contents of a directory
-----------------------------------

The ``gfal-ls`` command lists the contents of a directory. The result
of this command is similar to that of the ``ls`` command. The example
below illustrates how this command works:

.. code-block:: console

   $ gfal-ls ${BASE_URL}
   EOST
   IBMC
   IBMP
   ICS
   ICUBE
   IPHC

.. code-block:: console

   $ gfal-ls -l ${BASE_URL}
   drwxrwxrwx   0 0     0             0 Aug 22 07:42 EOST
   drwxrwxrwx   0 0     0             0 Aug 22 07:42 IBMC
   drwxrwxrwx   0 0     0             0 Aug 22 07:42 IBMP
   drwxrwxrwx   0 0     0             0 Aug 22 07:42 ICS
   drwxrwxrwx   0 0     0             0 Aug 22 07:42 ICUBE
   drwxrwxrwx   0 0     0             0 Aug 22 07:43 IPHC

**Note:** As the contents of this directory change over time, the result
displayed by the previous commands differs from the examples above.


Creation a directory
--------------------

The ``gfal-mkdir`` command permits to create a directory:

.. code-block:: console

   $ gfal-mkdir ${BASE_URL}/lab/user/data

Note that the value */lab/user/data* is given as an example and must
be adapted to your specific use case. By convention:

* **lab**  is the acronym of the laboratory in uppercase ;

* **user**  is the concatenation of the first letter of the first name
  and the surname.


Copying a file
--------------

The ``gfal-copy`` command is used to copy data from or to the dCache
server:

.. code-block:: console

   $ gfal-copy file://${PWD}/my_dataset.tgz ${BASE_URL}/lab/user/data/my_dataset.tgz
   Copying file://${PWD}/my_dataset.tgz   [DONE]  after 2s


Deleting a file
---------------

The ``gfal-rm`` command permits to delete a file on the dCache server:

.. code-block:: console

   $ gfal-rm ${BASE_URL}/lab/user/data/my_dataset.tar.gz
   https://sbgdcache.in2p3.fr/vo.scigne.fr/lab/user/my_dataset.tgz      DELETED


Deleting a directory
--------------------

The ``gfal-rm -r`` command is used to delete a directory (and its
contents):

.. code-block:: console

   $ gfal-rm -r ${BASE_URL}/lab/user/data


Additional documentation
========================

Ces liens proposent des références complémentaires pour approfondir la
gestion des données sur la plateforme SCIGNE :

-  `documentation sur les commandes gfal2 <https://dmc.web.cern.ch/projects/gfal-2/documentation>`_
   ;

-  `documentation sur dCache <https://www.dcache.org/manuals/UserGuide-8.2/>`_ ;