Globus GridFTP S3 DSI
=====================

The Globus GridFTP S3 DSI implements a data storage interface between
a Globus GridFTP server and a S3-compatible object store. 

It attempts to make the object store look like a filesystem.  The namespace
served by this DSI is rooted at the S3 object store root.
Items seen as "directories" in the top level are "buckets" in S3 terminology.
Creating a new toplevel directory creates a new bucket. Creating a directory
within a bucket or a subdirectory of a bucket adds a 0 byte placeholder object
to the object store, with a name that ends with the '/' character and metadata
indicating it's to be viewed as a directory by Globus.

The S3 DSI is compatible with gridmap files and GSI authorization callouts.
When a GridFTP session begins, the local user name is interpreted by the S3
DSI as a string containing the S3 access key information.

== Installation

Install the package globus-gridftp-server-s3 from the Globus repository.

For Red Hat-based systems:

  % yum install globus-gridftp-server-s3

For Debian-based systems:

  % apt-get install globus-gridftp-server-s3

For SLES 11-based systems:

  % zypper install globus-gridftp-server-s3

== Configuration

The S3 DSI requires the following steps for configuration:

- Configure the S3 DSI
- Create a gridmap to S3 credentials
- Restart the GridFTP server

=== Configure the S3 DSI

Create the file /etc/gridftp.d/gridftp-s3 containing these lines:

------------------------------------------------------------------------------
 threads 2
 load_dsi_module s3
------------------------------------------------------------------------------

Edit the file /etc/globus/globus-gridftp-server-s3.conf and set the 'host_name' 
option to be the appropriate Amazon S3 endpoint hostname for the Amazon
region that hosts the bucket(s) you want the S3 DSI to access.

For example, to configure the S3 DSI to use buckets in the US Standard Region:
------------------------------------------------------------------------------
host_name = s3.amazonaws.com
------------------------------------------------------------------------------

A list of Amazon S3 endpoint hostnames by region can be found here:

http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region

=== Create a file containing s3 credentials

The default configuration for the S3 DSI looks in $HOME/.globus/s3 for a
file mapping the current user's ID to S3 access keys. Each user who will
be using the DSI must create such a file with their credentials.

------------------------------------------------------------------------------
 % mkdir -m 0700 -p ~/.globus
 % (umask 077; echo "$(id -un);$S3_ACCESS_KEY_ID;$S3_SECRET_ACCESS_KEY" \
    > ~/.globus/s3)
------------------------------------------------------------------------------

The S3_ACCESS_KEY_ID and S3_SECRET_ACCESS_KEY correspond to the Access Key ID
and Secret Access Key for the user's AWS account that has been granted 
access to the S3 buckets the user intends to access.

=== Restart the gridftp server

  # service globus-gridftp-server restart

== Debugging Tips

To enable a debugging log for the S3 dsi, set the environment
variable GLOBUS_S3_DEBUG "1023,/tmp/s3.log" to enable a highly verbose log
of the DSI. This can be easily done for a gridftp configuration by creating
a file /etc/gridftp.d/s3-debug with the contents

------------------------------------------------------------------------------
 $GLOBUS_S3_DEBUG "1023,/tmp/s3.log"
------------------------------------------------------------------------------
