Getting Started with ECS - C + S3 (libs3)

Introduction

The most common way to interface with S3 from a C/C++ application is to use libs3.  Libs3 provides an interface to write multithreaded applications in C.  It even allows for single-threaded applications to execute concurrent requests by using a callback mechanism.  By default, libs3 will connect to Amazon, but by overriding the hostName parameter you can connect to ECS.

 

Installation

Libs3 is available in most Linux distributions.  We recommend you use the March 22, 2016 version or newer to ensure you have fixes for a couple bugs we found in our testing.  As of this writing, CentOS 7 is shipping the correct version so you can simply install it with:

 

$ sudo yum install libs3-devel


 

Configuration

Below is a test application that will create a bucket and upload an object into the bucket.  To connect to ECS, you'll need your endpoint, access key (userid), and secret key.  In general, we recommend you use "path-style" bucket addressing to increase compatibility with various private installations of ECS since virtual-hosted buckets require wildcard DNS and SSL certificates to be properly configured.  When connecting to ECS Test Drive, your configuration will look something like this:

 

    /* hostname can include the port, e.g. 10.1.1.1:9020 */
    char hostname[] = "object.ecstestdrive.com";
    S3Protocol protocol = S3ProtocolHTTPS;
    /* We recommend path-style for ECS since it removes the requirement for wildcard DNS/SSL */
    S3UriStyle uriStyle = S3UriStylePath;
    char bucketName[] = "libs3testbucket";
    /* In ECS terms, this is your object user */
    char accessKeyId[] = "xxxxxxxxxx@ecstestdrive.emc.com";
    char secretAccessKey[] = "yyyyyyyyyyyyyyyyyyyyyygWeRiQE";


 

When connecting to a local ECS, you will probably be using the default ECS ports for S3 of 9020 (HTTP) and 9021 (HTTPS).  In this case, you can insert the port number in the hostname parameter.  A local connection by IP would look something like this:

 

    /* hostname can include the port, e.g. 10.1.1.1:9020 */
    char hostname[] = "10.1.83.51:9020";
    S3Protocol protocol = S3ProtocolHTTP;
    /* We recommend path-style for ECS since it removes the requirement for wildcard DNS/SSL */
    S3UriStyle uriStyle = S3UriStylePath;
    char bucketName[] = "libs3testbucket";
    /* In ECS terms, this is your object user */
    char accessKeyId[] = "jason";
    char secretAccessKey[] = "eeeeeeeeasdfasdfewer/SgKZpyXTgQGXnD8y";


 

When creating a bucket, all the arguments are passed separately, so the call will end up looking like this:

 

    S3_create_bucket(protocol, accessKeyId, secretAccessKey, NULL, hostname, bucketName, S3CannedAclPrivate, NULL, NULL, &responseHandler, (void*)&operationStatus);


 

However, once the bucket is created, the rest of the functions will take an S3BucketContext struct so you can configure the structure with the above parameters:

 

    S3BucketContext ctx;
    ctx.hostName = hostname;
    ctx.protocol = protocol;
    ctx.uriStyle = uriStyle;
    /* In ECS terms, this is your object user */
    ctx.accessKeyId = accessKeyId;
    ctx.secretAccessKey = secretAccessKey;
    /* ECS Doesn't use this yet */
    ctx.securityToken = NULL;
    /* The name of a bucket to use */
    ctx.bucketName = bucketName;


 

Then a put object request will use the context structure:

 

    S3_put_object(&ctx, "hello.txt", strlen(hello), NULL, NULL, &putObjectHandler, &operationStatus);


 

The full source of the application looks like this and is attached at the bottom of the post. 

 

#include <libs3.h>
#include <stdio.h>
#include <string.h>

/***
* Some Dummy callbacks
*/
S3Status responsePropertiesCallback(
                const S3ResponseProperties *properties,
                void *callbackData)
{
    return S3StatusOK;
}


static void responseCompleteCallback(
                S3Status status,
                const S3ErrorDetails *error,
                void *callbackData)
{
    if(callbackData != NULL) {
        S3Status *cb = (S3Status*)callbackData;
        *cb = status;
    }
    return;
}


S3ResponseHandler responseHandler =
{
        &responsePropertiesCallback,
        &responseCompleteCallback
};


static const char hello[] = "Hello World!";

static int helloPutCallback(int bufferSize, char *buffer, void *callbackData) {
    memcpy(buffer, hello, strlen(hello));
    return strlen(hello);
}

int main(int argc, char **argv) {
    S3BucketContext ctx;
    S3PutObjectHandler putObjectHandler;
    /* hostname can include the port, e.g. 10.1.1.1:9020 */
    char hostname[] = "10.1.83.51:9020";
    S3Protocol protocol = S3ProtocolHTTP;
    /* We recommend path-style for ECS since it removes the requirement for wildcard DNS/SSL */
    S3UriStyle uriStyle = S3UriStylePath;
    char bucketName[] = "libs3testbucket";
    /* In ECS terms, this is your object user */
    char accessKeyId[] = "jason";
    char secretAccessKey[] = "ApGpRSbJt74uTm6k1hBgdQN/SgKZpyXTgQGXnD8y";
    //char accessKeyId[] = "130750084874740523@ecstestdrive.emc.com";
    //char secretAccessKey[] = "A/QcKfHzToP0M2NpwHcAe5rPLcXKjpZbBgWeRiQE";

    S3Status operationStatus;

    /* Init LibS3 */
    S3_initialize(NULL, S3_INIT_ALL, NULL);

    /**
     * Create a bucket.
     */
    printf("Creating bucket %s\n", bucketName);
    S3_create_bucket(protocol, accessKeyId, secretAccessKey, NULL, hostname, bucketName, S3CannedAclPrivate, NULL, NULL, &responseHandler, (void*)&operationStatus);
    if(operationStatus != S3StatusOK) {
        printf("Create bucket failed with S3Status: %d\n", operationStatus);
        return 1;
    }
    printf("Created bucket %s successfully\n", bucketName);


    /**
     * Create the S3BucketContext that contains all the connection info.  This will be used
     * for interacting with the bucket after it's created.
     */
    ctx.hostName = hostname;
    ctx.protocol = protocol;
    ctx.uriStyle = uriStyle;
    /* In ECS terms, this is your object user */
    ctx.accessKeyId = accessKeyId;
    ctx.secretAccessKey = secretAccessKey;
    /* ECS Doesn't use this yet */
    ctx.securityToken = NULL;
    /* The name of a bucket to use */
    ctx.bucketName = bucketName;

    /**
     * Create an object in the bucket
     */
    printf("Creating object hello.txt\n");
    putObjectHandler.responseHandler = responseHandler;
    putObjectHandler.putObjectDataCallback = &helloPutCallback;
    S3_put_object(&ctx, "hello.txt", strlen(hello), NULL, NULL, &putObjectHandler, &operationStatus);
    if(operationStatus != S3StatusOK) {
        printf("Create object failed with S3Status: %d\n", operationStatus);
        return 1;
    }
    printf("Created object hello.txt successfully\n");

    /* Destroy LibS3 */
    S3_deinitialize();

    return 0;
}



 

To compile this as an application, run gcc and link with libs3:

 

$ gcc -l s3 -o s3test libs3test.c