ECS S3 Java Extensions

The EMC ViPR Java SDK implements some useful extensions above and beyond the standard S3 protocol:

 

  • Object Append
  • Object Update
    • Replace Range
    • Extend Object
    • Sparse Object

 

To use these operations, you will need to use the ViPR S3 client instead of the standard AWS client.  See the Java SDK quickstart for instructions on how to upgrade your application.  The new methods exist in the com.emc.vipr.services.s3.ViPRS3 interface and are implemented by the com.emc.vipr.services.s3.ViPRS3Client class.

 

Object Append

The Object Append operation allows you to atomically append data to an existing object.  This is useful for applications such as logging where you want to write data incrementally to an object.  Note that this API does not allow you to specify the offset where the data is written, if multiple append requests arrive concurrently, they will be processed serially and the offset they were actually written at will be returned in the response.

 

For example, we can first create an object containing "Hello"

 

        String bucket = "testbucket";
        String key = "testkey";
        String testString = "Hello";
        byte[] data = testString.getBytes();
        int length1 = data.length;
        ObjectMetadata om = new ObjectMetadata();
        om.setContentLength(length1);
        om.setContentType("text/plain");
     
        // Create Object
        vipr.putObject(bucket, key, new ByteArrayInputStream(data), om);



 

Next, we can append " World!" to the existing object:

    

        // Append Object
        String testString2 = " World!";
        data = testString2.getBytes();
        om.setContentLength(data.length);
        AppendObjectResult appendRes = vipr.appendObject(bucket, key, new ByteArrayInputStream(data), om);
        System.out.println("Data appended to object at offset " + appendRes.getAppendOffset());



 

The method appendObject is the new method in the ViPRS3 interface that performs the atomic append operation.  The actual append offset is returned in the AppendObjectResult object.  Next, we read back the object to confirm its contents:

    

        // Read Back
        S3Object s3o = vipr.getObject(bucket, key);
        InputStream in = s3o.getObjectContent();
        data = new byte[length1+data.length];
        in.read(data);
        in.close();
        String outString = new String(data);
        System.out.println("Object content: " + outString);



 

Running the application outputs:

 

Data appended to object at offset 5
Object content: Hello World!



 

This is a very basic example.  For a more in-depth sample, see the testParallelAppends method in com.emc.vipr.services.s3.AppendTest.  This test creates 8 threads to concurrently perform 64 appends.  Since there are multiple execution threads, it is not guaranteed that all 64 appends will apply in order, but by using the append offset it is easy to verify that all the appends completed successfully.

 

Object Update

You can also update existing objects in ECS.  This can be done in one of three ways: replace a range, extend an object, or create a sparse object.

 

Replace Range

To replace a range, simply execute an UpdateObjectRequest with the exact range you want to replace.  Note that ranges are given as offsets from the beginning of the object and are inclusive (i.e. the ending byte is the last byte you want to replace -- a slight deviation from things like Java.String.substring() where the ending offset is exclusive).

 

For example, if you want to create an object with "Hello World!":

 

        // Create base object.
        String key = "testkey";
        String bucket = "testbucket";
        String testString = "Hello World!";
        byte[] data = testString.getBytes("US-ASCII");
        ObjectMetadata om = new ObjectMetadata();
        om.setContentLength(data.length);
        om.setContentType("text/plain");
     
        vipr.putObject(bucket, key, new ByteArrayInputStream(data), om);



 

And change it to "Hello Again!" by updating bytes 6-10:

 

        // Update "World" to "Again"
        String updateString = "Again";
        data = updateString.getBytes("US-ASCII");
        om = new ObjectMetadata();
        om.setContentLength(data.length);
        UpdateObjectRequest r = new UpdateObjectRequest(bucket, key,
                new ByteArrayInputStream(data), om).withUpdateRange(6, 10);
        vipr.updateObject(r);



Extend Object

You can also extend an object by giving an end offset beyond the end of the object.  To start, we'll create a base object containing "Hello World!":

 

        // Create base object.
        String bucket = "testbucket";
        String key = "testkey";
        String testString = "Hello World!";
        byte[] data = testString.getBytes("US-ASCII");
        ObjectMetadata om = new ObjectMetadata();
        om.setContentLength(data.length);
        om.setContentType("text/plain");
     
        vipr.putObject(bucket, key, new ByteArrayInputStream(data), om);



 

Next, we'll update with "Again and Again!", starting at offset 5.  This will overwrite "World!" but since the stream is longer than the existing object, the object will be extended:

 

        // Update content and extend the object
        String updateString = "Again and Again!";
        data = updateString.getBytes("US-ASCII");
        om = new ObjectMetadata();
        om.setContentLength(data.length);
        UpdateObjectRequest r = new UpdateObjectRequest(bucket, key,
                new ByteArrayInputStream(data), om).withUpdateOffset(6);
        vipr.updateObject(r);



 

We can read back the content to confirm:

 

        S3Object s3o = vipr.getObject(bucket, key);
        InputStream in = s3o.getObjectContent();
        data = new byte[255];
        int c = in.read(data);
        in.close();
        String outString = new String(data, 0, c, "US-ASCII");
        System.out.println("Extended Object: " + outString);



 

And the application will output:

 

Extended Object: Hello Again and Again!



 

Sparse Object

If the starting offset of the update is beyond the current end of the object, a sparse object will be created.  When the object is read back, zeroes will be returned for offsets where no data exists.

 

For example, we create a base object:


        // Create base object.
        String bucket = "testbucket";
        String key = "testkey";
        String testString = "Hello World!";
        byte[] data1 = testString.getBytes("US-ASCII");
        ObjectMetadata om = new ObjectMetadata();
        om.setContentLength(data1.length);
        om.setContentType("text/plain");
     
        vipr.putObject(bucket, key, new ByteArrayInputStream(data1), om);



 

Then we can add some data 1TB into the object:

 

        // Create a sparse object by appending some data 1TB into object
        String updateString = "I'm out there!";
        long offset = 1024L*1024L*1024L*1024L; // 1TB
        byte[] data2 = updateString.getBytes("US-ASCII");
        om = new ObjectMetadata();
        om.setContentLength(data2.length);
        UpdateObjectRequest r = new UpdateObjectRequest(bucket, key,
                new ByteArrayInputStream(data2), om).withUpdateRange(offset, offset+data2.length-1);
        vipr.updateObject(r);



 

We can read back the 1TB offset to verify

 

        GetObjectRequest getReq = new GetObjectRequest(bucket, key)
            .withRange(offset, offset+data2.length-1);
        S3Object obj = vipr.getObject(getReq);
        InputStream in = obj.getObjectContent();
        byte[] data = new byte[255];
        int c = in.read(data);
        in.close();
        String outString = new String(data, 0, c, "US-ASCII");
        System.out.printf("String at offset %d: %s\n", offset, outString);



 

We can also read back another offset in the middle to verify that it is all zeroes.

 

        // Confirm that read in the middle will return zeroes.
        GetObjectRequest zeroReq = new GetObjectRequest(bucket, key)
            .withRange(1000, 1003);
        S3Object zeroObj = vipr.getObject(zeroReq);
        byte[] expected = new byte[4];
        Arrays.fill(expected, (byte)0);
        byte[] actual = new byte[4];
        in = zeroObj.getObjectContent();
        in.read(actual);
        in.close();
     
        for(int i=0; i<actual.length; i++) {
            System.out.printf("Offset %d: %d\n", 1000+i, actual[i]);
        }



 

Running the application outputs:

 

String at offset 1099511627776: I'm out there!
Offset 1000: 0
Offset 1001: 0
Offset 1002: 0
Offset 1003: 0