Java Client for Amazon S3 with AWS SDK

AWS SDK provides a series of tools and APIs for developing and managing applications that work with Amazon AWS services such as EC2, S3, SimpleDB and so on. In this post I talk about J4S3, a command-line S3 client written in java which allows you to store and retrieve data to/from the AWS S3 cloud storage. I have to admit I started with this project to just play with the AWS Java APIs. Later I realized that it was need a way to encrypt my AWS credentials.

J4S3 stands for “Java For S3” and consists of a maven-based java project which makes use of the AWS Java APIs for AWS infrastructure services. The maven process for building and distributing the J4S3 artifact, generates an executable jar which allows a user to:

• encrypt the AWS credentials in a password-based encrypted file;
• send/receive objects on Amazon S3 bucket from a command line interface.

J4S3 comes from a number of needs:

• interact with S3 from remote workstations where only a SSH access is provided;
• use a java-based command line tool, as alternative to other well-known clients like AWS CLI o S3cmd;
• protect the confidentiality of AWS credentials by a password-based encryption (by default the AWS Id and AWS Key are clear text in a file typically located at ~/.aws/credential);
• avoid the REST API or the classic S3 web interface;
• customize AWSCredentialsProvider, the AWS SDK Java interface for providing AWS credentials.

Here is an image that shows the objects upload on the ES_WEST_1 region .

To access any AWS services, you must first create an AWS account. Upon sign-up, new AWS users receive 5 GB of Amazon S3 standard storage. Requests to AWS must be signed, that is, they must include information that AWS can use to authenticate the requestor. Requests are signed using the access key ID and secret access key of an account or of an IAM user.  J4S3 make use of IAM user access keys to send authenticated requests to S3. These keys (access ID and access KEY) are stored into a file with password-based encryption. You can use the interactive J4S3 CLI to generate first the encrypted file with your AWS credentials, then to send your files to a S3 bucket ( S3 stores data as objects within resources called “buckets”) . The following images show the two steps.

In order to quick build a testable version of the J4S3 tool, I’ve recycled an old project which makes use of JCA and Bouncycastle frameworks for a password-based encryption solution. I posted an example of asymmetric RSA encryption with Java JCA in this article.

The above image shows the class diagram of J4S3 artifact. PBECredentialsProvider implements the AWSCredentialsProvider interface for providing AWS credentials. The AWS user credentials come from a password-based encrypted file. J4S3 uses PasswordBasedEncryption object of jca artifact to encrypt credentials. The TransportAgent class is responsible for handling the transfer of directories and files from/to the S3 bucket. Finally the UI package provides the classes for the implementation of a minimal interactive command line interface which allows a user to generate the encrypted file with own credentials and send or receive objects to/from S3. The main() method in the CLI class makes use of the System.in object to manage the user input when the J4S3 executable jar is running.

Here is a snippet of code of my customized version AWSCredentialsProvider.

[...]
public class PBECredentialsProvider implements AWSCredentialsProvider {
[...]
public AWSCredentials getCredentials() {
		
		String awsAccessId="";
		String awsAccessKey="";
		try {
			awsAccessId = retrieveAWSKey(awsAccessIdProperty);
			awsAccessKey = retrieveAWSKey(awsAccessKeyProperty);
		} catch (UnsupportedEncodingException e) {
			e.printStackTrace();
		} catch (Exception e) {
			e.printStackTrace();
		}
		AWSCredentials awsc = new BasicAWSCredentials(awsAccessId, awsAccessKey);
		return awsc;
	}

Another very interesting object of AWS SDK is MultipleFileUpload that allows you to upload a single object as a set of parts (useful when your file size reaches 100 MB). This class provides also a series of useful methods which allow for example to check the transfer status and calculate the percentage of data transferred. Follows the code snippet of a simple progress bar that displays the transfer status to the user (I calculate also the approximate total bytes of data transferred at the end of the upload process).

private void uploadProgressBar(MultipleFileUpload multipleFileUpload){
      System.out.print("Progress:[=");
      try {
       while (multipleFileUpload.isDone() == false) {
        long perc = (long) multipleFileUpload.getProgress().getPercentTransferred();
        System.out.print("= ");
        System.out.print(perc+"%");
        if (perc<10) {
          System.out.print("\b\b\b");
        }else {
         System.out.print("\b\b\b\b");
        }
        Thread.currentThread();
        Thread.sleep(2000);
       }
      }catch (InterruptedException e) {
       e.printStackTrace();
      }
      System.out.print("\b] Done "+(long) multipleFileUpload.getProgress().getPercentTransferred()+" %\n");
      long total = multipleFileUpload.getProgress().getBytesTransferred();
      String roundOff = new DecimalFormat("#.##").format((double)total/1000000);
      System.out.println("Total transferred: "+total +" bytes (~ "+roundOff+ " MB)");
}

Finally I would also include a deep investigation about the network traffic during a session of data upload with J4S3. As you can see in the above image, all packet are securely transferred on Amazon S3 using SSL protocol. AWS APIs make use of  HTTPS connections for requests and data transfer. I was corious about this, so I used tcpdump and wireshark to capture traffic during a session of upload with J4S3.