This is an entry in a series explaining how you can use Cloud Sidecar to take your software and run it multiple clouds with very little code change. What is Cloud Sidecar? It is an application that runs next to your software as a sidecar, intercepts cloud API requests, and converts the API calls to a different cloud provider without your software noticing. Since it intercepts API calls, it works for virtually any software or language. In this entry I will show you how to convert your software from using Amazon S3 to Google GCS (Google Cloud Storage).

How To Do It

Make sure you have Golang installed, preferably 1.15.x or higher

Clone Cloud Sidecar from github

Run ./build.sh to build the binary for your target architecture

Create a config file named config.yaml. It should look something like

aws_configs:
  main_s3:
    service_type: "s3"
    port: 3450
    aws_destination_config:
      name: "example"
      access_key_id: "bleh" # place aws access key here
      secret_access_key: "bleh" # place aws secret key here
    gcp_destination_config:
      name: "example"
      key_file_location: "/location/to/gcp/creds.json" # place path to gcp creds here
      gcs_config:
        multipart_db_directory: "/tmp/"

Alter your application code to connect to connect to localhost on port 3450. Here are examples on how to do it in Python and Java

Python

import boto3
import os

client = boto3.resource(
    "s3",
    region_name="us-east",
    endpoint_url='http://localhost:3450',
    use_ssl=False,
)

client.Bucket("cool-bucket").put_object(
    Key = "/my-file", 
    Body = open("/home/someuser/sourcefile", 'rb')
)

Java

import com.amazonaws.AmazonServiceException;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;

final AmazonS3 s3 = AmazonS3ClientBuilder
    .standard()
    .withPathStyleAccessEnabled(true)
    .withEndpointConfiguration(new EndpointConfiguration("http://localhost:3450", Regions.US_EAST_1.getName))
    .build()
s3.putObject(bucket_name, key_name, new File(file_path));

Start up Cloud Sidecar by running ./cloudsidecar --config=config.yaml

And now your software is talking to GCS!!!

This was a very simple example, but can easily be extended to more complex applications, use Kubernetes, etc… If you have any questions feel free to reach out to me at larry@cloudsidecar.com or drop a comment in this blog entry.