This is an entry in a series explaining how you can use Cloud Sidecar to take your software and run it multiple clouds with very little code change. What is Cloud Sidecar? It is an application that runs next to your software as a sidecar, intercepts cloud API requests, and converts the API calls to a different cloud provider without your software noticing. Since it intercepts API calls, it works for virtually any software or language. In this entry I will show you how to convert your software from using Amazon S3 to Google GCS (Google Cloud Storage).
How To Do It
Make sure you have Golang installed, preferably 1.15.x or higher
Clone Cloud Sidecar from github
./build.sh to build the binary for your target architecture
Create a config file named config.yaml. It should look something like
aws_configs: main_s3: service_type: "s3" port: 3450 aws_destination_config: name: "example" access_key_id: "bleh" # place aws access key here secret_access_key: "bleh" # place aws secret key here gcp_destination_config: name: "example" key_file_location: "/location/to/gcp/creds.json" # place path to gcp creds here gcs_config: multipart_db_directory: "/tmp/"
Alter your application code to connect to connect to localhost on port 3450. Here are examples on how to do it in Python and Java
import boto3 import os client = boto3.resource( "s3", region_name="us-east", endpoint_url='http://localhost:3450', use_ssl=False, ) client.Bucket("cool-bucket").put_object( Key = "/my-file", Body = open("/home/someuser/sourcefile", 'rb') )
import com.amazonaws.AmazonServiceException; import com.amazonaws.regions.Regions; import com.amazonaws.services.s3.AmazonS3; final AmazonS3 s3 = AmazonS3ClientBuilder .standard() .withPathStyleAccessEnabled(true) .withEndpointConfiguration(new EndpointConfiguration("http://localhost:3450", Regions.US_EAST_1.getName)) .build() s3.putObject(bucket_name, key_name, new File(file_path));
Start up Cloud Sidecar by running
And now your software is talking to GCS!!!
This was a very simple example, but can easily be extended to more complex applications, use Kubernetes, etc… If you have any questions feel free to reach out to me at email@example.com or drop a comment in this blog entry.
One thought on “S3 to GCS Conversion using Cloud Sidecar”
Comments are closed.