This is an entry in a series explaining how you can use Cloud Sidecar to take your software and run it multiple clouds with very little code change. What is Cloud Sidecar? It is an application that runs next to your software as a sidecar, intercepts cloud API requests, and converts the API calls to a different cloud provider without your software noticing. Since it intercepts API calls, it works for virtually any software or language. In this entry I will show you how to convert your software from using Amazon S3 to Google GCS (Google Cloud Storage).
How To Do It
Make sure you have Golang installed, preferably 1.15.x or higher
Clone Cloud Sidecar from github
Run ./build.sh
to build the binary for your target architecture
Create a config file named config.yaml. It should look something like
aws_configs:
main_s3:
service_type: "s3"
port: 3450
aws_destination_config:
name: "example"
access_key_id: "bleh" # place aws access key here
secret_access_key: "bleh" # place aws secret key here
gcp_destination_config:
name: "example"
key_file_location: "/location/to/gcp/creds.json" # place path to gcp creds here
gcs_config:
multipart_db_directory: "/tmp/"
Alter your application code to connect to connect to localhost on port 3450. Here are examples on how to do it in Python and Java
Python
import boto3
import os
client = boto3.resource(
"s3",
region_name="us-east",
endpoint_url='http://localhost:3450',
use_ssl=False,
)
client.Bucket("cool-bucket").put_object(
Key = "/my-file",
Body = open("/home/someuser/sourcefile", 'rb')
)
Java
import com.amazonaws.AmazonServiceException;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
final AmazonS3 s3 = AmazonS3ClientBuilder
.standard()
.withPathStyleAccessEnabled(true)
.withEndpointConfiguration(new EndpointConfiguration("http://localhost:3450", Regions.US_EAST_1.getName))
.build()
s3.putObject(bucket_name, key_name, new File(file_path));
Start up Cloud Sidecar by running ./cloudsidecar --config=config.yaml
And now your software is talking to GCS!!!
This was a very simple example, but can easily be extended to more complex applications, use Kubernetes, etc… If you have any questions feel free to reach out to me at larry@cloudsidecar.com or drop a comment in this blog entry.
One thought on “S3 to GCS Conversion using Cloud Sidecar”
Comments are closed.