Transfer Service sample using Python

This app creates two types of transfers using the Transfer Service tool.

These samples are used on the following documentation pages:

• https://cloud.google.com/storage/transfer/create-client
• https://cloud.google.com/storage/transfer/create-manage-transfer-program

Prerequisites

1. Set up a project on Google Developers Console.
2. Go to the Developers Console and create or select your project. You will need the project ID later.
3. Within Developers Console, select APIs & auth > Credentials.
4. Select Add credentials > Service account > JSON key.
5. Set the environment variable GOOGLE_APPLICATION_CREDENTIALS to point to your JSON key.
6. Add the Storage Transfer service account as an editor of your project [email protected]
7. Set up gcloud for application default credentials.
8. gcloud components update
9. gcloud init
10. Install Google API Client Library for Python.

Transfer from Amazon S3 to Google Cloud Storage

Creating a one-time transfer from Amazon S3 to Google Cloud Storage.

1. Set up data sink.
2. Go to the Developers Console and create a bucket under Cloud Storage > Storage Browser.
3. Set up data source.
4. Go to AWS Management Console and create a bucket.
5. Under Security Credentials, create an IAM User with access to the bucket.
6. Create an Access Key for the user. Note the Access Key ID and Secret Access Key.
7. In aws_request.py, fill in the Transfer Job JSON template with relevant values.
8. Run with python aws_request.py
9. Note the job ID in the returned Transfer Job.

Transfer data from a standard Cloud Storage bucket to a Cloud Storage Nearline bucket

Creating a daily transfer from a standard Cloud Storage bucket to a Cloud Storage Nearline bucket for files untouched for 30 days.

1. Set up data sink.
2. Go to the Developers Console and create a bucket under Cloud Storage > Storage Browser.
3. Select Nearline for Storage Class.
4. Set up data source.
5. Go to the Developers Console and create a bucket under Cloud Storage > Storage Browser.
6. In nearline_request.py, fill in the Transfer Job JSON template with relevant values.
7. Run with python nearline_request.py
8. Note the job ID in the returned Transfer Job.

Checking the status of a transfer

1. In transfer_check.py, fill in the Transfer Job JSON template with relevant values. Use the Job Name you recorded earlier.
2. Run with python transfer_check.py