Manage Files in Google Cloud Storage With Python

Make sure the google-cloud-storage library is installed on your machine with pip3 install google-cloud-storage.

I’m going to set up our project with a config.

py file containing relevant information we’ll need to work with:"""Google Cloud Storage Configuration.

"""from os import environ# Google Cloud StoragebucketName = environ.

get('GCP_BUCKET_NAME')bucketFolder = environ.

get('GCP_BUCKET_FOLDER_NAME')# DatalocalFolder = environ.

get('LOCAL_FOLDER')bucketName is our bucket's given name.

The google-cloud-storage interacts with buckets by looking for buckets which match a name in your GCP account.

bucketFolder is a folder within our bucket that we'll be working with.

localFolder is where I'm keeping a bunch of local files to test uploading and downloading to GCP.

With that done, we can start our script by importing these values:"""Programatically interact with a Google Cloud Storage bucket.

"""from google.

cloud import storagefrom config import bucketName, localFolder, bucketFolder.

Managing Files in a GCP BucketBefore we do anything, we need to create an object representing our bucket.

I’m creating a global variable named bucket.

This is created by calling the get_bucket() method on our storage client and passing the name of our bucket:"""Programatically interact with a Google Cloud Storage bucket.

"""from google.

cloud import storagefrom config import bucketName, localFolder, bucketFolderstorage_client = storage.

Client()bucket = storage_client.

get_bucket(bucketName).

To demonstrate how to interact with Google Cloud Storage, we’re going to create 5 different functions to handle common tasks: uploading, downloading, listing, deleting, and renaming files.

Upload FilesOur first function will look at a local folder on our machine and upload the contents of that folder:from os import listdirfrom os.

path import isfile, join.

def upload_files(bucketName): """Upload files to GCP bucket.

""" files = [f for f in listdir(localFolder) if isfile(join(localFolder, f))] for file in files: localFile = localFolder + file blob = bucket.

blob(bucketFolder + file) blob.

upload_from_filename(localFile) return f'Uploaded {files} to "{bucketName}" bucket.

'The first thing we do is fetch all the files we have living in our local folder using listdir().

We verify that each item we fetch is a file (not a folder) by using isfile().

We then loop through each file in our array of files.

We set the desired destination of each file using bucket.

blob(), which accepts the desired file path where our file will live once uploaded to GCP.

We then upload the file with blob.

upload_from_filename(localFile):Uploaded ['sample_csv.

csv', 'sample_text.

txt', 'peas.

jpg', 'sample_image.

jpg'] to "hackers-data" bucket.

Listing FilesKnowing which files exist in our bucket is obviously important:def list_files(bucketName): """List all files in GCP bucket.

""" files = bucket.

list_blobs(prefix=bucketFolder) fileList = [file.

name for file in files if '.

' in file.

name] return fileListlist_blobs() gets us a list of files in our bucket.

By default this will return all files; we can restrict the files we want to list to those in a bucket by specifying the prefix attribute.

['storage-tutorial/sample_csv.

csv', 'storage-tutorial/sample_image.

jpg', 'storage-tutorial/sample_text.

txt', 'storage-tutorial/test.

csv']Looks like test.

csv lives in our bucket, but not in our local folder!Downloading FilesA feature of the chatbot I built was to fetch a randomized meme per meme keyword.

Let’s see how’d we’d accomplish this:from random import randint.

def download_random_file(bucketName, bucketFolder, localFolder): """Download random file from GCP bucket.

""" fileList = list_files(bucketName) rand = randint(0, len(fileList) – 1) blob = bucket.

blob(fileList[rand]) fileName = blob.

name.

split('/')[-1] blob.

download_to_filename(localFolder + fileName) return f'{fileName} downloaded from bucket.

'We leverage the list_files() function we already created to get a list of items in our bucket.

We then select a random item by generating a random index using randint.

It’s important to note here that .

blob() returns a "blob" object as opposed to a string (inspecting our blob with type() results in <class 'google.

cloud.

storage.

blob.

Blob'>).

This is why we see blob.

name come into play when setting our blob's filename.

Finally, we download our target file with download_to_filename().

Deleting FilesDeleting a file is as simple as .

delete_blob:def delete_file(bucketName, bucketFolder, fileName): """Delete file from GCP bucket.

""" bucket.

delete_blob(bucketFolder + fileName) return f'{fileName} deleted from bucket.

'Renaming FilesTo rename a file, we pass a blob object to rename_blob() and set the new name via the new_name attribute:def rename_file(bucketName, bucketFolder, fileName, newFileName): """Rename file in GCP bucket.

""" blob = bucket.

blob(bucketFolder + fileName) bucket.

rename_blob(blob, new_name=newFileName) return f'{fileName} renamed to {newFileName}.

'Managing BucketsWe can also use google-cloud-storage to interact with entire buckets:create_bucket('my_bucket_name') creates a new bucket with the given name.

bucket.

delete() deletes an existing bucket.

There are also ways to programmatically do things like access details about a bucket, or delete all the objects inside a bucket.

Unfortunately, these actions are only supported by the REST API.

I don’t find these actions particularly useful anyway, so whatever.

The source code for this tutorial can be found here.

That’s all, folks!Originally published at https://hackersandslackers.

com on June 18, 2019.

.

. More details

Leave a Reply