Many individuals consider Cloud Run simply as a method of internet hosting web sites. Cloud Run is nice at that, however there’s a lot extra you are able to do with it. Right here we’ll discover how you need to use Cloud Run and BigQuery collectively to create a list administration system. I am utilizing a subset of the Iowa Liquor Management Board knowledge set to create a smaller stock file for my fictional retailer. 

Bulk import of latest stock 

In my stock administration state of affairs we get a csv file dropped into Cloud Storage to bulk load new stock. You possibly can think about that we’re migrating to this new system from an previous stock administration system that may export a csv file. 

BigQuery can import CSV recordsdata from Cloud Storage instantly, and in the event you need not do any knowledge transformations I like to recommend you utilize the in-built CSV loading performance. For my use case I solely want a few the columns from the CSV and I wish to do some extra knowledge transformations so I will be writing some code. 

To make deploying the code simple I am utilizing the Python Functions Framework as a shell round my script. To entry Cloud Storage and BiqQuery I used the Python Consumer Libraries for every of these merchandise and I am following the samples within the pattern explorer. I solely bumped into two points when penning this up. 

First, I do not wish to use the native file system if doable to make sure my code is actually serverless, so I’ve to work with the contents of the CSV file completely in reminiscence. And second, the pattern for downloading a file from Cloud Storage leaves it as a blob as an alternative of one thing CSV can parse. A couple of easy instructions to translate the blob to bytes and the bytes to a StringIO object solves that drawback.



Leave a Reply

Your email address will not be published. Required fields are marked *