Periodically scraping websites, saving to Google Sheets, and firing Slack notifications

I hope this serves as a launch pad for users wanting to automate tedious tasks throughout their day to free up time for more interesting things!OutlineSimple scraperUsing Google Sheets as a DBSending Slack notificationsServerless cron using AWS Lambda and CloudWatch EventsOne task I would liked to have automated when I was graduating from college was looking up prices for an apartment I was interested in..While the prices weren’t super volatile day to day, they would suddenly drop when new rooms opened up and resulted in a race of who could send in a down payment to the leasing office first..If I had the current tools at my disposal, I could have saved a couple hundred dollars a month on rent, or at least freed myself from the headache of having to check the site every day.First and foremost, all code examples are located in the github repo below:yoheioka/mighty-scraperTemplate for creating a scraper that saves to Google Sheets, fires Slack notifications, and is scheduled using AWS…github.comMake sure to reference this while going through this tutorial..Let’s begin!1..Simple scraperThe simple python script uses Beautiful Soup to scrape this listing site, and prints out the minimum price..You can test out the script by running the following command (make sure to set up a virtualenv and install the necessary libraries):python apartment_scraper.py2..Using Google Sheets as a DBWhile using Google Sheets may not be the most efficient or scalable solution for storing data, it’s an easy way to share data with non-tech people..In my example, I would have shared the apartment data with my girlfriend at the time (now my wife) and gotten her feedback as well.It’s also free to use!Let’s jump right in.A..Create a new Google APIs projectHead over to https://console.developers.google.com/apis/dashboard and create a new project.B..Enable Google Sheets API for the newly created projectC..Create a service accountFrom the left menu, select IAM & admin and go to Service accounts.You can select the Owner role for now, but may want to restrict this if you are dealing with more sensitive data..The third step, Grant users access to this service account (optional), can be skipped for this tutorial.D.. More details

Leave a Reply