I’ve been waiting a long time to take a pottery class. In a city of almost 1 million people, there are surprisingly few pottery classes and there are even fewer pottery classes that happen outside working hours. One of the classes I knew about had only 12 available seats that instantly sold out when registration opened. In fact, I missed the registration for this class several times already.

I learned that the demand was so great for this pottery class that the website maintainers had to pick a random time to open registration to prevent the servers from crashing. Since I couldn’t spend all day refreshing the pottery website to see if registration was available, I decided to write a program to help sign up for this class.

A script to alert when a website changes

I made sure not to miss the registration this time by writing a script that monitors the registration website and sends an alert to my phone when it changes (when registration opens). Their are tons of online services that do this, but they cost money if you want to check a site more often than once every 15 minutes. However, writing this code is incredibly easy and satisfying.

The script I wrote is in Python which does the image processing and the alerting. The website snapshots are taken with a Node script (using puppeteer) which is run from the Python script.

I learned some subtle things about website tracking while doing this - namely website scraping is easier if you can block ads and that SMTP is the best free notification service. More on that below, but if you just want the code, the script and the instructions for using it are on my Github: https://github.com/schollz/websitechanges.

Block the ads before taking a snapshot

The screenshot of the website is downloaded by using puppeteer, which is very easy to do (its one of the examples!).

However, one subtlety here is that I need to compare two screenshots in time for changes. Since ads can change every time you reload a page, I realized it is important to remove ads to get a reproducible view of the website. This is really easy to do with puppeteer. First download a hosts file and then load it into the Node script:

 1const fs = require('fs');
 2
 3hosts = {};
 4//now we read the host file
 5var hostFile = fs.readFileSync('hosts', 'utf8').split('\n');
 6var hosts = {};
 7for (var i = 0; i < hostFile.length; i++) {
 8    if (hostFile[i].charAt(0) == "#") {
 9        continue
10    }
11    var frags = hostFile[i].split(' ');
12    if (frags.length > 1 && frags[0] === '0.0.0.0') {
13        hosts[frags[1].trim()] = true;
14    }
15}

And then in puppeteer you can block all the requests to everythin in the HOSTS file:

 1/* ... puppeteer setup omitted */
 2await page.setRequestInterception(true)
 3page.on('request', request => {
 4    var domain = null;
 5    var frags = request.url().split('/');
 6    if (frags.length > 2) {
 7        domain = frags[2];
 8    }
 9    // just abort if found
10    if (hosts[domain] === true) {
11        request.abort();
12    } else {
13        request.continue();
14    }
15});

This way, all the ads are removed and you just get a blank space or no space where they were.

SMTP is the easiest, cheapest way to send notifications

The pottery website registration could occur at anytime in the middle of the night (it ended up being at 4:43 AM). I needed a way that the website change could notify me, namely by sending a text message. I can use my phone to play a loud sound when the message arrives. But how to send a text message?

It turns out to be very easy! To send a notification to your phone, you simply send an email! Your phone provider usually supplies an email address for your phone. Here’s the ones I know of:

• Verizon: PHONENUMBER@vtext.com
• Sprint: PHONENUMBER@messaging.sprintpcs.com

But, then, how do you send a email from a program? You can use email API service. However some of these, like mailgun, entice you with an offer of a free tier only to later remove the free-tier. But the alternative is easy, fast, and free.

The alternative is to use SMTP which is provided with almost any free email service. For example, you can use a new Gmail account with a random username and password. To enable SMTP in Gmail:

0. If you are using a remote server, check this
1. Go to the “Settings”, e.g. click on the “Gears” icon and select “Settings”.
2. Click on “Forwarding and POP/IMAP”.
3. Enable “IMAP Access” and/or “POP Download”
4. Goto https://myaccount.google.com/lesssecureapps and turn “Allow less secure apps” to “ON”.

That’s it! Now you can send emails in Python using a function like this:

 1import os
 2import smtplib
 3from email.mime.text import MIMEText
 4from email.mime.image import MIMEImage
 5from email.mime.multipart import MIMEMultipart
 6
 7def send_email(smtpemail, smtppass, to, subject, body, attachment):
 8    img_data = open(attachment, "rb").read()
 9    msg = MIMEMultipart()
10    msg["Subject"] = subject
11    msg["From"] = smtpemail
12    msg["To"] = to
13
14    text = MIMEText(body)
15    msg.attach(text)
16    image = MIMEImage(img_data, name=os.path.basename(attachment))
17    msg.attach(image)
18
19    s = smtplib.SMTP("smtp.gmail.com", "587")
20    s.ehlo()
21    s.starttls()
22    s.ehlo()
23    s.login(smtpemail, smtppass)
24    s.sendmail(msg["From"], msg["To"], msg.as_string())
25    s.quit()

Now you can have your program send a notification to your phone, with an image of the changes.

Back to Pottery

I wrote this script the night before the pottery class registration was set to take place. The exact time the registration was set to open was random. But then, at 4:43 am, I got a notification:

I checked the website and saw that indeed the registration had opened and I got myself registered!

Interestingly even though I thought I’d be the first there were already two other people registered by the time I was done registering! That means I’m not the first to do this type of thing for this particular class. But my code is open-source at https://github.com/schollz/websitechanges so I hope everyone else will have a chance to try it too.

There is a caveat about using SMTP with Gmail. Gmail will tend to block SMTP access if you create the account on one computer and then use it on a remote server (like Digital Ocean).

To get around this, make sure to create the account on the remote server, if that’s where you plan to use it.

To do that, SSH into the remote server using

ssh -D 8123 -C -N user@remoteserver

The -D parameter will bind a SOCKS port to 8123. Now goto Firefox settings and change the SOCKS port to 8123. Now Firefox will use your remote server and you can setup SMTP remotely. Then change it back when you’re done!