Creating an Automatic Newsletter With Azure Functions
Short story on I how to use Azure Functions to scrape a website daily and send an automatic newsletter
Photo by Symeon Ekizoglou from Pexels
I want to begin by thanking dev.to for this fantastic platform, and Burke Holland for sharing this series. I probably wouldn't have been able to finish this project if I hadn't found this.
In Iceland, there's this football (soccer) news site called fotbolti.net. It's the most popular football news site in Iceland. Whenever a player transfer goes through, or a manager gets hired or sacked, fotbolti.net always has the word 'Staðfest' ('Confirmed') in their title. I believe they have 100% accuracy. Meaning, they have never published a news article with 'Staðfest' in their title that hasn't been legitimate.
So, since the transfer market is wide open in Europe and a lot going on, I had an idea of creating a newsletter that would only send me the news where 'Staðfest' is in the title. I usually visit fotbolti.net a few times a day and therefore usually don't miss this news. However, when I go on holiday, get sick or have a hectic day, I don't visit it as much (sometimes not at all) so that's when the newsletter could come in handy. Plus, I just thought it might be a fun project to try out serverless. :)
Proof of concept
I decided to create a small Python program for a proof of concept.
To complete the proof of concept, it needed to fulfil the following requirements:
- Get the RSS feed
- Get only news from the last day (24 hours)
- Get only news that has 'Staðfest' in the title
- Print out the title(s)
That all went well, and it didn't take me long to complete this part. After I finished the proof of concept, I thought to myself, "Now what?". I had no idea how I would gather email addresses, fetch the addresses in some sort of database to send the actual newsletter, or how I'd make sure the program ran at the same time every day if there were any news to report.
I was pretty sure from the beginning that I wanted to use Azure Functions for this, mainly because I wanted to learn more about it. I saw that it was possible the create a function using Python, and since I did my proof of concept program in Python, I decided to try it.
Unfortunately, I was never able to get that working correctly. That's when I started 'googling' quite heavily. I was sure that someone out there had done some Azure Functions and shared it on the internet. It didn't need to be exactly like my idea, just some sort of Azure Function, which I could base my idea on.
Dev.to to the rescue
That's when I came across the 'Azure Tips and Tricks' article series by Michael Crump that Burke Holland shared on dev.to. I couldn't believe my luck. This series had info on everything I wanted to do. At first, it seemed to me that the only change I would have to make was to change my Python code to C# code and use that.
After going through all the steps in the guide, I had some syntax errors in my code. I didn't understand why considering I had done everything the same. So, I tried different namespaces, approaches, libraries, but nothing seemed to work. Finally on a Sunday afternoon, after having spent all of my free time since Thursday on this (and no time with my girlfriend), I somehow managed to puzzle this together. I was so excited and happy. (For some reason, my girlfriend did not share my level of enthusiasm). But it fucking worked.
After cleaning everything up and making it pretty, I asked the site's permission to use the RSS and keep my project alive. They permitted me to use it, and for that, I'm thankful.
Thank you for reading, here is the final product on GitHub.
And if you understand Icelandic and want to subscribe to the newsletter, here’s the link to that.
To everyone reading this, please keep sharing your tips and tricks with the world. You never know when you could be helping someone.
Thank you for reading.