Last week I wrote a thing I’m calling SteamNews. The basic idea is that I want to follow the news of games I like, particularly Early Access games that are updated frequently. SteamNews does this, providing RSS feeds for every game, updated every hour.

First and foremost, I’m shocked Steam don’t make this easier. Steam makes it really hard to do this, actually. They offer RSS feeds, but they require like… two hundred fucking clicks to get to. You can’t guess the URLs either because they’re inconsistent - either the Steam app ID, or some short slug that you can’t guess.

So what makes SteamNews worth using? Well, it puts all the games on Steam onto a single searchable page, so it’s really easy to just find the news that you want and get on with life.

More interestingly, the code is available on Github. It’s written using asyncio, which I’ve written about before. Basically, it updates newsfeeds every hour via cron - the cron job runs every 15 minutes to cover errors, though. All the feeds are regenerated and written to disk as static files. It performs 2000 requests in ~10 seconds, meaning it can crawl the Steam catalogue very quickly. It doesn’t seem like much, but consider that that’s the client, not the server - the server is most certainly ratelimiting me. It also downloads the entire gamelist and filters out obvious garbage, and checks with Steam via unofficial APIs to clarify games that I couldn’t from the name alone.

The code is not of a particularly high quality - I’ve hacked together some really shitty HTTP parsing, because asyncio doesn’t have anything that high level, and http from the standard library was too hard to integrate. I would like it if this sort of thing was easier to hook into asyncio, but I don’t really see it getting better, which is a shame. I imagine that upon the release of 3.4, there are going to be a billion frameworks written for it, all in various states of shitty. This is already happening to some degree.

I’m contemplating pulling all the HTTP parsing out of python-icap and just using that from now on - it’s very, very robust, although biased at the moment towards chunked transfer-encoding, but that wouldn’t be difficult to change.

Anyway, I digress.

What writing this has taught me though, is that I’m a big fan of statically generated sites because they’re just simple. This was a case where I wrestled with the idea of implementing it in Flask, adding a database, and all sorts of things. I quickly realised that doing that would be way more work than I wanted, so I spat this out in a few hours and now it’s finished. No need to worry about security, keeping the service running, or any of that. It just works.