The 512KB Club is a collection of performance-focused web pages from across the Internet

The 512KB Club site has a leaderboard of sites from smallest to largest, all below 512KB in size. I came to wonder how the size was determined, and therefore updated. Reading through how to add a site showed the process was to manually run the site through CloudFlare Radar

I knew this meant the sites weren't getting updated... I checked the last-updated timestamps for some sites, many hadn't been updated in years.

I thought this could be automated!

Puppeteer

Puppeteer is a JS framework which runs a headless browser (Chrome or Firefox) and provides a JS API to interface with it. It's event oriented, you add listeners for events like Network.loadingFinished.

This seemed like it should be really easy. It quickly realised this wasn't the case, knowing which events to listen for and which events gave access to which information required an unhealthy amount of trial and error. Claude was some help but not a lot. It seems that the Chrome Developer Tools (CDT) events are just inconsistent. For example, I needed to listen for both Network.responseReceived and Network.loadingFinished, the former gave details on the status code and the later the request body size.

Then there were the edge cases, so many edge cases:

• OPTIONS requests don't have a body
• 404 requests only have their headers downloaded in Firefox but not Chrome
• Responses with a 204 (No Content) status code.
• Requests blocked by CSP
• Service workers
• Requests which pointed to dead domains
• Lazy-loaded elements required scrolling the page

After a while, I was able to get dev tools in the browser and my script to agree on total page weight. You can see the full script here.

The Results

I set the script running for all the URLs currently in The 512KB Club. This took a while, it could definitely benefit from parallelism. The results were in:

- Total sites: 960
- Larger: 596 (>512: 82)
- Smaller: 241
- Unchanged: 0
- Dead: 35
- Error: 88

I was pretty unsurprised that most of the sites had got bigger. The full list of changed sites can be seen in the PR:

The Future

If I get around to it, I'd like to run all the sites through Firefox and Chrome and compare. There is also the issue of the 88 sites which had an error, this isn't an error with the site but where something went wrong with my puppeteer script aka more edge cases.