Fetching RSS Feeds Respectfully With curl
I am a member of the Church of RSS, though I’m quite a late convert (I missed the whole Google Reader thing). I use my own self-written Golang tools to parse the RSS/Atom feed files and send myself the entries in an email.
I haven’t felt the need to do anything too fancy on the downloading the RSS feeds part, and use the venerable curl in a bash script.
I’ve tightened up this curl feed downloader in response to various blog posts by Rachel By The Bay and recent posts by CKS in the same vein.
Sending Conditional Requests to hundreds of web servers in curl is actually a little fiddly, so I thought I’d document how I go about it.
I have about 500 RSS feed URLs that I download, and I want to get as many 304 Not Modified responses as possible - that’s the “respectfully” part.
A naive way to use curl to download multiple URLs would be to call curl in a loop, once per URL. This is inefficient for you and the remote servers you’re hitting, so don’t do it that way.
