Online papers in philosophy feed

Long ago, I wrote a little script to automatize Brian Weatherson's (at the time) Online Papers in Philosophy blog. The script crawls the home pages of various philosophers and extracts author, title and abstract from every paper posted there. It then visits the pages again every other day or so to check for updates. This way, I'm currently tracking about 15000 papers from about 2000 pages.

Since the real OPP blog, maintained by Jonathan Ichikawa for the last few years, has caught a virus and is therefore not doing well right now, I've decided to dust off my script and make it available to the public. Then along came David Chalmers, who talked me into not making it public after all, but rather merging it into something even bigger that will hopefully go live very soonish. In the meantime, here is at least an RSS feed of my script, with daily updates of new papers it finds: OPP RSS.


# on 23 July 2008, 07:37

Wow! Thank you very much. I think that this is one of my favourite feeds this far (and of course the best). For some reason I haven't been a big friend of feeds but this seems to change everything. ;)

It's just so great to open your browser and notice how many new papers there are. Heh.. the only thing I really need is an effective cognitive filter.

# on 23 July 2008, 18:20

This is great. Thanks very, very much, Wo (and Dave)!

# on 23 July 2008, 18:24

Thank you VERY much. This is a wonderful resource. I am curious about the 'something even bigger' and will look forward to whatever that might be.

# on 24 July 2008, 15:03

Hi Wo,

This is great... fab stuff. Thanks for getting it running!

One quirk: when I set the rss as a live bookmark in firefox, it gives me a list of the dates of updates. That's great, but when I click through, it just says "Hi there". Not sure why that it happening.

It's not too much of a stress, though, since the feed itself has the relevant data, and it interacts nicely with bloglines.

# on 24 July 2008, 16:22

Thanks, yes, there's nowhere to click through to, as the page itself doesn't officially exist yet. I guess I could make each paper its own feed entry, a la But I think I'll just leave it like that for the meantime.

# on 24 July 2008, 18:05

Thank you thank you thank you!

# on 24 July 2008, 18:11

Hey, I'm really pleased that this works so well (using NetNewsWire on OS X). Thanks for making it available since OPP tanked.

# on 29 July 2008, 01:25

Thanks a lot for the great effort.

The use of a simple, minimalist rss feed like this is vastly different from "something .. bigger that will hopefully go live very soonish".

# on 30 July 2008, 20:59

Thank you, thank you, thank you! When the something bigger thing comes along, let me know so that I can advertise over at PEA Soup.

# on 09 August 2008, 12:15

A grateful reader thanks you for the rescue of OPP. The RSS feed in Yahoo now only delivering a hacker's message, but other RSS feeds are OK.

# on 18 August 2008, 12:08

Thanks! I was wondering if you could share the source of the script - I'd like to setup something like this for Polish philosophical pages (which are only of Polish interest), and that would probably require some tweaks in the script.

# on 18 August 2008, 12:46

Hey Marcin, the script is unfortunately rather hard to set up, and you'd definitely need a dedicated Linux server. (Among other things, I run Mozilla on Xvfb to parse HTML pages.) It is also still under heavy development, so I'd rather not spread the code at this stage.

That said, I've started classifying papers by topics, and classifying by language could be a natural extension of this. In the system that will soon go live, it will probably also be possible to get a feed that tracks exactly the philosophers you want to track. So hopefully there will not be much of a need to set up several different versions.

# on 18 August 2008, 21:55

Hm, if that requires a dedicated server, it's not an option for me. BTW, I thought of using some light-weight webscraping (no Mozilla involved, only lynx - that is definitely easier). Probably using a text browser for the visually impaired (it must deliver the text for the speech synth) could be an option as well.

But if you want to setup it with categories, I could probably be of some help with adding non-English philosophers (German, Polish and Spanish, as I can speak these languages to some extent).

# on 21 August 2008, 10:05

Thanks for the offer -- I'll keep it in mind. The problem with lynx etc. is to extract author, title and abstract from HTML papers. You mostly don't know anymore what was in large font and bold and centered, etc.

# on 09 December 2009, 16:27

I love this feed, but I think there's a minor bug. On November 10, it found "On the structure of explanatory unification: The case of geographical economics" and attributed it to a philosopher named P.O. Box instead of its true authors, Uskali Mäki and Caterina Marchionni.

# on 09 December 2009, 18:01

Thanks Ben. I also noticed the P.O. Box, but too late. Most RSS readers only look for new posts, so even if I fix the mistake here on the server, everyone would still see it in their reader. Not much I can do, except trying to prevent such mistakes for the future.

I'm currently planning to try out a few new extraction algorithms that will hopefully make fewer mistakes. I will also open source the code sometime in 2010, when that is done.

Add a comment

Please leave these fields blank (spam trap):

No HTML please.
You can edit this comment until 30 minutes after posting.