Evaluation heuristics

Gavin Baker points out a CHE article talking about the vexed question of evaluating “online scholarship” (whatever that is) for tenure and promotion purposes.

This came up at Project Bamboo, and also during a dinner out at data-curation bootcamp. I don’t have an answer for it; that kind of thing happens way above my pay grade. I do have a possibly-useful observation, however, videlicet and to wit: this is a heuristics problem.

If tenure and promotion committees wanted to, they could evaluate their colleagues by reading their actual work, print or online or both, and coming to conclusions about its worth and effectiveness. They do not do this. They do not want to do this. They would walk miles and miles over broken glass on bleeding bare feet to avoid doing this.

They give lots of reasons they don’t do it, ranging from not being experts in their colleagues’ areas of expertise, to not having time to read all that, to the tenure-and-promotion process being full enough of angst and drama as it is. Be that as it may, they don’t do it. So what do they do? They rely on heuristics instead.

Peer review. Impact factors. Citation counts. Quantity of output. Supporting letters from field experts. Publication-venue reputation for quality (or “branding” if you must). None of these evaluation methods was handed down from heaven; they’re just what’s evolved out of the system.

To expand the system to cover online scholarship, especially online scholarship that doesn’t have easy print analogues, we need to come up with judgment heuristics for it. It’s that simple.

I think the society-seal-of-approval idea in the CHE article is a good one, but I’m also selfish: such vetted collections would be nifty for librarians to have as well. For the sake of academic politics, I presume the evaluation process would be private, the list of sites under evaluation kept under wraps, and no list given of sites that didn’t make the grade for whatever reason. But again, that sort of thing happens above my pay grade.

Judgment heuristics for online scholarship turned out to be a major request made of Project Bamboo by the workshop I went to. I think PB can and should tackle that problem… but they can’t be the only ones who do, not least because the social sciences and hard sciences need heuristics just as badly, and their heuristics will be seriously different.

But it’s a tractable problem, en masse. It’s when we ask every single department to come up with its own heuristics that everything breaks down.