One of the more interesting sessions I was able to attend was “Dealing with Noise in Science / How should scientific articles be measured?” lead by Pedro Beltrao and Peter Binfield of PLoS ONE.  The session discussed how impact in science is currently measured and explored what some better alternatives might be (right up our alley!).

Photo by Easternblot

For those who couldn’t make it, there was plenty live blogging coverage in the BioBarCamp room on FriendFeed.  I hope to see more unconferences for scientists pop up in the future.  I might just have to organize one here in Southern California.

The first [barrier] is the lack of a crowd – not only is the total number of scientists in any one field pretty low in terms of internet numbers, but it’s even lower in reality with specialization. Before you disagree, riddle me this. Would you fund a company with a total potential customer base of 60,000? That’s neuroscience, all of it, globally, and it’s a generous number.

That’s just a small number for Social Web. There’s a lot more college students for facebook than there are neuroscientists for neurobook. And the scary thing is that leading authorities have recommended the best way to deal with the life sciences PhD glut is to lower the number of life scientists!

While he may be right, I am not yet convinced that the current size of the crowd is a limiting factor. To write a Wikipedia, you need a crowd of millions, since for any one topic most people are non-experts. However, if the focus is solving problems in neuroscience, a crowd of 60,000 neuroscientists may be just enough.

Second problem is that scientific communication is a different beast than normal human communication. Scientists talk to their friends, but when talking to people they don’t know, it’s much more formal. They use communication to spec theories and to claim ground as theirs.

Zoom in. The people in one subdiscipline (let’s say brain function) probably have a lot in common with another (brain pathology) – same genes, proteins, pathways, areas of the brain. But they don’t do a lot of talking. They’re outside each other’s social micro-circles. They don’t knock back pina coladas together at SfN, look at the same poster sessions, attend the same lunchtime roundtables. They sure don’t read the same papers. Who has time to do that? And there’s a good chance that a scientist would draw a tighter boundary around her communications outside that micro-circle – you might engage in wild-ass guesses with your friends, but if you did that in front of a stranger scientist, you could look stupid. Worse, you could be right and get scooped.

[...]

Put simply, if a scientist isn’t going to say it to her colleagues at a conference, she probably isn’t going to blog it.

Although subdivisions certainly exists on paper, in practice, neuroscience is increasingly interdisciplinary. For instance, in my lab of about a dozen, we focus on insect olfaction, and have people working on genetics, behavior, electrophysiology, imaging, and computational modeling. So in addition to staying current in the insect olfaction literature, individuals in the lab need to stay connected to researchers in other disciplines who use similar tools or perform similar analyses. I imagine if you linked all these people together you would end up with a crowd that was larger and more diverse than one may initially expect.

I agree that breaking away from the current model of scientific communication is not a trivial task. It may take some time before scientists are comfortable will making less formal scientific discourse public. However I do think there is plenty of valuable and insightful discourse that takes place at conferences, lectures, and in journal clubs, and we are only now starting to see some of that information appear online in places like PLoS ONE, Nature Precedings, and science blogs.

Third problem is that there are no rewards for participating in these new forms of communication. Thus the title of the post. Has anyone gotten tenure for a well-linked blog in the life sciences? I’ve asked at every university and I’ve yet to hear of even one place where there’s an entry on the tenure review form. And again, if you blogged a novel theory and someone else won the Nobel for proving it to be true, well..that’d be bad. There’s no way to reward you, even if you’d timestamped it on your blog.

There is however an extended space for your citations on every last one of those review forms. Because citations aren’t like blog posts, where you lay out a wild theory (see for reference: this very post) for the world. Citations are a place in science where you have demonstrated something to be true (or at least an approximation of truth). They’re a formal style of communication that says, not only is this a fact of nature, but I – ME - am the one who proved it to be so.

This isn’t the kind of communication one blogs, because there’s no reward for it. That’s a fundamental disconnect that isn’t yet addressed.

The current reward and tenure system is certainly problematic, especially for young non-tenured researchers who may be more willing to adopt the Web as a new form of communication. While it seems certain that the Web will drive a culture shift science, I wouldn’t expect the shift to start with the reward and tenure system. Expect it to start like all things on the Web, from the bottom up.

So far, the science 2.0 revolution has been off to a slow start, but it may just be that we’re still missing that key disruptive technology. More than likely, that technology is being developed right now. It can’t get here soon enough, because when it does, that’s when the fun starts.

The Library Journal Academic Newswire provided a brief summary of the outcome and the three year history of the proposed NIH legislation (Thanks to Peter Suber). Excerpt:

Despite heavy lobbying from publishers against the public access provision, as well as White House opposition and the threat of two last-second amendments to gut it, the legislative battle culminated yesterday with overwhelming approval of the Labor, Health and Human Services appropriations bill (75-19). If enacted with the NIH language fully intact, the law would require NIH researchers to deposit their papers in the NIH’s PubMed Central database to be publicly available within a year after publication.

The legislative process, however, is far from over. The bill must now be reconciled with the House Appropriations Bill, which contains a similar public access provision. Negotiators from the House and Senate are expected to meet this fall. The final, consolidated bill will then have to pass the House and the Senate before being delivered to the President, where it is expected to be vetoed. Although the public access provision enjoys broad support, and the LHHS appropriations bill passed with hefty margins, the House bill passed with 279 votes, 11 short of the number needed to override a presidential veto.

Nevertheless, SPARC executive director Heather Joseph said even with many hurdles remaining, passage by Congress was “a milestone.”

Indeed, getting a public access policy at NIH through Congress has been a three-year odyssey for SPARC, an early and integral champion of the policy. The initially proposed NIH policy was introduced in 2004 as a mandatory policy with a six-month embargo. In a bitter setback, it was gutted at the eleventh hour, and implemented in 2005 as a voluntary measure. Lawmakers and advocates, however, vowed they would monitor the policy’s effectiveness. By 2006, the policy was failing so spectacularly (less than five percent of individual investigators deposited papers) that it no doubt helped marshal the heavy bipartisan support for the revised NIH policy passed on Tuesday.

[...]

Publishers, meanwhile, remain opposed to the NIH policy, contending it could undermine scholarly publishing, and they will likely have more opportunities to fight the public access mandate, either during reconciliation and/or if the LHHS appropriations bill is vetoed. They have also laid the groundwork for a legal challenge to the suit centered on copyright. While copyright experts doubt that claim could ultimately prevail, it could nevertheless delay implementation, giving publishers another chance to organize opposition in 2009.

Journalfire provides a centralized location for you to share, discuss, and evaluate published journal articles. You, the scientists, are put in charge of determining what studies are significant and noteworthy.

Use Journalfire to create groups to share articles and ideas with scientists in your lab or from around the globe. Currently Journalfire links to every article in the PubMed database, with access to more databases on the horizon. When discussing articles, you decide whether the discussion is public or private, and whether or not to use your name or remain anonymous. Journalfire creates a permanent open access record of each discussion and links it directly to a specific article or group. See what articles your colleagues are discussing in their groups, and see what comments are being made about the papers that interest you most, including your own.

Journalfire provides a new way to assess scientific merit. The value of a journal article is often first assessed by the name of the journal in which it appears. Unfortunately, this is typically determined by fewer than five people. Journalfire offers a more democratic approach. By enabling you to evaluate published articles, the combined opinion of the scientific community can be used as a measure of scientific merit.

Journalfire search results take into account several criteria, including the amount of discussion an article has generated and the rating an article has received. Thus Journalfire enables you to quickly find the papers that other scientists are talking about, or papers that may have otherwise slipped through the cracks.

Journalfire was created by a group of graduate students who were frustrated with the current system of scientific discourse and publication. We believe Journalfire is a step in the right direction towards a more open and democratic scientific community. If you’d like to join us, visit journalfire.org.