Science - editorial, social, or both

It was almost two months ago that David Bradley asked the question (via LinkedIn), if science may benefit from social software? Several people responded, and especially David Crotty started a controversial discussion based on my raised points.

First, Crotty said that 'popularity' is a terrible measure of quality. Actually, I think I agree on this! Otherwise, any popular web page would be automatically one with a higher quality. I have some software engineering background and would like to take this into account. I would argue that not only the access rate of web pages, but also the cross-linking character between them, or the number of errors per page would be good metrics. Was anyone following the dispute about the Britannica-Wikipedia comparison (comment)? Again, what should we compare? Error rate in articles, access rate, or cross-linking rate? Do we not have the same problem for journal impact factors? I think we have to accept that any model (or metric) will never represent the full truth about reality, but it might give us some additional rational for making decisions. Are any molecular modellers still with me? Yes, then I have a question. Do you believe that this metric discussion about molecular docking and scoring will ever end? If you think so, what metric or benchmark data set will lead to a fair comparison? I do not think that this is possible, but I strongly believe that the metrics and benchmark data sets around will help us to improve over time. Even using wrong metrics does sometimes help improving metrics, and therefore rationality.

Second, I said that "I cannot see how any editorial process can cope with this (information overload) problem"; he asks the question "Is he (means me with that) implying that journal editors are not 'really interested' in the papers they’re reviewing?".
Not at all! The point I tried to make is that the editorial process will assign reviews to reviewers, which will in most cases do the reviews also in their spare-time. So, at this point it does not matter, if reviewers get assigned by editors or by any other social web system. I think my point of view is not really strong on this, but I believe that missing transparency might be a major source of mistakes of any process, e.g. a peer-review process (via spreadingscience). I am not saying that transparency can avoid mistakes, but a transparent process would allow democratic voting systems, which is one type of decision making on Wikipedia. Anyway, in cases where no decision can be reached by user votings or reviewer comments, the next step will be taken by a Wikipedia mediator or an editor. The major difference is that in one case the editor has to do the assignment, while in the social software model the dedicated people would come to the editor automatically. And especially in cases where no editor is in place, users would just start editing (be bold!),till one or several users get promoted as pseudo-editor.Third, David Crotty says that "you need a better editorial oversight, not less" and he cites Rob Malda (via Wired) "When you're building a system like this (means Slashdot) you're balancing the wisdom of the crowds versus the tyranny of the mob. Sometimes a crowd is really smart, but some things don't work so well by committee. Crowds work when you have a tightly knit group of people with similar interests ..."
Well, again I agree that you need rather more information than less. Though, I would like to challenge this a little. Where is the editor getting his information from? I would assume that he will use at least some social web services or (social) cross-links in journal publications for finding other experts.

I was never a friend of black and white views. I would conclude that

• any editorial process without some social software will run into an information overload problem, and you should not do all of this alone, but share workload.
  
• any social software process without editors runs into an information overload problem, and how can you discriminate information from noise? You need experts, editors, and some people call this ontology knowledge, or the semantic web.
• a combination of both, editorial expert knowledge in combination with social software is promising for creating massive knowledge networks, e.g. WikiProteins. Finally, any technological process will fail, if the people behind it, are not dedicated enough, run into workload problems, or are hold back by unclear intellectual property situations. So, whatever you do, do it in a team and make it a transparent process. If required, put some license, copyright, intellectual property, or whatever restrictions on it, so make steps towards users and do not wait till users will adapt to your user scenario. In all cases, avoid destroying information and allow a cross-linking between information.
  As already said on ChemSpider, "get the hell organized..." and talk to each other.

Finally, David Crotty and David Bradley, thanks for discussing this in an open-minded setup and via a social software system. I am looking forward to more news, questions, answers and controversial discussions ... at the end it is all about learning and helping people, right?