There are a range of conversations currently happening around ideas of community and quality. I feel these conversations are also about POV point of view.

Stephen Downes posted a blog post and note to facebook about starting an online journal.
The quotes in this post are from that post from Downes.

".. I think that when people talk about 'peer reviewed publications' they have a point, and that point is, that a piece of writing is not merely popular, but also, respected and recognized by a particular academic community.
We need such mechanisms because there is too much to read, too much even in narrowly defined disciplines. And there is no particular mechanism for identifying that which is important within a particular discipline. The popularity-based systems, like Slashdot and Digg, cater to certain communities, sure, but tend, eventually, to what we might call a scholarship of the middle - no particular discipline, no particular level of quality, no particular virtue."

Slippy from Utah State has been working on something similar.
Artichoke is talking about quality in student blog posts and comments.
The folks at educationau are working on similar thoughts although they dont describe it as a journal, the thinking is similar but the goal is more to make the edu sector community conversations more visible to the communities which make them rather than making a specific publication.
Peter Shanks has been doing related work on his unwrapping subjects and Training O2

Outside of the education sector similar conversations happen in other situations.

Conversations around trying to control for quality often happen where a project has developed sufficient 'value gravity' for people to want to be disruptive in the space and for others to want to protect it or a certain way of approaching it.

When a project is small/new people feel they have the freedom to choose to engage in the systems relevant to furthering the goals of the project and also to contribute their individual ity in a constructive fashion.

As projects become bigger the project attracts interest from more people who wish to further the purpose in different ways and also from people who disagree with its choices or purposes. Often the projects face choices around controlling who can participate or who is more credible. Any of the free negotiation spaces face this kind of challenge.

Campaigns wikia was one where people were talking about presenting an issue, posting the for and against of those issues and negotiating possible forward ideas. Basically an opportunity for providing qualitative politics rather than quantitive political input. Other folks were interested in posting the campaign details of individual standing politicians. The representation of the existing politicians ended up being the function which  took over the function of the site in terms of realestate on the front page and overall structure. This was partly because people could agree on what to put on those pages, and partly because there are people paid to put that kind of information into public space.

From the 'knowing what to put there' aspect, the issues pages struggled.
Issues like gay marriage or digital rights attracted people who felt passionately about their perspectives and who did not have a stronger commitment to making something new out of politics. Negotiation is a skill and a choice that the wider purpose is as important as my pov. I found this hard too. And I think on both fronts Campaigns wikia struggled with the state of play that our community reached in terms of expecting to be able to participate constructively in negotiated projects with binary non-negotiable perspectives. People understood the freedom to participate, but did not value the constructive role of a diversity of perspectives  sufficiently to make the pages something which people could generate community energy and discourse around. Perhaps politics in abstract terms is the hardest nut to crack and politics in specific applied terms might be easier to unpick but basically I think we have 'work to do' here.

Developing our skills in constructive pov are the skills which I feel we need to develop most in order to let go of fences as a means of ensuring quality. I think these skills are the skills we need to be able to employ if we want to get real data back into the decisions made on our behalf. In politics but also in education and in valuable projects generally.

ISO standards for example. The ooxml process has struggled to keep the focus on negotiating the fit for purpose of the proposal. POV has been louder than purpose. The proposal does not resemble a standard proposal in many ways but the fast track process was seen as an opportunity to push it through. Standards processes will be important for open source and participative development. Important areas for us to develop skills in authentic negotiation of fit for purpose. This particular proposal has been I hope, a wake up call on that front.

"That is not to discount the systems whereby content is selected and reified by the masses. I am a regular reader of such lists and they are a constant source of amazement and amusement. High quality content does get selected by the crowd, but not all of it, and not reliably within a certain discipline."

The crowd are the authors. The crowd are also the audience. If a journal is a selection of specific members of the crowd, that is a choice you can make now. That is a choice we are all making now. Sometimes we choose people because their perspective will be innovative, sometimes because it is close to an important project or process, sometimes because it is a voice which takes time to listen to a broad range of inputs, sometimes because there is heart or wit or something more directly inspiring about the writing of the author. Some people write as a finished work, and some people write as a thread and comments naturally flow from it.

We could probably do with a few ways to unpack the why of the choices of our peers, but these things can be learned perhaps. Tags are generally informal and personal ways of sharing information. Specific tags are also being used to map to specific materials or events.
Sarah Hayman and Nick Lothian have been talking about the potential for flow between folksonomies and taxonomies for finding  and sorting information.

"Historically, as I mentioned, content selection for academic materials has been by means of 'peer review'. The process varies across journals, but in its most typical instantiation, proceeds as follows: a writer submits a manuscript to an editor, who reads it. The editor, at his or her discretion, sends the manuscript to a small committee of reviewers. The reviewers rate the submission for appropriateness for publication. They will often recommend changes and improvements. A final version is drafted, and it is typeset and published."

This also occurs with conference papers. These events are interesting because they are organised with the goal of attracting people interested in specific topics. Their fit for purpose is finding things which people will feel they have discovered through the event which they might not have experienced otherwise. If there is value and energy outside that event which is missed, other events are likely to occur which look at other voices. The choosing approach makes them a kind of broadcast media.

Barcamps are an interesting flip for that kind of approach. They have open invitation with people coming to both listen and contribute something and there is no keynoting or important person loading on the voices present. Foo camps were the original form and were informal meetings between a closed list of invited people. They each have different kinds of value/challenge in terms of fences and quality, signal and noise. Business model.

"But what constitutes 'being a paleontologist'? Traditionally, we have required some sort of certification. A person needs to become a PhD in paleontology. Then they need to be selected by an editor of a journal to sit on a review board. This qualifies them to review publications in paleontology."

This is a pattern which suits broadcast media too. This is a means to identify who should be heard, what is authoritative. The costs of the processes are intended to be returned to the person by their exclusive access to voice. I would suggest that there is less commitment to these processes in contexts where peers are able to have effective voice in other ways.

"And in other cases it is by choice, as no PhD programs exist in a new area of study or invention. This was the case, for example, in internet technology. It had to be built, first, before people could become experts in it, while the people who built it became experts by building it."

Yes. Open source practices are more like this. The proof of quality is evident in the feel and focus of the community and project which has been generated. Perhaps it is easier to find these communities because the practice is visible and the quality is tangible in the sense of good effective code, but I do also see the same kinds of clarity or authority in the informal education networks online. It is true that some excellent folk are not so visible online and you have to work a bit harder to hear them and that people can function in specific threads online which means that we can miss the important conversations which are quite close by.

"Members are selecting not only a submission, but also the person. This means that to a degree, the candidate's previous body of work will be assessed as well as the actual submission. The role is not of 'gatekeeping' but of recognition."

This assumes a person is always right and authoritative. It assumes that people who sometimes write stuff which is not useful will not contribute something useful.
It assumes that there is one purpose. I think the kinds of material which would be chosen in this way would be likely to be a median pick because youre looking for someone who usually writes in ways which agree with existing members. This is likely to build a 'school of thought' approach to the journal. There might be value in printing a school of thought publication.
It would be something where the people in that group are prepared to invest their time in corrrecting/editing material and publishing it for their own purposes. People looking for that line of thought might be interested in that publication, but I think education and many fields are looking for a different kind of authority now and so thejournal might not get a bulk readership in the same way that traditional journals might have.

We are looking for authority with regard to our own specific purposes and thinking. Context is important. So for me the interesting part of the process would be the method for finding people who are interested in starting a journal on issues I am interested in. I would like to collect the kinds of ideas that some people have been talking about and would be interested in seeing what else theyve been reading on a specific line of thought, if we agreed on material which contributed to that debate then we could publish a journal of that debate to date. This would not be a collection of reliable members but would be a collection of pivotal posts or thoughts on a specific line of thinking. They would produce different kinds of collections. I think there would likely be less language editing on this type of journal but that referencing to sources would still be important. For me this would be a nice way to capture some of the good things which happen in transient places like blog comments, and enable people to collaborate arround a purpose rather than a membership.

"It is possible for a journal to become too much of a clique, for the members to select only each others' papers. If so, then the people who are being left out can found their own journal. Because nominations are public, it will be easily evident which journal is the most difficult to get into because of quality, and which are the most difficult to get into because of exclusivity."

If the criteria are about finding people who usually write in ways you usually agree with then the resulting community will be looking for that kind of normalising characteristic. It will be a fence around a group of people who agree sufficently.

If the collection is organised around defining a mix of perspectives on an issue and perhaps even on crafting some common ground between them as a part of the process then this is a piece of the kind of thinking process which I think we need to develop good skills in.

Will this work? I think it will. It might not work for any particular journal - some journals may simply not attract readership because the writers admitted were not of a high quality, or because the members make poor choices, or because the subject area is simply not useful or inappropriate. It will take a certain amount of momentum to launch a journal, a momentum that can be gained only by having qualified people and quality ideas to begin with.

Swap it around to distributed publishing thinking. A journal could be published to meet the reading needs of its authors. Its success will be based on whether the participants find enough value in the process to put the effort or $ in to producing an outcome. If that publication sells more copies then the participants get a greater return on their efforts.
It is successful for different people at different stages of that process.

Every time we make a fence around which voices are valid we are losing an opportunity to develop skills in crafting better collaboration and negotiated pov. We do need ways to capture perspectives in a more coherent and durable way which are representative of pivot points in our communities. The publishing functions of conferences are able to do this to some degree. Barcamps and foocamps have not really produced published outcomes at this stage. For materials which are explicitly focused this kind of Howard Rheingold Flashmob approach might be useful. For personal journeys it would be interesting to see these kinds of aggregations of specific learning journeys be able to be published as a single person's lulu book or to be able to view the reference paths of a range of people on a topic and to view their perspectives based on the sources they have looked at.

I think a traditional journal approach is possibly missing some of the adaptability and diversity of content and purpose which people are able to employ now. Something with more of a themed mashup approach would be more interesting to read for me.