Mohammed Nanabhay from Al Jazeera English (AJE) spoke about how Al Jazeera covered the Egyptian revolution. Political consultant Chris Faulkner spoke about Tea Party activism; Yesenia Sanchez, an organizer for the P.A.S.O./Illinois Coalition for Immigrant and Refugee Rights, talked about the "Undocumented, Unafraid and Unapologetic" campaign; NPR's Andy Carvin spoke about curating and verifying tweets from Egypt, Libya, Syria and elsewhere in the Arab Spring; and Baratunde Thurston, digital director of The Onion, gave a tremendous riff about his own -- and his mother's -- activism.

If discussions were not actually about Tahrir Square, Tunisia or the Gay Girl in Damascus, they were infused by the same spirit.

Given this activist spirit, it was highly fitting that, at the start of the conference last week, Chris Csikszentmihalyi announced that Ethan Zuckerman would be succeeding him as director of MIT's Center for Civic Media (where the conference was held). Zuckerman has been a central figure nurturing, filtering and aggregating civic media over the last decade at Harvard's Berkman Center and particularly through Global Voices Online that he set up with Rebecca McKinnon in 2005.

Civic media is hard to define, Zuckerman told the audience. It combines at least three elements:

• Organizing in a virtual and physical space simultaneously
• Self-documentation using participatory media
• Use of broadcast media as an amplifier

Digital Tools for Civic Purposes

In Tunisia, for example, people recorded themselves protesting and then published their recordings on Facebook. In Egypt, Facebook helped people organize political meetings and support groups. Zuckerman referred to other examples across the world where people were using digital tools for civic purposes. In Russia, people have been tracking wildfires using Ushahidi at Russian-Fires.ru. (Ushahidi is a Knight News Challenge winner.) In the United States, at LandmanReportcard.com, farmers and landowners have been keeping records of visits from "Landmen," negotiators for oil and gas companies, to expose disinformation and make sure they get a fair deal.

In Egypt, the public and the media learned from one another, AJE's Nanabhay told the conference attendees. People recorded themselves protesting and published it online. Al Jazeera amplified those recordings. As a consequence, people recorded themselves more. It was a self-perpetuating cycle of public media that grew and grew.

People are now all too conscious of the power of self-produced media, Nanabhay said. In the past, people committed dramatic "spectacles of dissent" in the belief that this was the only way of grabbing the attention of mainstream media. Now they stand with "a rock in one hand and a cell phone in the other," recording, publishing and promoting themselves and their causes, he said.

In the United States, the grown-up children of illegal immigrants have been taking videos of themselves "coming out" as having no documentation. The more people who take videos of themselves and publish them on the Net, the more empowered they feel, and the more others join them. See, for example, this YouTube video of an Undocumented, Unafraid and Unapologetic rally in March.

NPR's Carvin spoke about how many of his connections and sources in Syria, who had started tweeting anonymously, were now using their real names and pictures. They had crossed a line, they said, and there was no going back. If they were to die, then they wanted others to know who they were.

The conference captured the flavor of how people are now using digital tools to empower themselves and give volume to their dissent -- though this is by no means all about public anger and protest. Cronicas de Heroes Juarez, a project that came out of the Center for Future Civic Media, gathers and projects good news stories from the town of Juarez, Mexico. It was set up to balance the many bad news stories coming from the town that were creating an impression of a place in hopeless decline.

Public empowerment

A number of this year's Knight News Challenge prizes reflected this feeling of public empowerment, of people taking control of their own representation and information.

The biggest prize winner was The Public Laboratory, a project that initially appeared less digital and more paper, scissors, stone. The project uses string, balloons, kites and cameras to take aerial photographs of landscapes. These photographs are then threaded together digitally to provide detailed information about land use, pollution, and the progress of environmental initiatives. The project found its calling after the Gulf oil spill when satellite photographs simply were not detailed enough to see the spread of oil or its impact on the environment.

Zeega, another of this year's big winners, will help people video their own stories and edit them together on its open-source HTML5 platform. NextDrop gets even more practical still. It will provide a service that will tell communities on the ground in Hubli, Karnataka, India when water is available. The Tiziano project emerged from work done in Kurdistan and is intended to give communities the equipment, tools and training to illustrate their own lives.

These projects are highly pragmatic, focused on the public, not media professionals, and apply existing technologies to real-world problems. They don't start with the technology and then figure out what you might do with it.

In this world, in which the public organizes and records themselves, the role of the news media changes. Mainstream media shifts from recording media content itself to gathering existing material, verifying it, contextualizing it, and amplifying it. Other Knight News prizes recognized and were directed at this shift: iWitness and SwiftRiver, and -- for data -- Overview and Panda.

The Knight News Challenge has evolved a lot since its inauguration in 2006. But its strength lies in the consistency of its aims, and in the growing relevance of those aims: helping to inform and engage communities. Long may it continue.

The International Press Telecommunications Council (IPTC) has just launched rNews, a consistent, machine-readable way of expressing news metadata in RDFa (a linked data language). This post explains some of the differences between rNews and hNews and why, if you publish news on the web, you ought to be using one or the other.

In a now infamous incident at Cambridge University back in October 1946, mid-way through a seminar, the philosopher Ludwig Wittgenstein is said to have threatened the philosopher Karl Popper with a red-hot poker (the exact circumstances and use of the poker are still disputed, 65 years on). The argument? Over whether there are, or are not, such things as philosophical problems. Popper said there were, Wittgenstein said there were only puzzles.

Step into the similarly rarefied world of online publishing languages and, though you might not be threatened with a red-hot poker, someone will almost certainly wave its online equivalent at you -- as we found when we were developing hNews -- a news microformat -- with the Associated Press.

We started, back in 2008, with a problem: Very few online news stories had consistent, machine-readable information about their provenance (i.e. basic stuff like who wrote it, who published it, when it was first published, etc.). This was a problem because without this information -- or metadata -- it was incredibly difficult to differentiate news from other content on the web, or to figure out where news had come from.

Two Solutions to the Problem

We searched about for a solution to the problem, thanks to grants from the Knight and MacArthur Foundations, and found not one but two. The first was microformats -- which are straightforward, open mark-up formats built on existing standards. The second was RDFa, a method of embedding full RDF, the linked data language of the semantic web.

We made a decision to use microformats. We did this for highly pragmatic reasons. We figured that most news organizations (and journalists and bloggers) were not yet ready to make the big leap to linked data. The easier we made it to integrate consistent metadata, we thought, the more likely news organizations were to do it. Our chief concern was less about exactly how people made the provenance of online news more transparent, just that they did it.

The Associated Press came to a similar conclusion, and together we developed hNews. Our pragmatism has so far borne fruit. The hNews microformat has since been integrated in about 1,200 news sites in the U.S. This means that there must now be a hundred-plus million news stories on the web with hNews. And, the AP has based its new news registry business and its forthcoming rights clearinghouse around hNews.

This did not stop some semantic web evangelists from waving their metaphorical red-hot pokers, or from suggesting we were not born of parents in wedlock or other less warm and fuzzy responses.

So, when we learned that the IPTC were launching an equivalent of hNews in RDFa we were over the moon. Hooray! Now people have a choice to mark up their news in microformats or in linked data.

The Ambitious rNews

"Equivalent" is not quite right. rNews is more ambitious than hNews. If hNews is like a ham sandwich then rNews is like a baked Alaska. rNews covers lots of aspects of provenance and content. You can, if you want to mark up additional aspects of news stories, mix-and-match rNews with other RDF ontologies (i.e. different linked data vocabularies). It's also more "correct" than hNews, but as a result more verbose and intrusive. It's a much bigger change to existing HTML pages than hNews. That said, it is, by RDF standards, pretty straightforward. All this makes it a very good alternative way of creating consistent, machine-readable mark-up for news.

The big difference between two is in their complexity. Making a ham sandwich is much simpler and requires less expertise than cooking a baked Alaska. The same goes for hNews and rNews. As a result, my prediction is that rNews will be the format of choice for big news organizations who want to do things fully and properly and are willing to commit the time and resources (like the New York Times -- which was central to the development of rNews). In the same way it will probably suit high end proprietary content management systems. For smaller news organizations, journalists and bloggers, hNews goes a good part of the way there and is much easier to integrate and lighter to use.

In other words, the two complement each other rather well, and ought to provide the foundations for consistent, machine-readable metadata for news.

Pros and Cons of Each Approach

The AP's Stuart Myles was one of the creators of hNews and worked with the IPTC on rNews.

"The fact that hNews and rNews have similar names is no coincidence," Myles told me via email. "To me, microformats and RDFa are two different technical approaches to the same challenge. Each approach has pros and cons and many tools that support one also work with the other."

Evan Sandhaus of the New York Times, one of the original authors of rNews, also emphasizes the compatibility of the two standards: "rNews was designed from the start to provide publishers with many of the same features offered by hNews.  And future versions of the rNews will likely bring the standards into even closer alignment," he told me via email.

Should you care about hNews and rNews? If you publish news on the web then you most certainly should. The arrival of rNews and the continuing take-up of hNews show that metadata is central to the future of digital news. Consistent, machine-readable metadata makes your news easier to find, more distinguishable, more straightforward to check, more programmable, more targetable, and less hard to track. If you are not yet publishing your news with metadata then don't be surprised if someone soon comes at you flailing a red-hot poker.

Two weeks in, and the public response to Churnalism.com has been fantastic.

Since we launched the site on February 23, we have had 50,000 unique visitors, over 330,000 page impressions, and hundreds of press releases pasted in and saved. According to Google Analytics the site has been visited by people in 134 countries.

People have tracked down churnalism about eye-catching new products (such as "Baby Gaga," ice cream made with breast milk), about new research findings from universities (for example, on the "protective properties of green tea"), about new police initiatives (e.g., the recruitment of teenagers by police to prevent cyber-bullying), about the "happiest time of the week" (7:26 pm on a Saturday, says a poll sponsored by a multivitamin company), and about the prose of Jane Austen (which might not be all hers after all, according to an Oxford study). People have pointed us to stores of press releases like www.eurekalert.org and www.alphagalileo.org so we can build up a bigger bank of comparisons. And there have been discussions about what might constitute "signals of churnalism."

As importantly for us, the site has sparked lots of debate about churnalism. Here are some of the top questions that have come up:

Do the public care if journalists are churning out press releases?

Some felt the site's exposure of churnalism would not much bother the public.

Mark Stringer of Pretty Green told PR week he was "not sure why anyone would want to go to the time and effort of producing a website to prove something that no one really cares about."

Others thought the opposite was true.

"If you tell someone who is a punter rather than a journo that it's pretty standard practice to ctrl+C and ctrl+V huge chunks of a press release into a story," Steven Baxter wrote in his New Statesman blog, you'll get a revealing reaction. "I call it the 'Really?' face. People look at you as if to say 'Really? Is that what you do?'"

Our own experience to date appears to support Baxter's view rather than Stringer's.

Does the re-use of wire copy count as 'churnalism'?

There has been a fascinating discussion about the re-use of wire copy, especially when it is re-used almost verbatim, often with a byline from the news outlet added.

People have pointed out that news outlets subscribe to wire services to broaden their access to news, so why shouldn't they publish it?

Others have countered that using wire copy is not the problem, but passing it off as your own is.

"If you have to churn,"Minority Thought blogged, "at least be honest about it."

How can news organizations make their use of press releases more transparent?

On Memeburn, Tom Foremski wrote about a suggestion he made a few years back to color-code text that came from a press release. For example, distinguishing text "copied from a release or outside source (red)" from original text in black -- and potentially other colors to represent separate conflicts of interest. Others suggested just noting or linking to the release.

Professor George Brock, head of journalism at City University London, worried that rather than push journalists towards footnoting sources, Churnalism.com might discourage them.

Will Churnalism.com help reduce the production line approach to press releases?

A prominent communications professional, Mark Borkowski, welcomed the site, hoping it might help kill off the mass production of poor press releases.

So many are now produced, Borkowski wrote, that "the level of noise makes it hard for the true craft of the publicist to flourish."

Is all churnalism bad?

Alan Twigg of Seventy Seven PR told PR Week that "this site is making it sound like [public relations officers] getting coverage is a doddle and that PROs are taking over the media. If only it was that easy." Sounding a similar note, Stuart Skinner of PHA Media took to PR's defense on the same website, saying that "news is not a product of collusion between shady PROs and lazy journalists."

It is worth noting that the site does not say churnalism is easy, nor indeed that the reproduction of parts of press releases is necessarily unsavory.

"Of course not all churnalism is bad," the site's FAQ section says. "Some press releases are clearly in the public interest (medical breakthroughs, government announcements, school closures and so on). But even in these cases, it is better that people should know what press release the article is based on than for the source of the article to remain hidden."

Richard Sambrook also made an important point in his blog, that "there is of course Good PR and Bad PR just as there is Good Journalism and Bad Journalism."

Does Churnalism.com illustrate the self-correcting power of the web?

In the Guardian's online comment section Jimmy Wales, co-founder of Wikipedia, suggested that plagipedia and Churnalism.com "show us that the Internet is perfectly capable of correcting its own follies."

What's an equivalent word for "churnalism" in Spanish?

Great question. 1001Medios began a Twitter-hunt for a word in Spanish that captured the idea of "churnalism." Sadly, my Spanish is not good enough to work out if they've found one yet.

Building Buzz Without Legacy Media

The tremendous public response and debate almost certainly would not have happened without social media, blogs, and Chris Atkins. Chris' news stunts -- particularly about the chastity garter, the penazzle and Larry (or Jo) the cat -- captured public attention at the same time as making a serious point about how churn makes it into the mainstream media. (You can see Chris' film describing the stunts on the Guardian website, and his blog about it here.)

They also helped kick-start discussion about churnalism on social media, notably Twitter and Facebook. Thousands of people have tweeted about the "churnalism" problem, about Churnalism.com as a way to address the problem, about evidence of churn they have found, and yes, about Larry the Cat and the penazzle. It has been humbling and somewhat overwhelming to observe the level of public response and engagement.

Indeed, without social media and blogs there is every chance the site might have gone virtually unnoticed. The Guardian, which published the original "reveal" article about the news stunts, is still the only UK national newspaper site to have mentioned Churnalism.com.

Major news outlets that were fooled by Chris' PR stunts have yet to acknowledge their mistakes -- much less the website the hoaxes were intended to publicize. The BBC's Radio 5 Live is -- as far as we know -- yet to tell its listeners that the "Jo the Cat" story, which they discussed at length on their lunchtime program, was a fabrication. The Daily Mail does not appear to have informed its readers that Margaret Sutcliffe is not pursuing her custody claim about the Prime Minister's cat.

Contrast this with BBC Norfolk which immediately put its hands up and then used the hoax as a good way to start a discussion about churnalism.

Industry and International Attention

The public relations industry in the U.K. has been more direct in its response than the mainstream press. "PR Industry hits out at churnalism site" said an article on PRWeek.co.uk.

Various figures from the industry voiced their concern about the impact the site might have on the reputation of PR. Though in a measured and sensible leader, the editor Danny Rogers suggested churnalism was a genuine threat to both journalism and PR: "If organizations are churning out rubbish, and so-called journalists are mere accomplices in this process, we will all be taking part in a depressing downward spiral."

One of the really encouraging things about the response to the site in its first two weeks has been the international reaction. In addition to many kind words of encouragement, we have had expressions of interest from people to extend the site to the U.S., Germany, Finland, Spain, and Australia. We've spoken to NPR radio in New York, to CBC radio in Canada, BBC Radio Norfolk, BBC Wales and to community radio in Essex. We've been contacted by news organizations in Germany, Belgium, Australia, the U.S. and Russia.

What's Next for Churnalism.com?

Some of this interest is not in the site itself but in the technology that underlies it. The methodology we developed can be applied to many other uses beyond churnalism. It could be used, for example, to trace changes in the progress of legislation. It could be used to measure the re-use of Wikipedia. It could be applied to plagiarism in other parts of the web.

We're still pedaling furiously to respond to many of the questions people have raised and issues identified. We are, for example, about to introduce a page that allows people to explore the use of press releases by news outlet or sector (i.e. government, science). We are now highlighting, on the home page, what comparisons people are sharing (since people seem to prefer to share than to rate). We are adding a report button so people can tell us when something definitely is not churn.

Finally, we will start to link the site more directly with the other Media Standards Trust transparency projects -- notably journalisted.com and hNews. This should help us to create a whole toolbox of transparency and accountability mechanisms for online news and create an ecology that will foster and advantage original journalism.

But one of the unfortunate recent similarities has been the parallel crisis in local news, especially at newspapers. In both countries existing local news providers have been the hardest hit by the structural changes in news provision and consumption, each having relied so heavily on classified and recruitment advertising.

Yet the reactions of the two countries have been very different, particularly in the last couple of years. Comparing these different reactions helps illuminate why the U.S. is starting to see a future for local news in the digital era while the U.K. is still mired in the soup of its analog past.

U.S.: Experimentation

Over the last five years the U.S. has seen -- and continues to see -- lots of experimentation in the provision of local news and information. This has been due to:

Severity and speed of the American crisis.
Between 2007 and 2009, U.S. newspaper advertising revenues fell 43 percent, according to the State of the News Media 2010. Some news groups went under. In the majority of cases, however, this did not lead papers to close. Instead, the papers themselves became much less substantial (i.e. costs were carved out of editorial resources). At the same time, those within and outside the news industry searched frantically for new ways of gathering, publishing and delivering news.

Provision of foundation grants for new ideas and start-ups.
Since 2006, J-Lab estimates that "more than $141 million in non-profit funding flowed into new media." U.S. foundations -- most notably the Knight Foundation -- invested millions of dollars in experimentation. The Knight News Challenge, which funds this website, has given people a chance to compete for a share of a $5 million pot each year since 2006. The same foundation funded New Voices, an initiative that awarded "small grants to seed the launch of innovative community news venture." Other foundations, such as the Sandler Foundation, the Open Society Foundation, the MacArthur Foundation and Omidyar Network, also provided grants and start-up funds for projects as diverse as ProPublica, the Voice of San Diego, NewsTrust.net and Ushahidi.

Role of universities in hothousing and nourishing start-ups.
Many U.S. universities have, for many years, published highly professional local newspapers and news outlets. This has broadened and deepened since the crisis in local news kicked in. Some college news outlets, like the University of Miami's Grand Avenue News, have formed partnerships with commercial newspapers (in this case the Miami Herald). Some have developed news outlets and then sold them off to outside news companies (as with Montana University's Dutton County Courier to the Choteau Acantha newspaper). Others have won awards for their investigative journalism (like ChicagoTalks.org from Chicago's Columbia College). All these examples are taken from J-Lab's excellent research on What Works. There are many more.

Investment in regional or national networks of digital sites.
As the traditional news players collapsed, some new media players have jumped in to fill the gap. AOL launched Patch, a national network of local sites like Montclair Patch, and Chicago Heights Patch. In 2010, the Vocus State of the Media report found there were 724 online news launches, all but 36 of them on Patch. Other companies like Main Street Connect are trying to provide a similar local news template and service, if on a smaller scale than Patch (e.g. see the Daily Greenwich)

Through this experimentation, the U.S. has learned lots about what works and what doesn't. That is not to say it has "solved" the crisis in local news. That assumes there is a single solution, which there isn't. But there are different ways to address the underlying problem -- how people get the information they need to participate fully in democratic society -- and the U.S. has progressed along the road towards this.

U.K.: Conservatism

By comparison, there has been far less experimentation in the U.K. There are important exceptions to this rule but, compared to the U.S., the conservatism of the U.K. is striking. The reasons for this include:

The continuing dominance of four big news groups.
Four news groups control between 60 to 70 percent of the local news market: Trinity Mirror, Johnston Press, Northcliffe (owned by DMGT) and Newsquest (owned by Gannett). They have not distinguished themselves by their experimentation. Of the four, Trinity Mirror has, after a slow start, shown the most interest in trying to adapt to the digital era. It is launching hyper-local sites and collaborating with existing bloggers and community news sites. Northcliffe has a network of hyper-local sites but they are very cookie-cutter (see Market Harborough People vs. Melton Mowbray People), and appear to have minimal investment. Johnston and Newsquest are crippled by debt and many wonder how long they can continue. Yet while they do they help to squelch the development of nascent local media ventures.

The lack of foundation funding.
The U.K. does not have a similar legacy of supporting public media. Perhaps because of the dominance of the publicly funded BBC, foundations have not, in the past, tended to give grants to local media provision. This is now changing gradually, but we have yet to see a foundation investing heavily in local media in the way the Knight Foundation has in the U.S.

Introverted universities.
Similarly, though most universities have a university newspaper (and sometimes more than one), most of these are for and about the university, rather than for the wider community. Nor have many journalism departments sought to incubate, or launch, actual news startups. There are exceptions, of course. Goldsmiths College in London launched eastlondonlines.co.uk, an independent news website serving Hackney, Tower Hamlets, Lewisham and Croydon. But there is nothing on the scale or ambition of media ventures at U.S. universities.

Negative government intervention.
In the U.S., the government has stayed away from direct intervention in local media, and U.S. foundations have stepped in to partly fill the gap. In the U.K. not only have foundations not stepped in, but the government has, if anything, suppressed experimentation. It has done this partly by searching for ways to prop up the existing incumbents, and partly through its adherence to a top-down policy on local TV news.

Despite the conservatism of the incumbents, the lack of foundation funding, the lack of incubation at universities and the negative government intervention, there are British innovators, entrepreneurs and intrepid local startups (see here, here and here). But right now they are working against the grain in the U.K., which is not as it should be.

First, we should applaud the sentiment. Most of us would agree that it is a Good Thing that we should be able to distinguish between syndicated and non-syndicated content, and that we should be able to link back to original sources. So it is important to recognize that both of these are -- in theory -- important steps forward both from the perspective of news and the public.

But there are a number of problems with the meta tag scheme that Google proposes.

Problems With Google's Approach

Meta tags are clunky and likely to be gamed. They are clunky because they cover the whole page, not just the article. As such, if the page contains more than one article or, more likely, contains lots of other content besides the article (e.g. links, promos, ads), the meta tag will not distinguish between them. More important is that meta tags are, traditionally, what many people have used to game the web. Put in lots of meta tags about your content, the theory goes, and you will get bumped up the search engine results. Rather than address this problem, the new Google system is likely to make it worse, since there will be assumed to be a material value to adding the "original source" meta tag.

Though there is a clear value in being able to identify sources, distinguishing between an "original source" as opposed to a source is fraught with complications. This is something that those of us working on hNews, a microformat for news, have found when talking with news organizations. For example, if a journalist attends a press conference then writes up that press conference, is that the original source? Or is it the press release from the conference with a transcript of what was said? Or is it the report written by another journalist in the room published the following day? Google appears to suggest they could all be "original sources"; if this extends too far then it is hard to see what use it is.

Even when there is an obvious original source, like a scientific paper, news organizations rarely link back to it (even though it's easy to use a hyperlink). The BBC -- which is generally more willing to source than most -- has historically tended to link to the front page of a scientific publication or website rather than to the scientific paper itself (something the Corporation has sought to address in its more recent editorial guidelines). It is not even clear, in the Google meta-tagging scheme, whether a scientific paper is an original source, or the news article based on it is an original source.

And what about original additions to existing news stories? As Tom Krazit wrote on CNET:

The notion of 'original source' doesn't take into account incremental advances in news reporting, such as when one publication advances a story originally broken by another publication with new important details. In other words, if one publication broke the news of Prince William's engagement while another (hypothetically) later revealed exactly how he proposed, who is the "original source" for stories related to "Prince William engagement," a hot search term on Google today?

Differences with hNews

Something else Google's scheme does not acknowledge is that there are already methodologies out there that do much of what it is proposing, and are in widespread use (ironic given Google's blog post title "Credit where credit is due"). For example, our News Challenge-funded project, hNews already addresses the question of syndicated/non-syndicated, and in a much simpler and more effective way. Google's meta tags do not clash with hNews (both conventions can be used together), but neither do they build on its elements or work in concert with them.

One of the key elements of hNews is "source-org" or the source organization from which the article came. Not only does this go part-way toward the "original source" second tag Google suggests, it also cleverly avoids the difficult question of how to credit a news article that may be based on wire copy but has been adapted since -- a frequent occurence in journalism. The Google syndication method does not capture this important difference. hNews is also already the standard used by the largest American syndicator of content, the Associated Press, and is also used by more than 500 professional U.S. news organizations.

It's also not clear if Google has thought about how this will fit into the workflow of journalists. Every journalist we spoke to when developing hNews said they did not want to have to do things that would add time and effort to what they already do to gather, write up, edit and publish a story. It was partly for this reason that hNews was made easy to integrate into publishing systems; it's also why hNews marks information up automatically.

Finally, the new Google tags only give certain aspects of credit. They give credit to the news agency and the original source but not to the author, or to when the piece was first published, or how it was changed and updated. As such, they are a poor cousin to methodologies like hNews and linked data/RDFa.

Ways to Improve

In theory Google's initiative could be, as this post started by saying, a good thing. But there are a number of things Google should do if it is serious about encouraging better sourcing and wants to create a system that works and is sustainable. It should:

• Work out how to link its scheme to existing methodologies -- not just hNews but linked data and other meta tagging methods.
• Start a dialogue with news organizations about sourcing information in a more consistent and helpful way.
• Clarify what it means by original source and how it will deal with different types of sources.
• Explain how it will prevent its meta-tagging system from being misused such that the term "original source" becomes useless.
• Use its enormous power to encourage news organizations to include sources, authors, etc. by ranking properly marked-up news items over plain-text ones.

It is not clear whether the Google scheme -- as currently designed -- is more focused on helping Google with some of its own problems sorting news or with nurturing a broader ecology of good practice.

One cheer for intention, none yet for collaboration or execution.

I'm focused on the Chronicle these days because it is also one of 577 U.S. news sites now publishing articles with hNews (see the full list here).

hNews is the Knight News Challenge-funded news microformat we developed with the Associated Press that makes the provenance of news articles clear, consistent and machine readable. A news article with hNews will -- by definition -- identify its author, its source organization, its title, when it was published and, in most cases, the license associated with its use and a link to the principles to which it adheres (e.g. see AP essential news). It could also include where it was written, when it was updated, and a bunch of other useful stuff.

Essentially, hNews makes the provenance of a news article a lot more transparent, which is good news for whoever produces the article because it helps them receive credit for their work and creates potential revenue models, among other benefits. It's also good news for the end user because they are better able to assess an article's provenance, which helps them judge credibility and other important aspects.

A Growing List

We have long been aware that many sites have been integrating hNews; but there has not been a published list of these sites. This seemed to us a little unsatisfactory. So we went out and found as many of them as we could and have now published them on a list as an open Google doc.

There are, I understand, a few hundred more sites that have either already integrated hNews or are in the process of integrating it. We haven't found them yet, but will add them when we do. If you know of one (or if you are one) please let us know and we'll add it.

If you're interested in integrating hNews and are wondering why you should, you can read a piece I wrote for PBS MediaShift ("How metadata can eliminate the need for pay walls"), see the official specification at hNews microformats wiki, watch an hNews presentation by Stuart Myles, view a (slightly dated) slideshow on why it creates Value Added News, or see how to add hNews to WordPress.

If the news industry as a whole isn't exactly the picture of good health, local news is in the emergency room. News problems at a national level -- falls in circulation, and collapse in classified and advertising revenues -- are acute at a local level.

This has serious political implications, particularly in terms of who acts as the democratic watchdog, which is why this concerns not only news bosses but also politicians.

"We are concerned that ... the problems in the local media industry are leading to a scrutiny gap," read a report, Future for Local and Regional News, from the Parliamentary Select Committee for Culture, Media and Sport.

Defining Local

The problem is, when thinking about what to do about it, how do you define local? For Orkney Today this was pretty easy. It served a clearly defined geographic area -- the Orkney isles -- that is run by the Orkney local council, and that has a long established sense of community. But what about places that aren't surrounded by sea, that don't have a single local authority, and may not have such a long established sense of community?

This isn't an academic question. In political -- i.e. public policy -- terms how you define local will determine what you do and how you do it. How can a government, for example, even consider direct or indirect subsidies, for example, without knowing who to give them to and what parameters to set?

Boil it down and you can probably define "local" in three different ways: Politically, economically, or socially. (I'm deliberately ignoring random geographic boundaries even though that's how regional broadcast news appears to be defined right now). The way you choose to define local then has fundamental implications for the type of journalism you end up with.

If you're in government you're probably most worried about the health of democracy and so it makes logical sense to define "local" in political terms -- i.e. at the ward level, or the local authority or county council boundary, or the constituency. This way you highlight the watchdog role of journalism. You make clear that, as a society, you believe in the idea of a "Fourth Estate" -- a section of society whose role it is to scrutinize local politics, uncover corruption, and tell truth to power.

The problem with this is that political boundaries don't necessarily make economic sense or correspond to what people think of as local. Take my ward in England, called "Kingham, Rollright and Enstone." I don't live in Kingham, Rollright or Enstone, I live just outside Chipping Norton. So a news service called The Kingham, Rollright and Enstone Times wouldn't seem very relevant to me. On top of which my ward is pretty spread out (it's rural) and there are only about 4,000 people in it in total. That's too few for most professional news organizations to bother with, unless they can get costs close to nil.

Because if you're a news organization then while you're thinking about local politics you're also thinking economics. You have to be if you're going to survive. You have to think about how many eyeballs you need to make enough revenue via circulation, subscriptions, classifieds, etc. You're making a calculation that, say, you need to sell 10,000 print copies a week to get by. With 10 percent penetration that means you need to serve an area of about 100,000 people. Multiply the numbers considerably for bigger publications or for broadcast. But the problem with an economic definition of local is that it's unlikely to match the public's perception.

If you're a member of the public then local probably means your street, your neighborhood, your town. What the news industry likes to call hyper-local. As a participant in a recent Birmingham focus group said, "If it's not within a 10 mile radius, it's not local news as far as I'm concerned ... it might as well be national." That quote comes from "Meeting the News Needs of Local Communities," a research report released this month by Media Trust. News at this level is great for building community cohesion and for making people feel a part of a bigger society, but it's hard to imagine anyone but volunteers and non-profits providing it in a sustainable way.

Recipe for Success

That's why it's so hard for a government, or a news organization, to know what to do. You can't create this sort of genuine hyper-local news service from the top down. Neither the government nor a news organization can direct the public to produce news about where they live. This sort of news has to be from the ground up. It has to be participatory. It has to be by and for the local community.

Which is why the local news organizations/co-operatives/forums most likely to work are those that start from the bottom, and that build participation, collaboration, mutualization, and partnerships into their DNA. This is very hard indeed for legacy news organizations to do. And it means that the best a government can do is to create a framework in which people are able to fill the vacuum being left by the disappearance of local news, rather than trying to subsidize the existing industry or provide top-down direct support.

Last year the U.K.'s Daily Telegraph secreted half a dozen reporters in a room for nine days with about 4 million records of politicians' expenses. They were hidden away even from the paper's own employees. Now we learn that reporters from the Guardian, the New York Times and Der Spiegel did the same with Julian Assange of WikiLeaks somewhere in the Guardian's offices in King's Cross, London.

There is a wonderful irony that open data can generate such secrecy. Of course the purpose of this secrecy is to find -- and protect -- scoops buried in the data. From the perspective of many news organizations, these scoops are the main benefit of data dumps. Certainly the Daily Telegraph benefitted hugely from the scoops it dug out of the MPs' expenses data. Weeks of front pages on the print paper, national uproar, multiple resignations, court cases and much soul searching about the state of parliamentary politics.

The Guardian, the New York Times and Der Spiegel have not been able to stretch the WikiLeaks Afghan logs over multiple weeks, but they did dominate the news for awhile, and stories will almost certainly continue to emerge.

These massive data releases are not going to go away. In fact, they're likely to accelerate. The U.S. and U.K. governments are currently competing to see who can release more data sets. WikiLeaks will no doubt distribute more raw information, and WikiLeaks will spawn similar stateless news organizations. Therefore news organizations need to work out how best to deal with them, both to maximize the benefits to them and their readers, and to ensure they don't do evil, as Google might say.

5 Questions

Here are just five (of many) questions news orgs should ask themselves when they get their next data dump:

1. How do we harness public intelligence to generate a long tail of stories? Though the Telegraph succeeded in unearthing dozens of stories from the Parliamentary expenses data, the handful of reporters in the bunker could never trawl through each of the millions of receipts contained on the computer disks. It was The Guardian that first worked out how to deal with this; it not only made the receipts available online but provided tools to search through them and tag them (see Investigate your MP's expenses). This way it could harness the shared intelligence -- and curiosity -- of hundreds, if not thousands, more volunteer watchdogs, each of whom might be looking for a different story from the expenses data. As a result, the Guardian generated many more stories and helped nurture a community of citizen scrutineers

2. How do we make it personal? Massive quantities of data can be structured to be made directly relevant to whoever is looking at it. With crime data you can, for example, enable people to type in their postcode and see what crimes have happened in their neighborhood (e.g. San Francisco crimespotting). For MPs' expenses, people could look up their own MP and scour his/her receipts. The Afghan logs were different in this respect, but OWNI, Slate.fr and Le Monde Diplomatique put together an app that allows you to navigate the logs by country, by military activity, and by casualties (see here). The key is to develop a front end that allows people to make the data immediately relevant to them.

3. How can use the data to increase trust? The expenses files, the Afghan logs, the COINs database (a massive database of U.K. government spending released last month) are all original documents that can be tagged, referenced and linked to. They enable journalists not only to refer back to the original source material, but to show an unbroken narrative flow from original source to final article. This cements the credibility of the journalism and gives the reader the opportunity to explore the context within the original source material. Plus, if published in linked data, the published article can be directly linked to the original data reference.

4. How do we best -- and quickly -- filter the data (and work out what, and what not, to publish)? Those that are best able to filter this data using human and machine methods are those who are most likely to benefit from it. Right now only a very small number of news organizations appear to be developing these skills, notably the Guardian, the New York Times, and the BBC. The skills, and algorithms, they develop will give them a competitive advantage when dealing with future data releases (read, for example, Simon Rogers on how the Guardian handled the 92,201 rows of data and how Alastair Dant dealt with visualizing IED events at FlowingData). These skills will also help them work out what not to publish, such as data that could put people in danger.

5. How can we ensure future whistleblowers bring their data to us? It's impossible to predict where a whistleblower will take their information. John Wick, who brokered the MPs expenses disk to the Telegraph, went first to the Express, one of the U.K.'s least well resourced and least prepared national papers. But it is likely that the organizations that become known for handling big data sets will have more whistleblowers coming to them. Julian Assange went to the Guardian partly because the journalist Nick Davies sought him out in Brussels (from Clint Hendler in CJR) but Assange must also have been convinced the Guardian would be able to deal with the data.

The influence of the war logs continues to spin across the globe, particularly following the Afghan president's comments. But it is not the first -- and certainly won't be the last -- big data dump. Better that news organizations prepare themselves now.

After one of the sessions I spotted an editor whose newspaper had adopted hNews (the Knight-funded news metadata standard we developed with the AP). "How's it going?" I asked him. "Is it helping your online search? Are you using it to mark up your archive?"

Before I had even finished the editor was jotting something down on his notepad. "Here," he said, "Call this guy. He's our technical director -- he'll be able to help you out."

Technology and innovation still remain, for most editors, something the techies do.

So it's not that surprising that over much of the last decade, innovation in news has been happening outside the news industry. In news aggregation, the work of filtering and providing context has been done by Google News, YouTube, Digg, Reddit, NowPublic, Demotix and Wikipedia...I could go on. In community engagement, Facebook, MySpace, and Twitter led the way. In news-related services (the ones that tend to earn money) it has been Craigslist, Google AdWords and now mobile services like Foursquare.

Rather than trying to innovate themselves, many news organisations have chosen instead to gripe from the sidelines. Rupert Murdoch called Google a "thief" and a "parasite." The U.K.'s Daily Mail has published stories about how using Facebook could raise your risk of cancer,, referred to someone as a "Facebook killer" (as in murderer), and runs scare stories about Facebook and child safety. And let's not even start to take apart various news commentators' dismissive attitude towards Twitter.

When they have seen the value of innovation, news organizations have tended to try and buy it in rather than do it themselves, with decidedly mixed results. Murdoch's purchase of MySpace initially looked very smart, but now, as John Naughton wrote over the weekend, it "is beginning to look like a liability." The AOL /Time Warner mashup never worked. Associated Newspapers in the U.K. have done slightly better by making smaller investments in classified sites.

Most news organisations do not see innovation as a critical element of what they do. This is not that unexpected since they spend their day jobs gathering and publishing news. Unfortunately for them, if it doesn't become more central to their DNA they are liable to become extinct.

Speed and Unpredictability of Innovation

At last week's Guardian Activate Summit, Eric Schmidt, Google's CEO, was asked what kept him awake at nights. "Almost all deaths in the IT industry are self-inflicted," Schmidt said. "Large-scale companies make mistakes because they don't continue to innovate."

Schmidt does not need to look far to see how quickly startups can rise and fall. Bebo was started in 2005, was bought by AOL in 2008 for $850 million, and then was sold again this month to Criterion Capital for a fee reported to be under $10 million.

The problem for Schmidt -- and one that is even more acute for news organizations -- is the increasing speed and unpredictability of innovation. "I'm surprised at how random the future has become," Clay Shirky said at the same Activate summit, meaning that the breadth of participation in the digital economy is now so wide that innovation can come from almost anyone, anywhere.

As an example he cited Ushahidi, a service built by two young guys in Kenya to map violence following the election in early 2008 that has now become a platform that "allows anyone to gather distributed data via SMS, email or web and visualize it on a map or timeline." It has been used in South Africa, the Democratic Republic of Congo, India, Pakistan, Gaza, Haiti and in the U.S.

He might also have cited Mendeley, a company which aims to organize the world's academic research papers online. Though only 16 months old, the service already has over 29 million documents in its library, and is used by over 10,000 institutions and over 400,000 people. It won a prize at Activate for the startup "most likely to change the world for the better."

The tools to innovate are much more widely available than they were. Meaning a good idea could be conceived in Nairobi, Bangalore or Vilnius, and also developed and launched there too, and then spread across the world. "The future is harder to predict," Shirky said, "but easier to see."

That's why Google gives one day a week to its employees to work on an innovation of their choice (Google News famously emerged from one employee's hobby project). It is why foundations like Knight have recognized the value of competition to innovation. And it's why Facebook will only enjoy a spell at the peak.

Some Exceptions

There are exceptions in the news industry. The New York Times now has an R&D department, has taken the leap towards linked data, and published its whole archive in reusable RDF. The Guardian innovated with Comment is Free, its Open platform, and the Guardian Data Store. The BBC developed the iPlayer.

The Daily Telegraph had a go, setting up "Euston Partners" under then editor Will Lewis. (Although setting up an innovation center three miles away from the main office did not suggest it was seen as central to the future of the business.) The project was brought back in-house shortly after Lewis left the Telegraph in May 2010 and has been renamed the "Digital Futures Division."

But mostly people in news don't really do innovation. They're too focused on generating content. But as the Knight Foundation has recognized, doing news in the same old way not only doesn't pay -- it doesn't even solve the democratic problems many of those in news are so rightly concerned about. For some people FixMyStreet.com or its U.S. equivalent SeeClickFix is now more likely to give them a direct relationship with their council than the local newspaper.

News and media organizations have to realize that they are in the communications business, and being in that business means helping people to communicate. Giving them news to talk about is a big part of this, but it's not the only part. The sooner they realize this and start to innovate, the better chance they have of surviving the next couple of decades.

Still, the optimists have got it half right. As Stewart Kirkpatrick, founder of the Caledonian Mercury, said at a #futureofnews conference a week or so back (I paraphrase): "This is a great time to do journalism. It's just not a great time to earn your living as a journalist."

What I Know

But, in these turbulent times, as I earnestly make my way from one news conference to another, a few things are starting to become clear. So this much I know:

• Even if pay walls provide a secure financial future for news organizations -- which right now seems unlikely -- they will reduce the pool of shared information, and cut those news organizations' content off from the openness, sharing and linking that characterizes the web. "You cannot control distribution or create scarcity," Alan Rusbridger said in his January Hugh Cudlipp lecture, "without becoming isolated from this new networked world."
• The pay wall is not the only way to sustain the digital newsroom. Advertising, though much maligned by many, could yet make online non-pay wall newspaper content viable within five years. Peter Kirwan did the math in Wired, calculating that if Guardian News Media manages a 20 percent annualized growth of digital revenues (it estimates growth will be 30 percent this year) it will be able to maintain a £100m digital newsroom seven days a week by 2015.
• There are other revenue models for online news -- ones that allow you to keep your news open, linked and shared, and make money. For example, there is what I call the "carrier pigeon model." In this model you let people share, link to, recommend, search, aggregate, and even re-use you content -- you just make sure it's properly marked up and credited so you can keep track of it and develop revenue models off the back of it. You do this with -- excuse the geek terminology -- "metadata." Embedded metadata has all sorts of potential benefits we're only just starting to take advantage of (hence why we've spent so much time on hNews and linked data). I call it the carrier pigeon model because the news doesn't just go out, it comes back.
• The cost base is still going to have to go down. The cost of producing news will necessarily have to be a lot lower than it has been historically. This doesn't have to mean cutting journalist's jobs or getting out of print. There are lots of ways to rethink costs in a digital world. One of the most inventive is Roman Gallo's Czech model. Gallo opened cafés in the centre of towns across the Czech Republic. He then put his news teams in the cafés. Not only does this mean they have very low office overhead (the café covers basic costs), but it means the journalists are working in amongst the local community and getting readers directly involved in production.
• There will need to be accessible, re-usable public data provided regularly and in a consistent format. Without this it will be much harder to keeps costs low because of the amount of time it takes to coax information out of public authorities and then analyze that data. This is why the launch of data.gov.uk was such an important development, and why we need to join Sir Tim Berners-Lee's quest for "raw data now" (as he shouts in his wonderfully quirky TED appearance).
• Whether or not pay walls work or online news makes money, there will be a public interest gap. Some newsgathering and reporting will almost certainly never again be commercially profitable in an open market. Online news is highly unlikely ever to pay for a journalist to sit in a local court for days on end, for example. This was one of the most important things to come out of Michael Schudson and Leonard Downie's report, "The Reconstruction of American Journalism." Schudson and Downie could not find a market solution to some of the news problems they were exploring, and so settled instead on a mixture of tax breaks, subsidies, foundation grants, and donations.
• We will rely, for aspects of watchdog journalism, on a combination of journalists, NGOs, and motivated members of the public. Note the use of the word "motivated." News organisations will need to find ways -- other than money -- to motivate and sustain people to help them scour data, dig through school and healthcare records, and alert them to corruption and injustice.
• As well as motivating people, news organizations will need to build the tools that help the non-professional journalists be watchdogs -- tools like whatdotheyknow.com, a site built by MySociety that makes it relatively easy for people to make freedom of information requests and share the results of those requests with a wider community. Or the way the Guardian got the public to search through the millions of MPs expenses claims.
• News organizations and journalists will need to form and re-form partnerships with other organizations, journalism co-operatives, NGOs and members of the public. We're seeing this start to happen with sites like the Bay Citizen in San Francisco (see a good post by Mallary Jean Tenore on Poynter) and OpenFile, the beta site just launched by MediaShift managing editor Craig Silverman et al in Canada.

Even taking all this into account there's a good chance that, without some tweaking of the market, a few tax breaks here, maybe a start-up fund there, there will be a lot of public interest news blackspots.

So there it is. Not so bleak, but not so rosy, either. And take it with a big pinch of salt since the only ones who seem to know about profitable business model for news just now are those running #futureofnews conferences.

It shouldn't, and I'll share 10 reasons why.

Before I do, I should briefly explain what I mean by "linked data." Linked data is a way of publishing information so that it can easily -- and automatically -- be linked to other, similar data on the web. For example, if I refer to "Paris" in a news article, it's not immediately apparent to search engines whether that is Paris, France or Paris, Texas, or Paris Hilton or another Paris entirely. If published in linked data, Paris would be linked to another reference point that would make clear which one it referred to (e.g. the entry for Paris, France on dbpedia, the structured data version of Wikipedia).

Until a short while ago, I was reasonably clueless as to both the meaning and the value of linked data. I'm still far from an expert, but enough people who are far smarter than me have convinced me that it's worth trying. This was especially the case a couple of months back, at a News Linked Data Summit that we -- the Media Standards Trust -- organized with the BBC. You can read about it in a previous blog post.

10 Reasons

So, here are 10 reasons why news organizations should bump linked data up their priority list:

1. Linked data can boost SEO (search engine optimization).
   People who tell you they can boost your SEO usually sound like witch doctors, telling you to tag all sorts of hocus pocus that doesn't make rational sense or just seems like cynical populism. But at its simplest, SEO works through links. The more something is linked to, the higher it will rank in search results. So publishing content as linked data should, quite naturally, increase its SEO. A great example of this is the BBC's natural history output. Type "Lion" into Google and, chances are, a BBC linked data page will come in those results. This never used to happen until the BBC started tagging their natural history content as linked data.
2. Linked data allows others to link to your site more easily.
   The world wide web is, more and more, powered by algorithms; the Google search algorithm is perhaps the most obvious. But most sites now take advantage of some mechanized intelligence. "If you liked reading this, you might enjoy this..." is one example. Algorithms are intelligent -- but not that intelligent. They have trouble telling the difference between, for example, Martin Moore (me), Martin Moore (kitchens), and Daniel Martin Moore (the Kentucky singer/songwriter). But use linked data and they can tell the difference. And once they can, sites like the BBC can link externally much more easily and intelligently.
3. It helps you build services based on your content.
   It's difficult to get people to pay for news online, so news organizations will need to build services based on their news -- and other content -- that people will pay for. You could, for example, provide a service that enabled people to compare schools in different areas, based on inspection reports, league tables, news reports, and parents' stories. Creating services to do this is lots and lots easier if content is made machine-readable through linked data.
4. It enables other people to build services based on your content that you could profit from.
   Other people often have ideas you haven't thought of. Other people might also have the space and time to experiment. Give them the opportunity to build stuff through linked data and they might come up with killer apps that make you money. iPhone apps anyone?
5. It allows you to link direct to source.
   You're a news organization. Your brand is based partly on how much people trust the stuff you publish. Publishing in linked data enables you to link directly back to the report/research or statistics on which it was based -- especially if that source is itself linked data (such as this). That way, if you cite a crime statistic, say, you can link it directly back to the original source.
6. It helps journalists with their work.
   As a news organization publishes more of its news content in linked data, it can start providing its journalists with more helpful information to inform the articles they're writing. Existing linked data can also provide suggestions as to what else to link to.
7. It throws bait over the pay wall.
   Once content is behind a pay wall it becomes invisible unless you pay. (That's sort of the point.) This is the same for Joe Public as for a search engine. But how are you, Joe Public, supposed to work out whether you want to pay for something if it's invisible? Publish in linked data and there will be enough visible bits of information to help people decide if they want to pay for it. (This will probably be less of a deal with big search engines like Google, but more relevant to other search engines and third party services. Mind you, one of these bit players will, most likely, be the next Google or Facebook.)
8. It makes data associated with your content dynamic.
   There is an ever growing mountain of information on the Net that never gets updated. There are pages devoted to soccer teams whose last score was added in 2006. Or topic pages about political issues that haven't seen a new story in months. But if those pages were filled with linked data, and linked to others that were too, they'd be automatically updated -- rising from the dead like Frankenstein without you having to do diddly squat.
9. You could become a "canonical reference point."
   "What the heck is a canonical reference point," I hear you ask. Well, it's a little like a virtual Grand Central Station. It's a junction point for linked data; a hub that hundreds or even thousands of other sites link to as a way of helping to define their references. Examples of such hubs include: musicbrainz.org for music and musicians, data.gov.uk for U.K. government stuff, dbpedia.org for almost anything. If you're a news organization, why wouldn't you want to be a hub?
10. It raises the bar for all.
    A web of linked data is a more intelligent web. A more mature and less superficial web. Not quite a semantic web, but getting there.

Of course, some of these benefits will come disproportionately to first movers (as with the BBC's natural history pages). Which is exactly why news organizations, who have previously been pretty slow when it comes to web innovation, need to get their skates on.

Additional Reading on Linked Data:

These articles are worth reading:

• "Linked data is blooming - why you should care' on the ever readable Read Write Web, May 2009
• A graphic of the linked data web
• Tim Berners-Lee talking about linked data at TED 2009
• U.K. government public linked data
• My blog post about our linked data summit

Back in 2005 it took the Los Angeles Times a year of internal discussions to settle on its ethical guidelines for journalists. The Committee for Concerned Journalists took four years, did oodles of research and held 20 public forums, in order to come up with a Statement of Shared Purpose with nine principles (which was subsequently fleshed out in the excellent "The Elements of Journalism" by Kovach and Rosenstiel).

Time spent thinking can then translate into a lot of principles. The BBC's editorial guidelines -- which include guidance about more than just journalism -- run to 228 pages. The New York Times' policy on ethics in journalism has more than 10,000 words. Principles needn't be so wordy. The National Union of Journalists (U.K.) code of conduct, first drafted in 1936, has 12 principles adding up to barely more than 200 words.

But, once defined, these principles serve multiple functions. They act as a spur to good journalism, as well as a constraint on bad. They provide protection for freedom of speech and of the press -- particularly from threats or intimidation by the government or commercial organizations. And they protect the public by preventing undue intrusion and providing a means of response or redress.

Principles in the Online World

In an online world, principles can serve another function. They can help to differentiate journalism from other content published on the web, whether that be government information, advertising, promotion, or institutional or personal information.

One of the key elements of hNews -- the draft microformat the Media Standards Trust developed with the AP to make news more transparent -- is rel-principles. This is a line of code that embeds a link within each article to the news principles to which it adheres. It doesn't specify what those principles should be, just that the article should link to some.

Now that lots of news sites are implementing hNews (over 200 sites implemented the microformat in January), we're getting some pushback on this. News sites, and bloggers, generally recognize that transparent principles are a good idea but, having not previously made them explicit online, many of them aren't entirely sure what they should be.

When we started working with OpenDemocracy, for example, they realized they had not made their principles explicit. As a result of integrating hNews, they now have. Similarly, the information architect and blogger Martin Belam, who blogs at currybet.net and integrated hNews in January 2010, wrote: "it turned out that what I thought would be a technical implementation task actually generated a lot of questions addressing the fundamentals of what the site is about... It meant that for the first time I had to articulate my blogging principles."

So, in an effort to help those who haven't yet defined their principles, we're in the process of gathering together as many as we can find, and pulling out the key themes.

This is where you can help.

Asking for Feedback

We've identified 10 themes that we think characterize many journalism statements of principle. This is a result of reviewing dozens of different (English language) principles statements available on the web. The statements were accessed via the very useful journalism ethics page on Wikipedia; via links provided by the Project for Excellence in Journalism; and from the Media Accountability Systems listed on the website of Donald W. Reynolds Institute of Journalism.

These themes are by no means comprehensive -- nor are they intended to be. They are a starting point for those, be they news organizations or bloggers, who are drawing up their own principles and need a place to start.

We'd really like some feedback on whether these are right, if ten is too many, if there are any big themes missing, and which ones have most relevance to the web.

Ten Themes

Our 10 themes are:

1. Public interest Example: "... to serve the general welfare by informing the people and enabling them to make judgments on the issues of the time" (American Society of Newspaper Editors)
2. Truth and accuracy Example: "[The journalist] strives to ensure that information disseminated is honestly conveyed, accurate and fair" (National Union of Journalists, UK)
3. Verification Example: "Seeking out multiple witnesses, disclosing as much as possible about sources, or asking various sides for comment... [The] discipline of verification is what separates journalism from other modes of communication, such as propaganda, fiction or entertainment" (Principles of Journalism, from Project for Excellence in Journalism)
4. Fairness Example: "... our goal is to cover the news impartially and to treat readers, news sources, advertisers and all parts of our society fairly and openly, and to be seen as doing so" (New York Times Company Policy on Ethics in Journalism)
5. Distinguishing fact and comment Example: "... whilst free to be partisan, [the press] must distinguish clearly between comment, conjecture and fact" (Editors Code of Practice, PCC, U.K.)
6. Accountability Example: "The journalist shall do the utmost to rectify any published information which is found to be harmfully inaccurate" (International Federation of Journalists, Principles on the Conduct of Journalists)
7. Independence Example: "Journalists should be free of obligation to any interest other than the public's right to know... [and] Avoid conflicts of interest, real or perceived" (Society of Professional Journalists)
8. Transparency (regarding sources) Example: "Aim to attribute all information to its source. Where a source seeks anonymity, do not agree without first considering the source's motives and any alternative, attributable source. Where confidences are accepted, respect them in all circumstances" (Australian Journalists Code)
9. Restraint (around harassment and intrusion) Example: "The public has a right to know about its institutions and the people who are elected or hired to serve its interests. People also have a right to privacy and those accused of crimes have a right to a fair trial. There are inevitable conflicts between the right to privacy, the public good and the public's right to be informed. Each situation should be judged in the light of common sense, humanity and the public's rights to know" (Canadian Association of Journalists)
10. Originality (i.e. not plagiarizing) Example: "An AP staffer who reports and writes a story must use original content, language and phrasing. We do not plagiarise, meaning that we do not take the work of others and pass it off as our own" (Associated Press Statement of news values and principles)

There are, of course, many excluded from here. We could, for example, have gone into much more depth in the area of "limitation from harm," which is only briefly referred to in number nine. Principles to inform newsgathering could form another whole section in itself.

There is also the growing area of commercial influence. In the U.S., the FCC FTC has become pretty animated about bloggers taking money to promote goods while appearing to be impartial. In the online world, the line between editorial and commercial content can get pretty blurred. Right now this is just covered by number five, independence. Should there be a separate principle around independence from commercial influence?

Any and all responses are much appreciated, so please leave them in the comments. Also feel free to get in touch directly if you'd like to continue the discussion (I'm at martin DOT moore AT mediastandardstrust DOT org).

The already-infamous release of climate change emails was a fantastically successful drop. Though the emails themselves date from the late 1990s onwards, their release was perfectly timed to capture the media's attention just before the Copenhagen climate talks -- to achieve maximum impact. And it worked.

Why? Not because they undermined the science of global warming. Only a hardened rump of skeptics still believe the world is not warming (as opposed to the larger number who dispute the causes and implications). Nor because they proved there was a global conspiracy of scientists determined to hide the truth from us (rather than a handful of scientists who might well have been manipulating aspects of their data). Nor to help promote a climate change skeptic think tank, the Global Warming Policy Foundation, which coincidentally launched around the same time.

The release of climate change emails dominated the headlines because it was a good news story. It fit with the basic need of news: to reveal something previously hidden, to uncover alleged wrongdoing, and to cast doubt on a widely held consensus.

Media Not Built to Cover Climate Well

In a larger sense, however, climate change doesn't fit with news' needs. The climate doesn't change on an hour-by-hour or daily basis, but over years and decades. It is theoretically urgent but, for most of us, not immediately apparent. Structurally, mainstream news is not built to cover long-term climate change well.

Mainstream news has a short attention span. Alastair Campbell, Tony Blair's former director of communications, remarked that if a politician could weather a media storm for 10 days then he would survive. The media would move on.

Newspapers -- understandably -- don't want to fill their front pages day in and day out with gradual news. Gradual news doesn't sell. "Antarctic ice cap retreated one foot" is unlikely to get people reaching into their pockets for loose change.

As a result, we tend to get treated either to occasional apocalyptic headlines of "the end is nigh" variety, or to news that bucks the scientific consensus.

So, when something like these emails comes along, the story is irresistible to most news outlets. Not just irresistible to cover, but irresistible to elevate the emails from an indication of shabby scientific behavior by a small number of scientists into evidence of a massive global warming conspiracy.

Contrarians, But About the Wrong Things

The problem is that a lot of those within mainstream media are a little bored of climate change. Journalists don't like consensus, especially not when it is foisted on them by ivory-towered experts on the basis of "trust us, we know more about this than you do." A lot of journalists are contrarians, and, for the most part, this is a very good thing.

But when it comes to climate change, many seem to be misdirecting their contrariness. Rather than being contrary about the science, about which the vast majority of journalists know very little, shouldn't journalists be contrarian about the difficult political implications? Isn't that the territory most of us are going to have to live on for the next 50 odd years? And the territory that most journalists would feel more comfortable inhabiting?

Last week, hNews became an official draft microformat. Having been proposed as a new data format and then discussed within the microformats community, it is now in draft 0.1 at Microformats.org. This means it has reached a stage where the microformat community believes it is stable enough for widespread adoption. This also reaffirms hNews as an open standard, free for anyone to integrate to their news content, whether they're from big news agencies like AP, a non-profit like OpenDemocracy.net, or individual journalists blogging on WordPress.

We also learned last week that AOL is adopting hNews. Though AOL has yet to make a formal announcement, hNews is already live on a number of its sites, including AOL News. This article, for example, has hNews embedded in its source code.

Then, this week, TownNews announced it was integrating hNews into its content management system. TownNews provides technology to support the publication of newspaper interactive editions online. By integrating hNews to their CMS, they suddenly make it available to up to 1,500 news sites across the U.S. If these news organizations want to start making their news a lot more machine-readable -- or "semantic" -- pretty much all they have to do is flick a switch.

This news builds on the adoption of hNews by the Associated Press. AP has not yet made its hNews marked-up content public, but plans to before the end of this year.

Making News Machine Readable

These developments are the culmination of the first stage of our transparency initiative, a non-profit project jointly funded by the Knight Foundation (we won a Knight News Challenge Award in 2008) and the MacArthur Foundation. We have also worked with the AP in the latter stages.

hNews, for those unfamiliar with it, makes some basic, factual information about the provenance of an online news article machine-readable. In other words, it makes distinguishable a lot of information that is currently indistinguishable on the web (e.g. to search engines). hNews is not the same as "beacon," "web bug," the controversial data tag that Associated Press is attaching to its content to help track its use around the web, and allow it, as I understand it, to create a "News Registry" of its users to create a registry of news - i.e. who owns it and how you can use it.. AP is layering beacon web bug on top of hNews.

The reason hNews is so useful to anyone producing journalism and to the public is that it helps to differentiate news on the web. At the same time, it should make news easier to find, give greater credit to the author (or help "ascribenation", as Doc Searls called it on LinuxJournal), link the story to the news principles it adheres to (if any), unlock some of the value of the news archive, and enable untold unintended consequences.

Currently, only some articles published by AOL, and a few hundred published by OpenDemocracy.net, the first adopter of hNews, are marked up. But within a month or so, there will be thousands and then perhaps hundreds of thousands of stories. Once that happens, we will actually be able to truly see how helpful hNews can be. The aim will then be to develop features and tools built on hNews, and begin benefiting from the marked up information. For example, this could be done via searches and APIs.

For us, the Media Standards Trust, the next stage will involve juggling many balls simultaneously. We need to communicate what hNews is and how it works to as many people as possible. This means making sure people realize that hNews is for anyone producing journalism, not just big news organizations. We also need to develop applications based on hNews in order to illustrate what it's useful for. And we need to keep evolving hNews to include additional (optional) semantic information. At the same time, we'll have to be flexible enough to cope with the unintended consequences.

We are still a little ways from seeing what impact hNews will have, but now we have the opportunity, over the next few months, to see how it can make news more transparent.

On one side of the fence you have pro-pay-wallers, led by the Murdochs, for whom pay walls seem to answer the question, "How are we going to solve the economic crisis in news?" They're in the process of trying to convince a great swath of big news organizations to stop providing their content free at the point of delivery. By doing this, the theory goes, they will enhance the value of news content by reviving scarcity and convince a new generation to start paying for news. The "freeniacs are wrong," writes Nicholas Carr, who adds that "Charging people for news, even online, is by no means an impossible dream."

On the other side you have the anti-pay-wallers, led by a growing and increasingly coherent group of technologists, liberal educationalists, and bloggers. They see pay walls as a complete misunderstanding of the new era of information abundance. To them, the construction of pay walls is a frantic attempt to recover a 20th century era of constrained media by a generation that "just doesn't get it."

Building pay walls is "desperate stuff," writes Stephen Foley. "It won't work, and if newspaper executives on both sides of the Atlantic follow Mr. Murdoch's apparent lead, I predict we will witness the collective suicide of scores of news organizations in the U.S. and elsewhere."

A debate that needs to refocus

Both sides are becoming more and more trenchant in their beliefs and are ramping up the rhetoric. But, as with the fight between the Big-Endians and Little-Endians in Gulliver's Travels (about which end to crack open your egg), this ideological dogmatism is distracting us from the more difficult questions. And it doesn't get us much closer to working out long term ways that will enable journalism to pay for itself.

The pro-pay-wallers need to acknowledge that pay walls are not the Holy Grail that will solve all their economic woes. They should listen to polls -- like the PaidContent UK/Harris poll from this week -- indicating that most people would leave their favorite news site for a free site elsewhere on the web if a pay wall was built. They should accept that it will not be possible to close the digital Pandora's Box that is the Internet and recreate the constrained published content environment of the 20th century.

The anti-pay-wallers should concede that there will be areas of content where pay walls work. Pay walls do not have to cordon off all -- or even the majority -- of information on a site. The Racing Post has a smart and sustainable hybrid strategy of offering significant amounts of content free, and only charging for that which it knows its readers highly value (as reported in the Independent earlier this week).

For £7.50 a month, members get a horse racing TV channel streamed live to their computer (for which 3,000 people signed up in the first week). For £9.50 a month, members can receive a "premium tipping service," and for £199.95 a year they can get "ultimate membership" with access to tips, races and the Racing Post database. Equally, the anti's should acknowledge that journalism, as we've grown to understand it, is far from free to produce.

Mired in ideological silos, the pro and anti-pay-wallers are also missing some of the most important aspects of the debate. How do you add value to the content itself, such that people will be more willing to pay for it? It's a question made more urgent for the Murdoch camp by the fact that most content becomes "invisible" as soon as it goes behind a pay wall.

The Hansel and Gretel theory

Here's where my Hansel and Gretel theory comes in. For those that don't remember the Grimm fairy tale, it goes something like this: Woodcutter's wife convinces woodcutter they can't afford to feed the children. Woodcutter therefore dumps children in the forest. But clever children find their way back by leaving a trail of pebbles. So woodcutter dumps them in forest again. This time, with only a breadcrumb trail, they can't get home. They then get imprisoned in a gingerbread house by an old witch and... you can read the rest here.

News stories have been, up untill now, a little like Hansel and Gretel's breadcrumbs. Spread by news organizations round the web, they quickly attract an audience, but that audience gobbles them up and rarely follows the breadcrumbs back home. What if instead of breadcrumbs, journalists and news organizations dropped pebbles? That way people wouldn't eat them and there would be more chance they could lead them back home.

The difference between a breadcrumb story and a pebble story is metadata. Embed some good consistent metadata in a story and it turns something ephemeral into something much more solid. People suddenly know, for example, where it came from. A story can have the equivalent of an address and a zip code built into it, so people follow it back by whatever trail they want.

Metadata has the significant added benefits that it is visible and malleable. It can be identified and picked up by search engines and aggregators. It can then be displayed so that people have enough information to know if they want more. It's a little like seeing the front page headlines on the newsstand before deciding to put your hand in your pocket for some change. It can also be used for cross-referencing stories, for digging through the archive, for building mash-ups.

Google has an example of how, using metadata, it can display more information about a site in its "search snippets." Similarly, we (the Media Standards Trust) have been working out how to best integrate metadata in news through our Knight / MacArthur Transparency Initiative.

The great pay wall debate is not going to end anytime soon -- but needs to be a little less polarized than it has been to date. Working out how to leave a trail of pebbles would be a good start.

[For more on this debate, see an excellent two-part series, The Great Debate Over Micropayments and Paid Content at MediaShift.]