Last week, hNews became an official draft microformat. Having been proposed as a new data format and then discussed within the microformats community, it is now in draft 0.1 at Microformats.org. This means it has reached a stage where the microformat community believes it is stable enough for widespread adoption. This also reaffirms hNews as an open standard, free for anyone to integrate to their news content, whether they're from big news agencies like AP, a non-profit like OpenDemocracy.net, or individual journalists blogging on WordPress.

We also learned last week that AOL is adopting hNews. Though AOL has yet to make a formal announcement, hNews is already live on a number of its sites, including AOL News. This article, for example, has hNews embedded in its source code.

Then, this week, TownNews announced it was integrating hNews into its content management system. TownNews provides technology to support the publication of newspaper interactive editions online. By integrating hNews to their CMS, they suddenly make it available to up to 1,500 news sites across the U.S. If these news organizations want to start making their news a lot more machine-readable -- or "semantic" -- pretty much all they have to do is flick a switch.

This news builds on the adoption of hNews by the Associated Press. AP has not yet made its hNews marked-up content public, but plans to before the end of this year.

Making News Machine Readable

These developments are the culmination of the first stage of our transparency initiative, a non-profit project jointly funded by the Knight Foundation (we won a Knight News Challenge Award in 2008) and the MacArthur Foundation. We have also worked with the AP in the latter stages.

hNews, for those unfamiliar with it, makes some basic, factual information about the provenance of an online news article machine-readable. In other words, it makes distinguishable a lot of information that is currently indistinguishable on the web (e.g. to search engines). hNews is not the same as "beacon," "web bug," the controversial data tag that Associated Press is attaching to its content to help track its use around the web, and allow it, as I understand it, to create a "News Registry" of its users to create a registry of news - i.e. who owns it and how you can use it.. AP is layering beacon web bug on top of hNews.

The reason hNews is so useful to anyone producing journalism and to the public is that it helps to differentiate news on the web. At the same time, it should make news easier to find, give greater credit to the author (or help "ascribenation", as Doc Searls called it on LinuxJournal), link the story to the news principles it adheres to (if any), unlock some of the value of the news archive, and enable untold unintended consequences.

Currently, only some articles published by AOL, and a few hundred published by OpenDemocracy.net, the first adopter of hNews, are marked up. But within a month or so, there will be thousands and then perhaps hundreds of thousands of stories. Once that happens, we will actually be able to truly see how helpful hNews can be. The aim will then be to develop features and tools built on hNews, and begin benefiting from the marked up information. For example, this could be done via searches and APIs.

For us, the Media Standards Trust, the next stage will involve juggling many balls simultaneously. We need to communicate what hNews is and how it works to as many people as possible. This means making sure people realize that hNews is for anyone producing journalism, not just big news organizations. We also need to develop applications based on hNews in order to illustrate what it's useful for. And we need to keep evolving hNews to include additional (optional) semantic information. At the same time, we'll have to be flexible enough to cope with the unintended consequences.

We are still a little ways from seeing what impact hNews will have, but now we have the opportunity, over the next few months, to see how it can make news more transparent.

On one side of the fence you have pro-pay-wallers, led by the Murdochs, for whom pay walls seem to answer the question, "How are we going to solve the economic crisis in news?" They're in the process of trying to convince a great swath of big news organizations to stop providing their content free at the point of delivery. By doing this, the theory goes, they will enhance the value of news content by reviving scarcity and convince a new generation to start paying for news. The "freeniacs are wrong," writes Nicholas Carr, who adds that "Charging people for news, even online, is by no means an impossible dream."

On the other side you have the anti-pay-wallers, led by a growing and increasingly coherent group of technologists, liberal educationalists, and bloggers. They see pay walls as a complete misunderstanding of the new era of information abundance. To them, the construction of pay walls is a frantic attempt to recover a 20th century era of constrained media by a generation that "just doesn't get it."

Building pay walls is "desperate stuff," writes Stephen Foley. "It won't work, and if newspaper executives on both sides of the Atlantic follow Mr. Murdoch's apparent lead, I predict we will witness the collective suicide of scores of news organizations in the U.S. and elsewhere."

A debate that needs to refocus

Both sides are becoming more and more trenchant in their beliefs and are ramping up the rhetoric. But, as with the fight between the Big-Endians and Little-Endians in Gulliver's Travels (about which end to crack open your egg), this ideological dogmatism is distracting us from the more difficult questions. And it doesn't get us much closer to working out long term ways that will enable journalism to pay for itself.

The pro-pay-wallers need to acknowledge that pay walls are not the Holy Grail that will solve all their economic woes. They should listen to polls -- like the PaidContent UK/Harris poll from this week -- indicating that most people would leave their favorite news site for a free site elsewhere on the web if a pay wall was built. They should accept that it will not be possible to close the digital Pandora's Box that is the Internet and recreate the constrained published content environment of the 20th century.

The anti-pay-wallers should concede that there will be areas of content where pay walls work. Pay walls do not have to cordon off all -- or even the majority -- of information on a site. The Racing Post has a smart and sustainable hybrid strategy of offering significant amounts of content free, and only charging for that which it knows its readers highly value (as reported in the Independent earlier this week).

For £7.50 a month, members get a horse racing TV channel streamed live to their computer (for which 3,000 people signed up in the first week). For £9.50 a month, members can receive a "premium tipping service," and for £199.95 a year they can get "ultimate membership" with access to tips, races and the Racing Post database. Equally, the anti's should acknowledge that journalism, as we've grown to understand it, is far from free to produce.

Mired in ideological silos, the pro and anti-pay-wallers are also missing some of the most important aspects of the debate. How do you add value to the content itself, such that people will be more willing to pay for it? It's a question made more urgent for the Murdoch camp by the fact that most content becomes "invisible" as soon as it goes behind a pay wall.

The Hansel and Gretel theory

Here's where my Hansel and Gretel theory comes in. For those that don't remember the Grimm fairy tale, it goes something like this: Woodcutter's wife convinces woodcutter they can't afford to feed the children. Woodcutter therefore dumps children in the forest. But clever children find their way back by leaving a trail of pebbles. So woodcutter dumps them in forest again. This time, with only a breadcrumb trail, they can't get home. They then get imprisoned in a gingerbread house by an old witch and... you can read the rest here.

News stories have been, up untill now, a little like Hansel and Gretel's breadcrumbs. Spread by news organizations round the web, they quickly attract an audience, but that audience gobbles them up and rarely follows the breadcrumbs back home. What if instead of breadcrumbs, journalists and news organizations dropped pebbles? That way people wouldn't eat them and there would be more chance they could lead them back home.

The difference between a breadcrumb story and a pebble story is metadata. Embed some good consistent metadata in a story and it turns something ephemeral into something much more solid. People suddenly know, for example, where it came from. A story can have the equivalent of an address and a zip code built into it, so people follow it back by whatever trail they want.

Metadata has the significant added benefits that it is visible and malleable. It can be identified and picked up by search engines and aggregators. It can then be displayed so that people have enough information to know if they want more. It's a little like seeing the front page headlines on the newsstand before deciding to put your hand in your pocket for some change. It can also be used for cross-referencing stories, for digging through the archive, for building mash-ups.

Google has an example of how, using metadata, it can display more information about a site in its "search snippets." Similarly, we (the Media Standards Trust) have been working out how to best integrate metadata in news through our Knight / MacArthur Transparency Initiative.

The great pay wall debate is not going to end anytime soon -- but needs to be a little less polarized than it has been to date. Working out how to leave a trail of pebbles would be a good start.

[For more on this debate, see an excellent two-part series, The Great Debate Over Micropayments and Paid Content at MediaShift.]

In Porto, I've spent the last couple of days at an official IPTC conference (the International Press Telecommunications Council). The folk at the IPTC have been thinking about information and metadata for over 40 years. These are the high priests of news metadata.

For a long while this was, as you'd figure, rather a minority pursuit (though mighty profitable for those that went to the trouble to do it).

Now, in our age of 'infobesity', it suddenly has significant new relevance and urgency.

Why? Because describing your content in a consistent, machine-readable way (through metadata) makes searching for it an awful lot easier. It also means you can label it so people know where it's come from. It also frees up the information so it can be used in creative, unanticipated ways (like journalisted, or dipity).

Problem is, almost all the rich IPTC metadata is stripped out before it gets to the end user. Once it has served its purpose - i.e. as a means of fast data transfer between different content businesses - the metadata is lost. By the time you and I see an article on a website we'll be lucky if it even has a date stamp (e.g. see" United Airlines story":http://www.internetbusiness.co.uk/13092008/how-google-destroyed-1-billion-of-united-airlines/ from last August).

Should you care? Well, if you want to know when and where a story was first published, yes. If you want to be able to search for stories by a specific journalist, or news organisation, then yes. If you're interested in knowing where the news you're reading has come from, then yes.

Which is why the Transparency Initiative - the MacArthur and Knight funded news project - and IPTC metadata standards, are so complementary. While the IPTC worry about labelling data at source, we're concerned with how to make sure those labels (or at least those ones that are relevant to the public) don't get lost along the way. Which is why we're hoping to work with the IPTC to see how we can retain just a little of this rich metadata and carry it all the way to you and I, the end user.

This will be in addition to the main aim of the initiative which is looking to create simple conventions for highlighting the basic provenance of a news article in a clear and consistent way - i.e. who wrote it, who first published it, when it was first written, when it was updates, where it was written from (for more see www.newscredit.org).

By learning from the IPTC's 40 odd years of experience and working with them make sure news' basic provenance doesn't disappear, we hope we can help people find news and assess it more easily - before we all get swamped by the information tsunami.

Some local areas have it worse than others. This week two Welsh politicians called Wales a 'media wasteland' where stories of public and political interest simply go unreported - despite the devolution of power to Wales a decade ago. "Since 1999", Dai Davies said, "we have seen a vast increase in powers to politicians in Wales and yet more and more journalists losing their jobs, and less and less reporting of politics and political debate and decision-making" (from BBC News Wales).

Now English politicians are also starting to become animated about the decline in reporting and lack of political coverage. Ashok Kumar, Labour MP for Middlesbrough South and East Cleveland says he is next week going to ask the government to provide state support for the regional press (from Press Gazette).

Kumar and other politicians follow a growing number of voices from within the media itself who are suggesting the government should subsidise local newspapers. Most notably Alan Rusbridger, editor of The Guardian, who wrote back in November:

"Is there any reason why local newspapers - whether in print, on broadband or broadcast - shouldn't compete with the broadcasters for some form of subsidy in return for providing the public service of keeping a community informed about itself?"

But subsidising the local press is not, IMHO, a good idea. For at least three reasons:

1. An independent commercial press would be neither independent nor commercial if it was taking hand-outs from the government. The watchdog role played by the local press would be seriously compromised were it to be state subsidised. Imagine the attitude of local councillors to reporters whose salary was partly dependent on government financing?

2. There are huge changes taking place in the way news is collected, edited, published, delivered and consumed. These changes are forcing news organisations to completely rethink how they do business. Subsidising a 20th century model will not help them rethink and reform, it will just encourage them to keep doing what they're doing

3. It would distort local editors and journalists view of who they serve. Instead of feeling - at root - responsible to the public, they would inevitably feel a degree of responsibility to the government.

This is not an argument against intervention per se. The government can set parameters - particularly fiscal parameters (i.e. tax) that incentivise people to collect and publish public interest news. But this is fundamentally different from providing a subsidy, however arm's length, that organisations can apply for.

What does that mean? It means enabling journalists, and people creating journalism, to embed basic information to their online news articles which helps the public establish an article's authorship and provenance (the same methodology applies to photos and video but I'll stick with articles right now).

Big ambition, I hear you say. You're right, it is, but also an increasingly urgent one. The accumulation of content on the web is fast becoming overwhelming. A couple of weeks back Flickr announced it had passed the 3 billion photo mark. In 2007 YouTube consumed as much capacity as the entire Internet took up in 2000.

As stuff accumulates it's becoming harder and harder to distinguish between content or to assess its credibility. What is supposed to be journalism and what isn't? When was an article first published and by whom?

This is made harder still because very few people describe what they've produced or how they've produced it, and even fewer describe it consistently. News organizations and journalists included. I'm not talking about the subjective aspects. I'm talking about the objective stuff -- the basic who, what, where, when -- who wrote it, who it was written on behalf of, when was it published -- that sort of thing.

To some people this might seem like teaching grandmothers to suck eggs. But you would be astonished at the lack of basic information out there.

Take the "recent" United Airlines story. On Sunday 7th September, at 1:32am EST a web user was browsing the Florida Sun Sentinel site. He found a story about United Airlines filing for bankruptcy. Being the only user on the site at the time, the story automatically popped into the "most read today" box on the home page.

The Google News bot, which happened to be scraping the site at the time, picked up the story and displayed it on Google News. An investment firm saw the story and posted a summary on the Bloomberg financial information service.

That morning $1 billion was wiped off the value of United Airlines. Yet the story dated not from 2008 but from 2002. The Sun Sentinel had not date-stamped the article. When it was first published was not transparent.

Enabling journalists to add more information to their content, and giving people the means to find more information out about content -- in structured and consistent ways -- could be of enormous benefit both to journalists and to the public.

So that's what we're working on. In future blog posts I'll talk more about where we are and what we've learned these past eight months (the answer to which is "lots").