The first year of work on OpenBlock Rural focused on successfully overcoming some unknown unknowns with the application. The year ahead has our attention focused on the challenges of building sustainable revenue that will support the product's editorial mission of more informed communities.

OpenBlock's End Users

Since its inception, OpenBlock has been primarily a publishing tool -- taking structured and unstructured data and plotting it on a map so that consumers could home in on news that was relevant because of its proximity. The presumed need for such a tool has been grounded in both the editorial and business mission of news organizations. Assuming the newsroom is already collecting a dataset and using it to create broad stories for a mass audience, chopping up and delivering pieces of the data to smaller but more relevant audiences is a great way of reducing waste in the news manufacturing processes. A minor vandalism may have no interest to a mass audience, but a great interest to a very few.

The challenge with using OpenBlock to sweep up and reconstitute the scraps of the mass-audience reporting process is that the data that's most accessible is not often the data that's most interesting to consumers.

Crime data -- a driver of online traffic for many local news sites -- is rarely current and rarely completely online in rural communities. Even in cities it is almost always a week old.

Education data and government salary data are also popular, but they fail to be geographically compelling and don't change very often. OpenBlock's audience wants an application that can be searched by more than just "when" and "where."

For Reporters: A Data Dashboard

With the right sets of data, OpenBlock can do a great job showing individual users a wide variety of news items that are happening near their home. But the most valuable news is reports about what's going on behind the scenes. OpenBlock can't achieve its full value in a news organization without alerting reporters to stories they might not otherwise see.

That's why we're developing a new "data dashboard" feature for OpenBlock that will allow reporters to quickly see trends across time as well as compare data from different towns or neighborhoods. This dashboard doesn't just show where the car is going, it also serves as an early warning system that will be designed to alert reporters to outliers -- the kinds of oddity or magnitude that makes big news out of small points of data.

In the first year, we've already given journalists more control over the data inside OpenBlock by building a module that more easily allows producers to see when and why data fails to geocode. Producers can then easily correct misspellings or other "dirty data" errors that previously just failed silently without notifying either journalists or the end-users of OpenBlock.

These improvements are already available on our GitHub for you to use, improve or adapt to your needs.

For Content Sponsors: An Engaged Audience

OpenBlock only works if it generates at least as much annual revenue as it costs to produce, so everything we do keeps in mind the audience of potential content sponsors.

The small, rural newspaper our project targets will never achieve the scale of audience needed to make a profit off of rapidly falling cost-per-impression online advertising. That's why we're working with The Whiteville News-Reporter to develop a sales and pricing strategy that positions OpenBlock as part of a larger cross-platform sponsorship. These sponsorships provide an opportunity to engage with customers regularly online, creatively in print and personally at topic-specific events. Each piece adds value to the others, and works best in communities such as Whiteville where the newspaper already commands industry-leading loyalty.

My colleague Penny Abernathy, the Knight Chair in Digital Media Economics here at UNC, is leading students in a project that will teach local newspaper sales staffs how explain the value of OpenBlock sponsorships to businesses whose success depends on developing a trusted and lasting relationship with rural audiences. This is one of the ways that we've been able to leverage our School's diversity among advertising, public relations and reporting to show students how to take a holistic approach to editorial product development. They've already had success with this approach to high school sports news in Whiteville.

For Audiences: Relevance and Usability

The work we did with OpenBlock in 2012 was incredibly resource intensive, and the bottom line is that unless we can make our successes in Whiteville easily replicable elsewhere we will not have a sustainable business that meets the information needs of the rural communities we set out to serve. We must lower the cost of deployment, sales training, and public records acquisition.

The cost of deployment right now is kept high by two factors: the amount of location-based customization the code still requires and the cost of creating a front-end design that matches each client's website.

Developers who specialize in Django and GIS demand some of the highest hourly rates in the industry. The less dependent installation of OpenBlock is on those high-priced skills, the more we will able to focus those resources on new feature development.

We need complete and accurate geographic data for rural communities, which we don't have. We learned something we probably should have known when we started -- that U.S. Census data is missing address ranges on about 60 percent of the road segments in rural areas. It was a good reminder of the reason we wanted to focus on rural communities -- we're amplifying a challenge that doesn't get much discussion at conferences dominated by urban innovators.

We've solved this problem in the short term by obtaining streets data from counties, but this solution isn't sustainable. It's not replicable at all in the 34 of North Carolina's 60 rural counties that don't have road data available for online download. The road data that the state Department of Transportation provides online is several years out of date and is missing major interstates around Raleigh. My informal request for more complete, current roads data was denied. That conversation with the DOT will continue in 2013.

We also need to keep working on the best way for multiple installations of OpenBlock to share a single set of data. For example, we are scraping the Secretary of State's new corporation filings. That site has data for every county, and we need to make it easy for new installations to grab only the data in which their communities are interested. But there are relatively few statewide datasets that are so easy to scrape and keep current.

Scaling the cost of installation also requires us to streamline the design of the site so that each installation isn't a complete template customization. It will be important for us to understand how much this kind of customization is worth to local papers, and how many would be willing to pay for a basic choice of color palettes and text branding.

Our data acquisition process also needs to scale. Right now, very little government data is current, complete, online and granular. Of the data that is online, much of it needs to be scraped out of forms and HTML pages. And when the structure of the forms and results changes, the scrapers must be editing. That drives up the cost of maintenance in an unpredictable way.

We are still looking at how we might crowdsource data acquisition and how we can work with local governments to lower their costs of doing business while also better serving our needs for structured data.

Scaling data also needs to happen on the presentation side. Right now, only data that contains a meaningful time and precise location can be displayed in OpenBlock. That means annual school testing data -- which is in high demand among all three of our key audiences -- doesn't look very good on OpenBlock. It maps the student-teacher ratio to the address of the school, but that data is relevant to a much wider audience than the folks who live on the same block as the school and is relevant for 364 days after it's first published. We are looking at how we can efficiently move away from OpenBlock's tyranny of the map without running up huge development costs and recreating the wheel.

The scaling of data acquisition and presentation is the important civic challenge, but it's our business of bringing together these three key audiences that continues to drive our focus.

Ryan Thornburg researches and teaches online news writing, editing, producing and reporting as an assistant professor in the School of Journalism and Mass Communication at the University of North Carolina at Chapel Hill. He has helped news organizations on four continents develop digital editorial products and use new media to hold powerful people accountable, shine light in dark places and explain a complex world. Previously, Thornburg was managing editor of USNews.com, managing editor for Congressional Quarterly's website and national/international editor for washingtonpost.com. He has a master's degree from George Washington University's Graduate School of Political Management and a bachelor's from the University of North Carolina at Chapel Hill.

In my Social Media for Reporters class at UNC, 20 percent of each student's grade will be based on the number of points that his or her Klout score goes up over the course of the semester. But the best thing about doing so is probably not that it's easy, but that it is flawed.

Boiling a semester's worth of effort and accomplishment down into a single number has always seemed to me to have a certain false sense of precision to it. More than once I've looked down at the end of the semester and wondered to myself how one student or another ended up with a grade that was so much worse -- or better -- than I would have handed out just based on gut instinct.

That's the problem that many folks seem to have with Klout and other similar social media metrics tools. Boiling continuous interaction across a variety of social networks down into a single number opens lots of room for argument -- sort of like debating whether the impact of Angel outfielder Mike Trout's 129 runs and 49 stolen bases is more deserving of the MVP award than Detroit third-baseman Miguel Cabrera's .330 batting average, 40 home runs and 139 RBI.

Boiling multiple data points from disparate contexts down to one, final judgment is often overly simplistic. But we do it.

Room for improvement

The question for me is whether we're doing it the best way possible, and how we might do it better in the future.

Klout doesn't release the algorithm it uses for calculating scores, and it doesn't disclose the distribution of its scores. The most precise piece of data they share is that the average Klout score is 40.

How is that possibly fair to students who are struggling to raise this arbitrary number that's contrived inside a black box? It's fair because it transforms the class from a workshop on button-pushing to an exercise in hypothesis testing, strategy and critical thinking. Students -- who often approach grades with calculating economy of effort -- don't know what they have to do to boost their Klout scores, so they are forced to design simple experiments, isolate variables, and generalize their findings.

We aren't totally shooting blind. Here's what we know about how Klout creates its scores:

• There are more than 400 variables in its calculation.
• New variables were added and scores were redistributed in August, just before the semester started.
• It only counts networks it can see -- so either public posts, or private posts that you've connected to Klout.
• Your Klout score is a reflection of your activity within the last 90 days.
• New Klout scores are released each morning. Older data is decayed in favor of newer data. (But Klout doesn't say at what rate data is decayed.)
• The score factors how much content you create compared to how much engagement you are receiving
• Klout says it attempts to measure engagement equally across all the networks it monitors, so that it doesn't favor activity on one network over another.
• On Twitter, Klout looks at retweets and mentions. And it is better to be re-tweeted or liked by people who do those things rarely than by people who do those things often.
• On Facebook personal profiles, Klout measures comments, wall posts and likes. Since late last month, you can have Klout measure your Facebook page instead of your personal profile. For Facebook pages, Klout measures the number of fans and how many people are talking about your page. Having a Facebook page increases scores by an average of seven points. A Klout user with a score between 70-80 has an average of 13,000 users talking about their Facebook page. (Although we don't know which is the cause and which is the effect.)
• On Google+, it measures comments, reshares, and "+1"s.
• On LinkedIn, it measures comments and likes.
• On Foursquare, it measures Todo's and Tips.
• Since August, if there is a Wikipedia page about you, Klout measures the page's rank, number of inlinks and outlinks.
• Since late last month, you get extra Klout credit if people search for your Wikipedia page on Bing. And if you appear as an expert in the "People Who Know" section of Bing's sidebar.
• Some networks -- YouTube, Instagram, Tumblr, Blogger, Wordpress.com, Last.fm and Flickr accounts -- can be connected to Klout, but don't affect your score.
• There is no reward for just adding networks that you do not participate in. Neither is there punishment.
• Adding a new account is reflected in your Klout score within 24-48 hours.

What's missing

But there's also a lot we don't know. Perhaps the most important piece of missing transparency is the "difficulty rating" that students should receive for each additional point increase in their Klout score.

Two students who had almost no social media activity when they started the semester registered initial Klout scores of 12 and 18 within the first week, but have had little movement since. But the two students who started at 55 have also seen little growth.

The most rapid growth in Klout scores during the first four weeks I've been tracking them has come from the students who had scores in the 30s and 40s. One student jumped from 33 to 52 and another from 42 to 58. But another moved only from 43 to 46.
I wanted to measure only growth that happens during the semester, so as not to punish students who started out with little or no social media experience. But what I may have ended up with is a system that punishes students who began with extensive social media engagement.

In an effort to prevent sandbagging, I'm distributing the Klout portions of their grade on a curve relative to the class. But that means all of my students could end up with a number that's in the top 10 percent of all Klout scores and still get an average grade. The only safety valve for that is my promise that I will give an "A" Klout grade to any of my students who end the semester with a score higher than mine, regardless of where they started. Right now, that bar is set at 62 -- third among my UNC colleagues, below Chris Roush and Paul Jones.

In the end, I'll add my own judgment about my students' effort and ability to use social media as reporters. I'll consider qualitative measures such as how trusted they are on their beat, whether they used it to give voice to the voiceless, hold powerful people accountable, shine light in dark places, explain our increasingly complex and interconnected world, and get the right information to the right people at the right time.

A high Klout score is something I'd expect from a solidly average student. A B student will be able to pick apart and critique Klout's system. And an A student? Someone who will one day build a better Klout.

Ryan Thornburg researches and teaches online news writing, editing, producing and reporting as an assistant professor in the School of Journalism and Mass Communication at the University of North Carolina at Chapel Hill. He has helped news organizations on four continents develop digital editorial products and use new media to hold powerful people accountable, shine light in dark places and explain a complex world. Previously, Thornburg was managing editor of USNews.com, managing editor for Congressional Quarterly's website and national/international editor for washingtonpost.com. He has a master's degree from George Washington University's Graduate School of Political Management and a bachelor's from the University of North Carolina at Chapel Hill.

Photo by Tyler Ingram on Flickr and used here with Creative Commons license.

Who do you turn to when you want to see the world as it is, rather than the world as others tell you it is? My parents would have turned on network television. Or read the Progress-Bulletin or Daily Report. I went to YouTube and searched for "PussyRiot" and watched what struck me as the video of the actions I had heard about second- and third-hand. The video, I thought, was edited in such a way that made both the church and the band look like victims, depending on your point of view. To me, that was a good indication of its authenticity.

But I don't really know, and I trust sources like the New York Times, and especially its reporters on the ground in Moscow, to tell me whether what I'm really seeing is accurate. So I next went to nytimes.com and its story. The Times had links to videos. But a quick look around the other five top news sites in the U.S showed that it was the only popular publication that linked to the videos of the band's action that landed it in prison for three months while awaiting trial. So why was the Times the only source to have linked to the video? And what does that news organization's unusual behavior mean?

a lack of links

The other sites -- Yahoo News, Huffington Post, ABC News, NBC News and USA Today -- failed me. These are sites that are both praised and vilified as "aggregators" or "MSM." But all made the same editorial decision -- and didn't help their audience see the key fact of this case for itself.

But I wonder why the link wasn't made? The people who work there are professionals. And I have no reason to believe they are more or less immoral than I am.

Going back more than a decade, academic studies have found that few news stories actually link to source information. In 2001, one in 23 stories about the Timothy McVeigh execution linked to external sources. And a 2010 study indicates that U.S. journalists are less inclined to link to foreign sources than domestic sources, with fewer than 1 percent of foreign new stories on U.S. news sites containing links in their stories.

So, why?

Two prominent academic studies seem to indicate that the presence of inbound and outbound links increase credibility in both professional and amateur sites. Are professional journalists unaware of those studies? Are they aware, but think they're bunk?

One study indicates that journalists don't link because they are concerned about the financial implications -- that users who leave the site will not return to drive up ad impressions. Another seems to indicate that U.S. journalists are particularly skeptical of foreign sources of news because they are less confident of their own ability to judge the credibility of foreign sources.

enhancing credibility

From my experience in online newsrooms, both those findings seem plausible. But they also seem incomplete. My own additional hypothesis is that hyperlinking has been left primarily to automation and that editors and reporters who've been asked for the last decade to "do more with less" have decided that links to original source material -- which, at least according to a few studies, enhance their credibility, are not worth their time.

But other studies have shown that hyperlinks in the text of a story distract readers -- even the small percentage of readers who click on the links -- and reduce reading comprehension. That said, I suspect the journalists who didn't include links to the Pussy Riots videos are completely unaware of such studies (which are summarized nicely throughout Nicholas Carr's book "The Shallows."

If there's credit to be given in The New York Times' decision to include the links in the story, then it goes to the reporter in Moscow, David Herzenhorn, according to three sources who work at the Times. The role that Herzenhorn played is important. This was a task not left to an editor or producer in New York, but one that the Moscow correspondent took upon himself. The links add to his credibility.

"I have to say I am completely floored that other news organizations would not link to the videos, since they explain so much about the story," Kyle Crichton, the editor who worked on the story, wrote to me in response to an email query.

My rather slack Friday afternoon efforts to obtain comment from other news organizations that didn't link to the videos yielded no responses. I still hope to hear from them in hopes of understanding whether the lack of links was merely an oversight or a conscious omission. Herzenhorn also did not reply to my email on late Friday.

The reporter -- and at this point he, rather than his employer, deserves credit for the links -- selected the more popular Russian-language versions on YouTube rather than the English subtitled versions, which had fewer views but would be more useful to the Times' English-language audience.

"There is some profanity on the soundtrack, so I presume that is why David chose not to include [the videos with English subtitles]," Crichton said in his email to me. "That strikes me as fair, since the text isn't as important as the overall spectacle of their 'performance.'"

the political impact of linking

I also wondered what the political impact of including such links might be. I've had
newsroom conversations about whether linking to a source constitutes endorsement. The modern version of this is manifested in newsroom social media policies that discourage journalists from re-tweeting information from sources and in Twitter bios that say "RT ≠ endorsement."

I teach my students, and write in Chapter 7 of "Producing Online News," that links in a story are akin to quotes. You're responsible for the facts of the source's statement, but not the opinions. And stories without links today seem as incomplete as stories without quotes from named sources have always been.

In foreign stories, though, links to banned material could have an effect on both the news
organization's ability to distribute news and on its reporters' ability to collect it. Crichton wasn't concerned.

"I don't think our including the videos will have any impact on our future ability to report in Russia," Crichton said in his email to me. "If it were Iran, maybe, but Russia isn't like that, yet."

What discussion to you have in your newsroom about including or excluding links? If you aren't having any, consider consulting with -- and funding -- the mass communication researchers who can help you make your journalism more credible, more memorable and more useful.

Related links:

• What Aggregators Do: Rhetoric, Practice, and Evidentiary Cultures Inside Web-Era Journalism
• Jurisdictional protectionism in online news: American journalists and their perceptions of hyperlinks
• Exploring Online News Credibility: The Relative Influence of Traditional and Technological Factors
• The International Network Structure of News Media: An Analysis of Hyperlinks Usage in News Websites
• Enhancing Perceived Credibility of Citizen Journalism Web Sites
• Hyperlinking as Gatekeeping: online newspaper coverage of the execution of an American terrorist

Ryan Thornburg researches and teaches online news writing, editing, producing and reporting as an assistant professor in the School of Journalism and Mass Communication at the University of North Carolina at Chapel Hill. He has helped news organizations on four continents develop digital editorial products and use new media to hold powerful people accountable, shine light in dark places and explain a complex world. Previously, Thornburg was managing editor of USNews.com, managing editor for Congressional Quarterly's website and national/international editor for washingtonpost.com. He has a master's degree from George Washington University's Graduate School of Political Management and a bachelor's from the University of North Carolina at Chapel Hill.

Or, if you're like many journalistically and technically savvy people I've spoken over the last few months, you could ask why would they? There's plenty of evidence out there to suggest the OpenBlock application is essentially a great experiment and proof of concept, but a dud as a useful tool for journalists. Many of the public records portions of Everyblock.com -- OpenBlock's commercial iteration -- are months if not years out of date. It can't be found anywhere on the public sites of the two news organizations in which the Knight Foundation invested $223,625. There are only three sites running the open-source code -- two of those are at universities and only one of which was created without funding from the Knight Foundation.

And, you, Thornburg. You don't have a site up and running yet, either.

All excellent points, dear friends. OpenBlock has its problems -- it doesn't work well in multi-city installations, some search functions don't work as you'd expect, there's no easy way to correct incorrect geocoding or even identify possible failures, among other obstacles that I'll describe in greater detail in a later blog post. But the alternatives also have shortcomings. And deciding whether to use OpenBlock depends on which shortcomings will be more tolerable to your journalists, advertisers and readers.

SHOULD I USE OPENBLOCK?

If you want to publish news from multiple cities or from unincorporated areas, or if you serve a rural community I'd hold off for now. If you visit our public repositories on GitHub you can see the good work the developers at Caktus have been doing to remove these limitations, and I'm proud to say that we have a private staging site that's up and running for our initial partner site. But until we make the set-up process easier, you're going to have to hire a Django developer (at anywhere from $48,000 a year to $150 an hour) to customize the site with your logo, your geographic data, and your news items.

The other limitation to OpenBlock right now is that it isn't going to be cheap to maintain once you do get it up and running. The next priority for me is to make the application scale better to multiple installations and therefore lower the maintenance costs. Within the small OpenBlock community, there's debate about how large of a server it requires. The very good developers at OpenPlans who did a lot of heavy lifting on the code between the time it was open sourced and the time that it should run nicely on a "micro" instance of Amazon's EC2 cloud hosting service -- about $180 a year.

But we and Tim Shedor, the University of Kansas student who built LarryvilleKU, find OpenBlock a little too memory intensive for the "micro" instance. We're on an Amazon Web Services "small" instance, and LarryvilleKU is on a similar sized virtual server at MediaTemple. That costs more like $720 a year. And if you add a staging server to make sure your code changes break in private instead of public, you're looking at hosting costs of nearly $1,500 a year.

And that's before your scrapers start breaking. Depending on how conservative you are, you'll want to set aside a budget for fixing each scraper somewhere between one and three times a year. Each fix might be an hour or maybe up to 12 hours of work for a Python programmer (or the good folks at ScraperWiki). If you have three data sources -- arrests, restaurant inspections and home sales, let's say -- then you may get away with a $300 annual scraper maintenance cost, or it may set you back as much as $15,000 a year.

I've got some ideas on how to reduce those scraper costs, too, but more on that later as well.

Of course, if you have someone on staff who does Python programming and whose done some work with public records and published a few Django sites and they've got time to spare, then your costs will go down significantly.

But just in case you don't have such a person on staff or aren't ready to make this kind of investment, what are your alternatives?

GOOGLE MAPS AND FUSION TABLES

Using a Google Map on your news website is a little like playing the saxophone. It's probably the easiest instrument to learn how to play poorly, but pretty difficult to make it really sing. Anyone can create a Google Map of homicides or parking garages or whatever, but it's going to be a static map of only one schema, and it won't be searchable or sortable.

On the other hand, you can also use Google Maps and Fusion Tables to build some really amazing applications, like the ones you might see in The Guardian or on The Texas Tribune or WNYC or The Bay Citizen. You can do all this, but it also takes some coding effort and probably a bit more regular hand care and feeding to keep the site up-to-date.

I've taken a look at how you might use Google's data tools to replicate something like OpenBlock, although I've not actually done it. If you want to give it a whirl and report back, here's my recipe.

A RECIPE FOR REPLICATING OPENBLOCK

Step 1. Create one Google Docs spreadsheet for each schema, up to a maximum of four spreadsheets. And create one Google Fusion Table for each scheme, up to a maximum of four tables.

Step 2. If the data you want is in a CSV file that's been published to the web, you can populate it with a Google Docs function called ImportData. This function -- as well as its sister functions ImportHTML and ImportXML -- will only update 50 records a time. And I believe this function will pull in new data from the CSV about once an hour. I don't know whether it will append the new rows or overwrite them, or what it would do if only a few of the fields in a record change. If you're really lucky, the data would be in an RSS feed and you could use the ImportFeed function to get past this 50-record limit.

Of course, in the real world almost none of your data will be in these formats. None of mine are. And in that case, you'd have to either re-enter the data into Google Docs by hand or use something like ScraperWiki to scrape a datasource and present it as a CSV or a feed.

Step 3. Use a modification of this script to automatically pull the data -- including updates -- from the Google Docs spreadsheet into the corresponding Fusion table you created for that schema.

Step 4. Find the U.S. Census or local county shapefiles for any geographies you want -- such as ZIP codes or cities or school districts -- and convert them to KML.

Step 5. Upload that geographic information into another Fusion Table.

Step 6. Merge the the Fusion table from Step 3 with the Fusion table from Step 5.

Step 7. This is really a thousand little steps, each depending on which of OpenBlock's user interface features you'd like to replicate. And, really, it should be preceded by step 6a -- learn JavaScript, SQL, CSS and HTML. Once you've done that, you can build tools so that users can:

• Use filters to select different points to appear on the map. (Examples: https://developers.google.com/fusiontables/docs/samples/change_query and http://googlegeodevelopers.blogspot.com/2011/05/turn-up-power-of-your-fusion-tables.html)
• Display points from different tables -- known in OpenBlock parlance as "NewsItem types" -- on a single Google Map (up to five). (Tutorial: http://michelleminkoff.com/2011/08/21/how-to-combine-multiple-fusion-tables-into-one-map/)
• Have maps and tables interact with each other, so that filtering by one shows information in another. (Example: http://www.texastribune.org/library/data/redistricting-viewer/)

And there's even at least one prototype of using server-side scripting and Google's APIs to build a relatively full-functioning GIS-type web application: https://github.com/odi86/GFTPrototype

After all that, you will have some of the features of OpenBlock, but not others.

Some key OpenBlock features you can replicate with Google Maps and Fusion Tables:

• Filter by date, street, city, ZIP code or any other field you choose. Fusion Tables is actually a much better interface for searching and filtering -- or doing any kind of reporting work -- than OpenBlock.
• Show up to four different kinds of news items on one map (five if you don't include a geography layer).
• Conduct proximity searches. "Show me crimes reported within 1 mile of a specific address."

WHAT YOU CAN'T REPLICATE

The OpenBlock features you can't replicate with Google:

• Use a data source that is anything other than an RSS feed, HTML table, CSV or TSV. That's right, no XLS files unless you manually import them.
• Use a data source for which you need to combine two CSV files before import. This is the case with our property transactions and restaurant inspections.
• Update more than 50 records at a time. Definitely a problem for police reports in all but the smallest towns.
• Use a data source that doesn't store the entire address in a single field. That's a problem for all the records with which we're working.
• Map more than 100,000 rows in any one Fusion table. In rural counties, this probably wouldn't be a concern. In Columbus County, N.C., there are only 45,000 parcels of land and 9,000 incidents and arrests a year.
• Use data sources that are larger than 20MB or 400,000 cells. I don't anticipate this would be a problem for any dataset in any county we're working.
• Plot more than 2,500 records a day on a map. Don't anticipate hitting this limit either, especially after the initial upload of data.
• Parse text for an address -- so you can't map news articles, for example.
• Filter to the block level. If Main Street runs for miles through several miles, you're not going to be able to narrow your search to anything relevant.
• Create a custom RSS feed, or email alert.

THE SEO ADVANTAGE

And there's one final feature of OpenBlock that you can't replicate using Google tools without investing a good deal of manual, rote set-up work -- taking advantage of SEO or social media sharing by having a unique URL for a particular geography or news item type. Ideally, if someone searches for "home sales in 27514" I want them to come to my site. And if someone wants to post to Facebook a link to a particular restaurant that was scolded for having an employee with a finger-licking tendency (true story), I'd want them to be able to link directly to that specific inspection incident without forcing their friends to hunt through a bunch of irrelevant 100 scores.

To replicate OpenBlock's URL structure using Google Maps and Fusion Tables, you'd have to create a unique web page and a unique Google map for each city and ZIP code. The geography pages would display a polygon of the selected geography, whether it's a ZIP code or city or anything else, and all of the news items for that geography (up to four schemas, such as arrests, incidents, property sales, and restaurant inspections). That's 55 map pages.

Then you'd have to create a map and a page for each news item type. That's four pages, four Fusion tables, and four Google Docs spreadsheets.

Whew. I'm going to stick with our work in improving the flexibility and scalability of OpenBlock. But it's still worth looking at Google Maps and Fusion Tables for some small and static data use cases. Other tools such as Socrata's Open Data, Caspio and Tableau Public are also worth your time as you begin to think about publishing public data. Each of those have some maintenance costs and their own strengths and weaknesses, but the real trick for using all of these tools is public data that isn't in any usable format. We're looking hard at solving that problem with a combination of scraping and crowdsourcing, and I'll report what we've found in an upcoming post.

Ryan Thornburg researches and teaches online news writing, editing, producing and reporting as an assistant professor in the School of Journalism and Mass Communication at the University of North Carolina at Chapel Hill. He has helped news organizations on four continents develop digital editorial products and use new media to hold powerful people accountable, shine light in dark places and explain a complex world. Previously, Thornburg was managing editor of USNews.com, managing editor for Congressional Quarterly's website and national/international editor for washingtonpost.com. He has a master's degree from George Washington University's Graduate School of Political Management and a bachelor's from the University of North Carolina at Chapel Hill.

Which creates an interesting problem for copy editors and AP style sticklers alike -- how do you ensure that, in a county with 38,000 addresses, each shows up on the map "close enough?"

For several months earlier this year I struggled with increasing our precision and automation of mapping addresses using OpenBlock in rural areas. We've made several technical adjustments to the application's code, but we're now at the point where spending more time with the code will be more expensive than turning the remaining errors over to a human to fix one-by-one.

What's "close enough" for us? Well, when an address of a news event goes into OpenBlock it will fail 5-25 percent of the time, depending on the source of the data. So the question is not whether we can weed out all errors -- as it is in my news writing class -- but to make sure that when addresses fail they do so in the right way.

hit rate and precision

Geocoding accuracy has two components -- the hit rate and precision. The hit rate measures how often you type an address and get back any sort of location on a map. Google Maps, for example, has a very high hit rate. But it's not always accurate. In fact, if you're looking for the Burger King in Chadbourn, N.C., Google will erroneously send you across the state line into South Carolina.

So that's a hit, but not a very precise one. Google puts the Burger King in another state because it uses its address to guess where it might fall along a very long road. OpenBlock won't guess like that much, so it has a lower hit rate. And we want any address that fails to end up in the hands of an editor -- so that she or he can check the problem and correct it.

The higher the hit rate, the more editors have to be concerned with precision. This isn't a topic I've not heard widely discussed at conferences or in newsrooms. I suspect that if they are anything like I was a few months ago, most editors are blissfully unaware of just how imprecise their geocoding and how many unchecked fact errors might be surfacing on their sites.

We checked our address precision for the pilot installation of OpenBlock Rural by getting a shapefile from the Columbus County's GIS office that provides latitude and longitude coordinates for all addresses in the county. We then took the list of the address from that file and ran them through OpenBlock's geocoder, which gave us another set of latitude and longitude coordinates. We ran the two sets of points through a tool for calculating the distance.

With that dataset we found, for example, that the point at which the county says is "235 Hammond Drive" is about 3 miles away from the point that OpenBlock's geocoder thinks is "235 Hammond Drive." OpenBlock puts that address on a map, but not with very much precision.

I'll come back to Hammond Drive in a bit, but first want to describe some of the other ways we've tried to determine whether our automatic-F-in-intro-newswriting level of accuracy is tolerable.

Of our more than 38,000 addresses ...

• Only 9 percent of the points are within a foot or less of the point given by the county for that address. That doesn't seem very good.
• Garmin GPS advertises accuracy within about 50 feet. Only 16 percent of our geocoding is that accurate. Our audience isn't driving around, so maybe we can afford to be less precise than that.
• Google's point for "235 Hammond Drive" is 482 feet away from the front doorstep of the house at that address. About 94 percent of our addresses are at least that close to the points at which the county places them.
• Our median error was 118 feet, which is better than the median error of 127 feet found in a similar test of rural addresses using the Gold Standard of geocoding with ESRI's ArcGIS software.

So I'm not going to fuss over each address like I would over each fact in a story. Using these industry benchmarks we're going to triage the most imprecise addresses and worry first about those.

And all errors won't be created equal. Going back to our "235 Hammond Drive" example, a user who searches for that address would get two results: "235 Hammond St., Fair Bluff, NC 28439" and "235 Hammond, Cerro Gordo environs, NC 28439."

For the first problem, you might quickly point out that the search term itself is ambiguous -- why wouldn't someone simply search again and include the city this time? The bottom line is we have no data on how often people search on street alone, as opposed to other permutations of the complete address. A good copy editor should be able to work with developers to produce a test that would yield us that data.

The other difficulty -- and this is going to be trouble more often in rural areas than urban -- is that the second address is outside the Fair Bluff city limits, but has a Fair Bluff mailing address. So the most likely alternate search that a user would perform seems like it would be "235 Hammond Drive, Fair Bluff, NC, 28439." That search would yield only the first result shown here. And it would be the wrong one.

So copy editors who work with online maps have to have enough local knowledge so that when they look at a map of 235 Hammond Drive, they know it's out in the country and not downtown.

the copy editor/debugger

Copy editors also have to be good debuggers. Again, they have to think about breaking the problem down into testable parts. The first problem is that deep in the OpenBlock code, a human made the correct assumption that sometimes people make mistakes -- and that when they type "Hammond Drive" they really might mean "Hammond Street." What in this case looks like a bug is often a feature. Fix this problem and you'll break a solution.

The other possible problem is that the underlying geographic data is incorrect. The Census Bureau's TIGER/Line Shapefiles are the most common source for geographic data, but in rural areas the Census omits many address ranges for street segments. In this particular case, it had "Hammond Drive" in the wrong ZIP code. The true ZIP code is 28430.

So instead of using Census data -- which is a key assumption of OpenBlock -- we're now using a shapefile given to us by Columbus County -- but only 49 of North Carolina's 100 counties provide such information on their websites. This local data source improves the completeness and accuracy of our underlying geographic data, but will make it more expensive to deploy OpenBlock. We won't be able to rely on a standardized script to import geographic data if each county's file format is different.

Geocoding in OpenBlock has been a good warning to me, and a reminder to newsrooms everywhere that the copy editor/debugger is going to be at least as valuable as the reporter/programmer. And that means I need to figure out how to teach that skill to my students at UNC.

How does your newsroom provide quality control for its online maps? What's your hit rate? And precision? Share your experiences in the comments here or by tweeting to @OpenRural.

I've just finished teaching a course that was, in part, inspired by the lack of young journalists who have these skills. And, as is almost always the case, I learned at least as much as the students -- mostly that teaching the tools and techniques of data-driven accountability reporting takes much more than a semester.

The idea for the class began more than a year ago during conversations with Sarah Cohen -- my former Washington Post colleague and current Knight Chair in Computational Journalism at Duke University -- to try an unheard-of experiment -- teach a class that would include both Duke and University of North Carolina students and that would provide a soup-to-nuts workshop in investigative reporting and digital publishing. How do you find, report, produce and distribute information that's so tough to do and so valuable that nobody else would do it for free and that everyone would pay to not just know the information in your report -- but experience and engage in it?

Looking for model classes that our friends were teaching elsewhere, we found many classes dedicated to production and digital publishing -- especially visual communication -- and a few that focused on conceptualizing investigative story ideas. But as we talked, our conversations kept turning back to the blurring of lines in many newsrooms between web production and computer-assisted reporting. We wanted our students to be able to data crunch in a way that would make it easy to data viz.

find a story, not a report

Our class met once a week for 16 weeks -- first at Duke University, then at UNC. The first half focused on basic reporting tools and techniques to background both an individual and a corporation. We introduced TimeFlow and DocumentCloud, as well as how to find story ideas in digital public records and newspaper archives. Our mantra: Find a story, not a report. Both have data, to be sure, but stories have characters, movement, conflict, and heroism and villainy -- sometimes in the same person. And news stories must have impact.

The 11 students came up with probably four good story ideas that we thought we could pursue. So the second half of the semester was turned to story reporting and production. During the second half, we simply tried to do too much -- introduce them to Google Fusion Tables, Tableau Public, OutWit and web scraping with Python, Excel and a little Access -- all while writing public records requests and interviewing principals in an effort to better understand the work of a $4.6 billion statewide charity funded with half of North Carolina's take in the settlement with the tobacco industry.

A story like this would likely take a team of pros six to 18 months to do well, including interactive and multimedia features. We tried to do it in eight weeks. So, as we suspected going into it, there was just too much ground to cover. It was an experiment, and here's what I learned.

key takeaways

• As a student suggested after it was over, this class really needs to be three semesters long. In her ideal world, she said, she'd spend a semester on data tools, such as Excel, Access, web scraping and GIS; another semester on reading and watching great investigative stories and talking to the journalists who produced them; and a final semester on reporting and production. She said that we should force every journalism student to at least learn the data-reporting tools. I tend to agree with her points.
• Students want to focus on the tools more than the data, when their priorities should be reverse. But it's easier on the instructor to go get the dataset, and then simply give students a recipe for managing it. I'll have to find a way to manage those expectations and the workload.
• If getting the kind of long-term commitment and teamwork required for multimedia investigative projects is tough in a newsroom, it's twice as tough in an undergraduate classroom. Many of the students haven't met each other before the class; they know they're leaving "the job" in three months and are spending -- at most -- nine hours a week on the project. Individual projects are better during the confines of a semester. Group projects work great over a one- or two-year extracurricular commitment.

I hope to tackle a similar class in the future. And while I'd love to integrate OpenBlock directly into the class, the application is really suited now as a publishing tool. It lacks a native way to use it for data analysis and story idea generation, but I could see it integrating into a newsroom/classroom suite of tools that might include ScraperWiki, DocumentCloud and PANDA. I wonder if one day the Knight News Challenge will yield the Knight News Curriculum.

Image courtesy of Flickr user Tyler Ingram and used here under the Creative Commons license.

At the end of last year, Raju Narisetti proposed a pay wall alternative he dubbed the "'Why don't we pay you?' pay wall" ... and then left the unwalled Washington Post for the walled Wall Street Journal.

The conversation all this time has been focused on whether the shift toward digital subscriptions will save the news business. But the more interesting and important question is whether and how it will change the news content and public discourse.

There's never been a question that people will pay for digital content. Give people information they need to profit professionally or enjoy personally, and they will pay for it. But what about all the boring and bad stuff? What about the kind of iron-butt reporting that has journalists cover legislative subcommittee meetings just so powerful people know the public is watching? And the quarter million-dollar investigations that find the hidden winners and losers?

That news doesn't entertain; it doesn't give me a competitive edge; and it doesn't save my family money in the short run. Those kind of stories make big waves every now and again, but no matter how high the pay wall, once the story is out, it spreads via broadcast news, social media and word of mouth. Even those who don't pay for it get to benefit from its impact.

social media's role

The role that social media plays in the subscription pay model isn't fully understood -- by me at least. I'd like to find the time to ask about whether paying subscribers share more or different stories than non-subscribers.

In any case, with a pay wall in place, subscribers will -- as always -- set the agenda more than non-subscribers. Some subscribers will be more influential than others, either because they have more followers or because they provide a better filter. In either case, the future of public discourse lies with subscribers. We need to know more about who they are and how their desired public agenda differs from non-subscribers.

It's easy to suspect that only the elite would pay for news -- only people whose personal social and economic decisions are determined by taxpayer money and public markets -- and that the topics that interest those folks may not be particularly populist.

But then I stumbled across a January 2011 survey by the Pew Research Center that seems to indicate that the willingness to pay for news may not be as elitist as I originally thought: African Americans and Hispanics are significantly more likely than whites to say that they would pay a monthly subscription fee if that was the only way to get full access to their local newspaper online. But there's no significant difference among any age groups under 65, nor is there a difference between men and women. On the other hand, college grads and people who make more than $75,000 a year are more likely to say they would pay for online local news than people who make less and have less education.

So does the public discourse look different if the people who subsidize original reporting -- and then share it -- are rich, educated, racial and ethnic minorities? After paying to see the news, what would they share? And who would they share it with?

the social distribution of news

The democratization of publishing means that alternative points of view would always be waiting in the on-deck circle anytime the paid-stream media misses a story its audience cares about. So it's also important to predict what kind of effect the audience's sharing patterns would have on journalists who want to make sure their pay walled reports remain valuable enough to make ends meet.

The social distribution of news has two benefits for news organizations -- they sell advertising against each unique visitor, and they have an opportunity to convert the social media samplers into paying subscribers. But if the role of advertising at news organizations becomes a significantly lower share of revenue, then eyeballs alone won't matter as much. News organizations might be less interested in running "water cooler" stories that are cute and fun alone. And they might be more inclined to run stories that target an audience that wants more than 140-character summaries.

Research collaborations between academics and industry could help us make better guesses -- and making good guesses on this topic will be important for any news organization that understands it doesn't sell ads or subscriptions, but trust and influence.

Image courtesy of Flickr user Aunty P.

The panel was organized by Fiona Morgan, a researcher at the DeWitt Wallace Center for Media and Democracy at Duke University, who worries that as newspaper companies try to harvest more revenue from a shrinking audience, they are catering both content and delivery to a wealthy, educated, white audience. She asked me to join the panel to discuss how the old idea of the "penny press," which revolutionized journalism by covering news that appealed to a broader audience, might be updated from the digital age.

In rural communities, newspapers actually don't have much choice but to serve an audience that has less money and education on average than a typical newspaper reader. For example, in Columbus County, N.C., The News Reporter can't ignore people who have little "spending power" because there are simply too many of them. Of about 22,000 households in the county, 38 percent have incomes of less than $25,000 a year, according to the Census Bureau' s American Community Survey. Only 19 percent have incomes above $75,000. Compare that to Wake County, where the numbers are reversed -- only 18 percent of households live on less than $25,000 a year and 45 percent make more than $75,000.

But poverty in Columbus County doesn't translate into a lack of interest in the news (although it does translate into lower voter turnout). The twice-weekly, family owned News Reporter in Whiteville reaches just more than half the households in the county. The News & Observer, the biggest paper in Wake County, reaches only about 20 percent of its home-county households.

the threat of shifting revenues

With market strength like that, it looks at first blush like the best way to recreate the penny press for the digital age is to print on dead trees. But that ignores threats just over the horizon. Right now, the average visitor to whiteville.com spends just over four minutes per visit, compared with the 20 minutes that the average visitor spends on each trip to Facebook. Based on Census data and survey research done by the Pew Research Center, there's a good bet that there are just as many people in Columbus County on Facebook as there are readers of the print newspaper.

Poor or not, the people of Columbus County use the Internet. Even though we know that income has a positive correlation with newspaper and online news consumption, there are just so many poor people in Columbus County that they dominate every media measurement. When looking at raw numbers, there's probably about 25 percent more very poor people on Facebook in Columbus County than very rich. And on Twitter, the raw numbers of very poor and very rich are likely about even in Columbus County.

That's important because the business staff at The News Reporter has just within the last few months begun to see signs that people are not putting their classified ads in the newspaper because they are posting items for sale on Facebook -- a company whose mission isn't to support accountability journalism.

As big city newspaper companies can tell you, a future of shifting revenues means a future in which you have to either live with smaller margins or spend less on reporting and other expenses.

The News Reporter is a family owned company with a Pulitzer Prize in its history. I'd bet it spends a higher percentage of its revenue on reporting and editing than most newspapers. The challenge for rural newspapers, though, is the economy of scale. They simply can't divide the cost of covering local government among as many readers. If The News & Observer sends someone to the Wake County courthouse, the cost of that reporting is shared among 70,000 readers.

But if The News Reporter sends someone to the Columbus County courthouse, it can only split the bill about 11,000 ways. So, per capita, it's nearly seven times as expensive for Columbus County readers to get local news coverage as it is for Wake County residents.

where openrural comes in

OpenRural aims to lower the cost of gathering and publishing basic data about government and public life. Property sales, arrest reports, new business openings and restaurant inspections have long been a staple of community newspapers. But until now, publishing them has required a reporter to go down to a county office, pick up a piece of paper, and re-type the information into her newspaper's publishing system. We aim to automate as much of that as possible.

Lowering costs and serving an audience across all demographics, OpenBlock appears to meet all the requirements for itself being a penny press for the digital age. But generating cheap content doesn't solve the revenue problem in a world of abundance and bad competitors who are willing to provide a similar or better service for even lower margins.

The only way that OpenBlock -- or any penny press in the digital age -- is going to solve revenue woes is by increasing audience loyalty in both print and online.

If OpenBlock lowers the cost of collecting and publishing commodity news in rural markets and staves off some bad competitors, then the next step will be for publishers to reinvest the savings into high-quality, high-impact public affairs reporting. Reporters who once gathered paper and went to meetings will need to do more stories about the "how" and the "why" rather than simply the who, what, when and where. For these rural communities to lift themselves out of poverty, they need to be able to look at trends in the data.

In his book "The Vanishing Newspaper," Philip Meyer looks at how newspapers use reporting to create trust and influence in their communities and how they then sell that influence to advertisers. So the secret to building a penny press for the digital age isn't just about generating abundance of content any more than it's about catering to an elite audience. Page views without impact have no value.

And that's why Meyer also said in another book, "Precision Journalism," that the ante is being raised on what it takes to be a journalist. Data can't just be an input in a low-cost publishing process. It must also be the raw material that reporters are able to use for analytical and accountability journalism.

Newspaper image courtesy of Flickr user Binuri Ranasinghe.

Over the course of the next several posts, I'm going to give you an X-ray view into the guts of the OpenBlock application. Together, we're going to watch how data gets ingested and processed into information and insights that residents of rural communities can use to make decisions about their daily lives.

We knew basically two things when we started this project. First, we knew that public data went into OpenBlock. And we knew that digital public data is for the most part in a poor condition to be easily digested. Second, we knew that the meaningful geographies of rural areas were going to be different than the geographies of urban areas. But beyond that, the anatomy of OpenBlock remained almost completely obscured. No instances of the application lived in the wild, and the code itself was still missing significant documentation.

Step 1: Getting data into OpenBlock

Our first step was to figure out how to get data into OpenBlock. And while many of us probably think about data as being some sort of news event -- a transaction, creation, deletion, inspection, election, rejection, incarceration or some other function of government that takes place at a specific time -- the initial data you need for OpenBlock is about a location. You have to tell it where it is. And you do that by ingesting data from the U.S. Census Bureau.

The Census Bureau is an amazing resource of geographic data, which it calls TIGER/Line files. The bureau provides TIGER/Line files that show information about various "layers" of geography -- most of which aren't relevant to OpenBlock. Each layer actually consists of several files that you download from the Census website as a single zip file. And inside that zip file is a file with a .shp extension -- that's a shapefile, and it is the only one that has nutritious value as far as OpenBlock is concerned.

Several layers are important -- one file contains data about the boundaries of all ZIP codes in a state. Even though most ZIP codes aren't going to be relevant, you need them anyway. And while loading ZIP code shapefiles into OpenBlock may not be simple for people who don't have at least some familiarity with Django, it doesn't require a lot of human judgment. Plug in some code and you're done.

But nobody lives their lives by ZIP codes. We care about geographies like counties and cities and streets. And knowing which geographic data to load next does require some editorial thinking. Most of us are familiar with counties as political entities that have some meaning. But counties are different in each state. For example, in Virginia there is a City of Fairfax and County of Fairfax. Fairfax City isn't in Fairfax County or any county. They touch each other, but one does not have jurisdiction over the other. Now, here in North Carolina we have Durham County and Durham City. Durham City is inside Durham County -- mostly -- except for the part of Durham City that is in Orange County. Orange County is the home of Chapel Hill, except for the part of Chapel Hill that's in Durham County. And both Durham and Orange counties contain large parts of land that aren't in any city at all.

teaching journalism to programmers and vice versa

So when we talk about teaching journalism to programmers and programming to journalists -- this is really the kind of thing we're talking about. Somewhere here we have to have someone who knows the political geography of North Carolina and who can also describe the rules of that geography to a computer program so that it doesn't leave out anything it shouldn't and also includes everything it should.

Let's say we want to show the locations of all new business that have been incorporated in Orange County. The N.C. Secretary of State, which records new businesses, may have the address of the business, but not the county. We're going to have to tell the OpenBlock application which addresses are inside Orange County -- regardless of whether they are in Chapel Hill, Durham City or no city. If we tell it just to grab the Chapel Hill addresses, we will erroneously include the businesses that are in the part of Chapel Hill that's in Durham County. Or, since most people think of Chapel Hill as being an Orange County city and they might be confused if they know a business has opened but isn't listed on our website, we may want to tell OpenBlock to include all Orange County addresses, but not the ones that are in Durham City, and also include the Durham County addresses if they are in Chapel Hill.

And then we have to write for the reader an explanation of whatever assumptions we're making in our data, and it has to be brief and clear.

Once you've made your editorial decisions, county bounders can also be downloaded from the Census Bureau as well. Just to be proper, Census calls this layer "County and Equivalent."

Cities, however, are a more delicate matter that again require some knowledge of Census terminology and local political geography. There is no Census layer called "cities."

Until recently, the practice among the OpenBlock community had been to look for geographic information about city boundaries from local governments. Most county governments are pretty good about publishing their geographic data on the web. Many large public universities have a GIS (geographic information system) section in their library, such as this one at N.C. State University that indexes links to the geographic data source for many of the state's county governments.

lesson learned: local data varies

The problem with local data is that poor and rural counties are less likely to have the online GIS data. And you will also find yourself dealing with a wide variety of standards. We spent a few days flummoxed by our inability to load Columbus County data. For whatever reason, the county decided not to include the necessary .prj projection file you need to make a shapefile work in OpenBlock. It's a lesson that's going to be important for us to remember throughout this project -- local data varies widely in quality. And sometimes it's not obvious to the layperson's eye what is missing.

Our thinking right now is that we're going to be able to turn back to the Census Bureau for city information. But we're not using the Census layer called "Consolidated Cities." Nor the one called "County subdivisions." We're using the "Places" layer.

You can read more about our "city" solution on the OpenBlock discussion group. But there are a few bits worth mentioning here:

• Places do not cover 100 percent of a county. So we're going to need to be on guard that events happening outside the boundaries of one of our places doesn't somehow get left out of the search results.
• Some of the places in Columbus County probably have little or no meaning to the audience there. In the "Government" navigation of our partner site, Whiteville.com, not all of the Census places are listed. That said, several of them show up occasionally as the location of obituaries that run on the site.

Geography is hardly the dynamic data we think of as news. But we've already seen several road bumps that are big enough to deter almost all small news organizations from using the OpenBlock application. So post this as one of our goals: to automate the process of loading basic ZIP, county and city geographic data about your community. Again, you can read more in the OpenBlock discussion group about how we might use FIPS codes -- the unique number given by the Census Bureau to each state, county and "place" -- to do that.

And there's still a big elephant in the room -- the geography of "blocks" that is one of the core concepts of OpenBlock. That topic is big enough for its own post a few weeks from now. Before then, I'm going to walk you through some of our experiences loading "news" data into OpenBlock and how we're hoping we might be able to work with fellow Knight News Challenge winner ScraperWiki to build our own API for North Carolina state government.

Understanding each step of the set-up, production and editing process involved with OpenBlock is critical to our ability to describe the expense side of the equation, which we hope and expect will lead to the financial viability of the application as a tool to fill the information needs of rural communities.

"Sure, you can buy the bird," I told him. "But what are going to feed it?"

With the launch of our OpenBlock project in North Carolina, rural newspapers from across the state have called or emailed to express their interest in getting our help installing and using the application. Installing the application isn't much of a challenge, I tell them, but what are you going to feed it?

OpenBlock, a "hyper-local news" platform, is a beast that eats data. So before we can make the Tar Heel State a good breeding ground for the application, we're setting out on a digital public records census. We aim to figure out how well city and county government agencies are living up to the recommendation of the Knight Commission on the Information Needs of Communities that "governments at all levels should ordinarily collect data electronically and in standardized formats."

Unlike many similar audits of public records that have been done by the Associated Press and others, this isn't some sort of exercise to see how governments comply with state and federal open records laws when they think they aren't being watched, so we're happy to describe here how we're going to go about gathering our data. In my dream world, the N.C. Association of County Commissioners sends out a link to this article to its members and implores them to help.

We're focusing our census on a few of the public records that rural newspaper publishers and editors have told us will be most valuable to their readers and advertisers -- births, deaths, land transactions, crime reports and health inspections.

Crime reports are particularly interesting. We know that people love police blotters, but also have real concerns about the safety of victims and the fairness of the criminal process. We know that state and federal agencies collect crime information in digital formats, but it's old and aggregated so it no longer has news value by the time it reaches that place in the information food chain.

To properly gauge the state of digital police records, we have to go to the city and county level. So our first step was to try to find or create a comprehensive list of every law enforcement agency in North Carolina that might generate incident or arrest reports. Thanks to a great report that a state agency submitted to the legislature earlier this year, we have the names of 569 police agencies.

From there, we're in the process of tracking down the website addresses of each agency to examine whether they publish incident and arrest reports there. (We will publish that list shortly, and may ask for your help filling in the blanks.)

Taking an 'Element' State of Mind

The bad news is that there's no indication we'll find a single agency that produces reports in a GeoRSS feed. The good news is that most police departments in the state appear to use a relatively standardized paper form to record police incidents and arrests.

In most cases, we can at least get those pieces of paper. But we've already run into cases in which police departments are unwilling to turn over standard incident reports without first heavily redacting them with misused citation of the state's open records law.

We're interested in the financial viability of OpenBlock, and paper records raise the cost. We'd have to pay people or recruit reliable volunteers to gather the paper records, scan them, and upload them to a DocumentCloud service that could use the layout of the page to extract editorially meaningful elements such as the date, time, location, and description of each document. That becomes almost impossible if we run into handwritten paper reports, which would force us to re-key the documents using local volunteers or perhaps something like Mechanical Turk.

For our census, it is not going to be enough to report that police records are online or offline, or that they are digital or not digital. We really need to be able to describe the format, location and timeliness of each data element. Taking a look at the website for the Winston-Salem Police Department, gives a good idea why we have to get more granular than the "documents state of mind" of traditional investigative reporters.

Winston-Salem publishes to its site what amounts to an index of incident and arrest reports. Each record includes the date, time, "type," case number, "primary offense" and "location." But for incident reports, it also links to a fuller record that provides information that's important for readers and reporters who want to determine the relative news value of each event -- data elements such as whether a weapon was used; the name, age, race and gender of the victim; whether drugs and alcohol were involved; whether anyone was injured; the amount of time the crime went unreported; and descriptions of the items that were stolen.

But missing from even those fuller records are data elements that would be useful for journalists who want to report trends and patterns rather than simple events. Some of the data is omitted with claims of too vague "information security purposes" and other data is omitted because of technical limitations of the departments' digital records management systems.

Each element brings with it a different cost of transforming it into a complete and current digital public record.

The variety of formats that our initial tests have already turned up seem to be limited. We've come across PDFs on the web, Word documents delivered daily via email, HTML tables, CSV file on the web, CD and via email, and the Mac-proof SNP filetype courtesy of Microsoft Access.

Most of these digital formats are created for police departments by one of three vendors that have a corner on 95 percent of the market. But we also know that 15 percent of the state's police agencies -- covering 1 percent of the population -- maintain no digital records.

police records play a key role

Police records are far from the most important -- and have proven throughout the history of this and other similar applications to be the hardest to get. But they play a key role in determining the viability of OpenBlock at rural papers. When compared to other interesting public records such as real estate or health inspections, there are simply more police reports that come out more often than other record types. Volume and frequency drive most common measures of audience engagement such as time on site and return visits.

OpenBlock is a hungry animal, and we've got to find a way to help rural papers feed it without going broke. That's the whole point of our census.

As we set off on our survey, we'll report findings and failings here. We're beginning to imagine some interesting things we'll be able to measure once we have a fuller picture of the state of records in North Carolina.

In the meantime, let me know what your experiences have been gathering digital public records at the state, county and city level. Share your experiences with @OpenRural Twitter and I'll re-tweet them. I've got lots to learn from you as well.

This is an America where people still read and trust the local newspaper, where print advertising hasn't completely migrated online. But it's also an America where reporters keep their crime database in a 300-page Word document. It's a place where the inspections department doesn't have a live XML feed, and where they may even be a little reluctant to give you paper copies if they don't recognize your face. And it's a place where a good Python developer is darn hard to find.

OpenBlock Rural is going to help these rural newspapers get ahead of the oncoming wave of digital interlopers by lowering the cost of deploying OpenBlock and using it as a tool to engage younger audiences, as well as increase advertising revenue.

Ultimately, we need to help figure out how to give rural North Carolinians -- about a third of the state's 9 million people -- the news and information they need to take advantage of life's opportunities and to participate fully in our system of self-government. This is but one of the ways that the project's goals overlap nicely with the Knight Commission on the Information Needs of Communities in a Democracy.

For us and our focus on OpenBlock, that's going to mean finding a way -- or, more likely, many ways -- to acquire, organize and produce relevant government data. While getting data into OpenBlock, and publishing it in a way that makes sense for rural areas, may have some unique technical hurdles, I don't expect the technology of scraping a site in Washington, D.C., to be much different than scraping a site in Washington, N.C.

how to produce high-quality public records

The real challenge is going to be to open government data inexpensively. Small newspaper staffs do not have access to $100-per-hour programmers. Done well, our project will show both rural journalists and county governments a way to produce high-quality -- more in a later post on what that means -- public records. My expectation is that we'll be able to recycle one or two efficient methods of getting data into the OpenBlock application. For example, just three vendors supply the record management systems to 95 percent of all police agencies in the state. So rather than finding 500 different ways to gather incident and arrest reports, we should be able to come up with three templates.

An inventory of digital public records will be one of the first things you will see from this project. Eliza Kern, a senior journalism major at the University of North Carolina and the student leader of ReeseNews.org, is already hard at work on a research project that will eventually yield a website with the locations and descriptions of local digital public records in North Carolina, as well as a report in which we're going to put a dollar figure on what it will cost private publishers to acquire and convert records.

Whatever the cost of records management, it's going to need to come in far below the amount of revenue we aim to help rural newspapers generate from this product. As I'll describe later, OpenBlock is but one piece of a digital revenue strategy for rural newspapers that my colleague Penny Muse Abernathy, the Knight chair in Digital Media Economics here at UNC, has developed with the help of her undergraduate and graduate students. (Read more about it in the textbook and workbook.) As part of this Knight News Challenge grant, she will be training the ad sales staff at our partner newspapers on how to create and sell opportunities for local advertisers to sponsor this application.

With revenues outpacing the operational costs of each rural OpenBlock installation, profits will be available to produce analytical reporting that builds and sustains an informed community. The data we free up for use in this project should allow both journalists and their readers to ask themselves questions that are vital to economic and cultural development -- how are we doing compared to other communities like us, and are we heading in the right direction?

Using hard data to answer hard questions

And just like those communities, we're going to use hard data to find the answers to difficult questions we'll face throughout this project.

Questions such as:

• How does OpenBlock deal with a rural county that has 17 local governments, including a tribal government?
• What geographies are important in sparsely populated rural communities, and how do we display them in a meaningful way?
• In communities where everyone knows everyone, does "public information" take on a different meaning when it gets published online?
• Can a relevant dataset foster technical innovation and STEM education in rural communities?
• What role will broadband penetration and mobile Internet play in the user experience of rural OpenBlock?
• How can UNC's Journalism & Mass Communication students continue to be cross-discliplinary leaders in editorial product development?

I'm looking forward to your questions as well. Send them my way at @rtburg or @openrural, and watch this space as we learn and share.