At the end of last year, Raju Narisetti proposed a pay wall alternative he dubbed the "'Why don't we pay you?' pay wall" ... and then left the unwalled Washington Post for the walled Wall Street Journal.

The conversation all this time has been focused on whether the shift toward digital subscriptions will save the news business. But the more interesting and important question is whether and how it will change the news content and public discourse.

There's never been a question that people will pay for digital content. Give people information they need to profit professionally or enjoy personally, and they will pay for it. But what about all the boring and bad stuff? What about the kind of iron-butt reporting that has journalists cover legislative subcommittee meetings just so powerful people know the public is watching? And the quarter million-dollar investigations that find the hidden winners and losers?

That news doesn't entertain; it doesn't give me a competitive edge; and it doesn't save my family money in the short run. Those kind of stories make big waves every now and again, but no matter how high the pay wall, once the story is out, it spreads via broadcast news, social media and word of mouth. Even those who don't pay for it get to benefit from its impact.

social media's role

The role that social media plays in the subscription pay model isn't fully understood -- by me at least. I'd like to find the time to ask about whether paying subscribers share more or different stories than non-subscribers.

In any case, with a pay wall in place, subscribers will -- as always -- set the agenda more than non-subscribers. Some subscribers will be more influential than others, either because they have more followers or because they provide a better filter. In either case, the future of public discourse lies with subscribers. We need to know more about who they are and how their desired public agenda differs from non-subscribers.

It's easy to suspect that only the elite would pay for news -- only people whose personal social and economic decisions are determined by taxpayer money and public markets -- and that the topics that interest those folks may not be particularly populist.

But then I stumbled across a January 2011 survey by the Pew Research Center that seems to indicate that the willingness to pay for news may not be as elitist as I originally thought: African Americans and Hispanics are significantly more likely than whites to say that they would pay a monthly subscription fee if that was the only way to get full access to their local newspaper online. But there's no significant difference among any age groups under 65, nor is there a difference between men and women. On the other hand, college grads and people who make more than $75,000 a year are more likely to say they would pay for online local news than people who make less and have less education.

So does the public discourse look different if the people who subsidize original reporting -- and then share it -- are rich, educated, racial and ethnic minorities? After paying to see the news, what would they share? And who would they share it with?

the social distribution of news

The democratization of publishing means that alternative points of view would always be waiting in the on-deck circle anytime the paid-stream media misses a story its audience cares about. So it's also important to predict what kind of effect the audience's sharing patterns would have on journalists who want to make sure their pay walled reports remain valuable enough to make ends meet.

The social distribution of news has two benefits for news organizations -- they sell advertising against each unique visitor, and they have an opportunity to convert the social media samplers into paying subscribers. But if the role of advertising at news organizations becomes a significantly lower share of revenue, then eyeballs alone won't matter as much. News organizations might be less interested in running "water cooler" stories that are cute and fun alone. And they might be more inclined to run stories that target an audience that wants more than 140-character summaries.

Research collaborations between academics and industry could help us make better guesses -- and making good guesses on this topic will be important for any news organization that understands it doesn't sell ads or subscriptions, but trust and influence.

Image courtesy of Flickr user Aunty P.

The panel was organized by Fiona Morgan, a researcher at the DeWitt Wallace Center for Media and Democracy at Duke University, who worries that as newspaper companies try to harvest more revenue from a shrinking audience, they are catering both content and delivery to a wealthy, educated, white audience. She asked me to join the panel to discuss how the old idea of the "penny press," which revolutionized journalism by covering news that appealed to a broader audience, might be updated from the digital age.

In rural communities, newspapers actually don't have much choice but to serve an audience that has less money and education on average than a typical newspaper reader. For example, in Columbus County, N.C., The News Reporter can't ignore people who have little "spending power" because there are simply too many of them. Of about 22,000 households in the county, 38 percent have incomes of less than $25,000 a year, according to the Census Bureau' s American Community Survey. Only 19 percent have incomes above $75,000. Compare that to Wake County, where the numbers are reversed -- only 18 percent of households live on less than $25,000 a year and 45 percent make more than $75,000.

But poverty in Columbus County doesn't translate into a lack of interest in the news (although it does translate into lower voter turnout). The twice-weekly, family owned News Reporter in Whiteville reaches just more than half the households in the county. The News & Observer, the biggest paper in Wake County, reaches only about 20 percent of its home-county households.

the threat of shifting revenues

With market strength like that, it looks at first blush like the best way to recreate the penny press for the digital age is to print on dead trees. But that ignores threats just over the horizon. Right now, the average visitor to whiteville.com spends just over four minutes per visit, compared with the 20 minutes that the average visitor spends on each trip to Facebook. Based on Census data and survey research done by the Pew Research Center, there's a good bet that there are just as many people in Columbus County on Facebook as there are readers of the print newspaper.

Poor or not, the people of Columbus County use the Internet. Even though we know that income has a positive correlation with newspaper and online news consumption, there are just so many poor people in Columbus County that they dominate every media measurement. When looking at raw numbers, there's probably about 25 percent more very poor people on Facebook in Columbus County than very rich. And on Twitter, the raw numbers of very poor and very rich are likely about even in Columbus County.

That's important because the business staff at The News Reporter has just within the last few months begun to see signs that people are not putting their classified ads in the newspaper because they are posting items for sale on Facebook -- a company whose mission isn't to support accountability journalism.

As big city newspaper companies can tell you, a future of shifting revenues means a future in which you have to either live with smaller margins or spend less on reporting and other expenses.

The News Reporter is a family owned company with a Pulitzer Prize in its history. I'd bet it spends a higher percentage of its revenue on reporting and editing than most newspapers. The challenge for rural newspapers, though, is the economy of scale. They simply can't divide the cost of covering local government among as many readers. If The News & Observer sends someone to the Wake County courthouse, the cost of that reporting is shared among 70,000 readers.

But if The News Reporter sends someone to the Columbus County courthouse, it can only split the bill about 11,000 ways. So, per capita, it's nearly seven times as expensive for Columbus County readers to get local news coverage as it is for Wake County residents.

where openrural comes in

OpenRural aims to lower the cost of gathering and publishing basic data about government and public life. Property sales, arrest reports, new business openings and restaurant inspections have long been a staple of community newspapers. But until now, publishing them has required a reporter to go down to a county office, pick up a piece of paper, and re-type the information into her newspaper's publishing system. We aim to automate as much of that as possible.

Lowering costs and serving an audience across all demographics, OpenBlock appears to meet all the requirements for itself being a penny press for the digital age. But generating cheap content doesn't solve the revenue problem in a world of abundance and bad competitors who are willing to provide a similar or better service for even lower margins.

The only way that OpenBlock -- or any penny press in the digital age -- is going to solve revenue woes is by increasing audience loyalty in both print and online.

If OpenBlock lowers the cost of collecting and publishing commodity news in rural markets and staves off some bad competitors, then the next step will be for publishers to reinvest the savings into high-quality, high-impact public affairs reporting. Reporters who once gathered paper and went to meetings will need to do more stories about the "how" and the "why" rather than simply the who, what, when and where. For these rural communities to lift themselves out of poverty, they need to be able to look at trends in the data.

In his book "The Vanishing Newspaper," Philip Meyer looks at how newspapers use reporting to create trust and influence in their communities and how they then sell that influence to advertisers. So the secret to building a penny press for the digital age isn't just about generating abundance of content any more than it's about catering to an elite audience. Page views without impact have no value.

And that's why Meyer also said in another book, "Precision Journalism," that the ante is being raised on what it takes to be a journalist. Data can't just be an input in a low-cost publishing process. It must also be the raw material that reporters are able to use for analytical and accountability journalism.

Newspaper image courtesy of Flickr user Binuri Ranasinghe.

Over the course of the next several posts, I'm going to give you an X-ray view into the guts of the OpenBlock application. Together, we're going to watch how data gets ingested and processed into information and insights that residents of rural communities can use to make decisions about their daily lives.

We knew basically two things when we started this project. First, we knew that public data went into OpenBlock. And we knew that digital public data is for the most part in a poor condition to be easily digested. Second, we knew that the meaningful geographies of rural areas were going to be different than the geographies of urban areas. But beyond that, the anatomy of OpenBlock remained almost completely obscured. No instances of the application lived in the wild, and the code itself was still missing significant documentation.

Step 1: Getting data into OpenBlock

Our first step was to figure out how to get data into OpenBlock. And while many of us probably think about data as being some sort of news event -- a transaction, creation, deletion, inspection, election, rejection, incarceration or some other function of government that takes place at a specific time -- the initial data you need for OpenBlock is about a location. You have to tell it where it is. And you do that by ingesting data from the U.S. Census Bureau.

The Census Bureau is an amazing resource of geographic data, which it calls TIGER/Line files. The bureau provides TIGER/Line files that show information about various "layers" of geography -- most of which aren't relevant to OpenBlock. Each layer actually consists of several files that you download from the Census website as a single zip file. And inside that zip file is a file with a .shp extension -- that's a shapefile, and it is the only one that has nutritious value as far as OpenBlock is concerned.

Several layers are important -- one file contains data about the boundaries of all ZIP codes in a state. Even though most ZIP codes aren't going to be relevant, you need them anyway. And while loading ZIP code shapefiles into OpenBlock may not be simple for people who don't have at least some familiarity with Django, it doesn't require a lot of human judgment. Plug in some code and you're done.

But nobody lives their lives by ZIP codes. We care about geographies like counties and cities and streets. And knowing which geographic data to load next does require some editorial thinking. Most of us are familiar with counties as political entities that have some meaning. But counties are different in each state. For example, in Virginia there is a City of Fairfax and County of Fairfax. Fairfax City isn't in Fairfax County or any county. They touch each other, but one does not have jurisdiction over the other. Now, here in North Carolina we have Durham County and Durham City. Durham City is inside Durham County -- mostly -- except for the part of Durham City that is in Orange County. Orange County is the home of Chapel Hill, except for the part of Chapel Hill that's in Durham County. And both Durham and Orange counties contain large parts of land that aren't in any city at all.

teaching journalism to programmers and vice versa

So when we talk about teaching journalism to programmers and programming to journalists -- this is really the kind of thing we're talking about. Somewhere here we have to have someone who knows the political geography of North Carolina and who can also describe the rules of that geography to a computer program so that it doesn't leave out anything it shouldn't and also includes everything it should.

Let's say we want to show the locations of all new business that have been incorporated in Orange County. The N.C. Secretary of State, which records new businesses, may have the address of the business, but not the county. We're going to have to tell the OpenBlock application which addresses are inside Orange County -- regardless of whether they are in Chapel Hill, Durham City or no city. If we tell it just to grab the Chapel Hill addresses, we will erroneously include the businesses that are in the part of Chapel Hill that's in Durham County. Or, since most people think of Chapel Hill as being an Orange County city and they might be confused if they know a business has opened but isn't listed on our website, we may want to tell OpenBlock to include all Orange County addresses, but not the ones that are in Durham City, and also include the Durham County addresses if they are in Chapel Hill.

And then we have to write for the reader an explanation of whatever assumptions we're making in our data, and it has to be brief and clear.

Once you've made your editorial decisions, county bounders can also be downloaded from the Census Bureau as well. Just to be proper, Census calls this layer "County and Equivalent."

Cities, however, are a more delicate matter that again require some knowledge of Census terminology and local political geography. There is no Census layer called "cities."

Until recently, the practice among the OpenBlock community had been to look for geographic information about city boundaries from local governments. Most county governments are pretty good about publishing their geographic data on the web. Many large public universities have a GIS (geographic information system) section in their library, such as this one at N.C. State University that indexes links to the geographic data source for many of the state's county governments.

lesson learned: local data varies

The problem with local data is that poor and rural counties are less likely to have the online GIS data. And you will also find yourself dealing with a wide variety of standards. We spent a few days flummoxed by our inability to load Columbus County data. For whatever reason, the county decided not to include the necessary .prj projection file you need to make a shapefile work in OpenBlock. It's a lesson that's going to be important for us to remember throughout this project -- local data varies widely in quality. And sometimes it's not obvious to the layperson's eye what is missing.

Our thinking right now is that we're going to be able to turn back to the Census Bureau for city information. But we're not using the Census layer called "Consolidated Cities." Nor the one called "County subdivisions." We're using the "Places" layer.

You can read more about our "city" solution on the OpenBlock discussion group. But there are a few bits worth mentioning here:

• Places do not cover 100 percent of a county. So we're going to need to be on guard that events happening outside the boundaries of one of our places doesn't somehow get left out of the search results.
• Some of the places in Columbus County probably have little or no meaning to the audience there. In the "Government" navigation of our partner site, Whiteville.com, not all of the Census places are listed. That said, several of them show up occasionally as the location of obituaries that run on the site.

Geography is hardly the dynamic data we think of as news. But we've already seen several road bumps that are big enough to deter almost all small news organizations from using the OpenBlock application. So post this as one of our goals: to automate the process of loading basic ZIP, county and city geographic data about your community. Again, you can read more in the OpenBlock discussion group about how we might use FIPS codes -- the unique number given by the Census Bureau to each state, county and "place" -- to do that.

And there's still a big elephant in the room -- the geography of "blocks" that is one of the core concepts of OpenBlock. That topic is big enough for its own post a few weeks from now. Before then, I'm going to walk you through some of our experiences loading "news" data into OpenBlock and how we're hoping we might be able to work with fellow Knight News Challenge winner ScraperWiki to build our own API for North Carolina state government.

Understanding each step of the set-up, production and editing process involved with OpenBlock is critical to our ability to describe the expense side of the equation, which we hope and expect will lead to the financial viability of the application as a tool to fill the information needs of rural communities.

"Sure, you can buy the bird," I told him. "But what are going to feed it?"

With the launch of our OpenBlock project in North Carolina, rural newspapers from across the state have called or emailed to express their interest in getting our help installing and using the application. Installing the application isn't much of a challenge, I tell them, but what are you going to feed it?

OpenBlock, a "hyper-local news" platform, is a beast that eats data. So before we can make the Tar Heel State a good breeding ground for the application, we're setting out on a digital public records census. We aim to figure out how well city and county government agencies are living up to the recommendation of the Knight Commission on the Information Needs of Communities that "governments at all levels should ordinarily collect data electronically and in standardized formats."

Unlike many similar audits of public records that have been done by the Associated Press and others, this isn't some sort of exercise to see how governments comply with state and federal open records laws when they think they aren't being watched, so we're happy to describe here how we're going to go about gathering our data. In my dream world, the N.C. Association of County Commissioners sends out a link to this article to its members and implores them to help.

We're focusing our census on a few of the public records that rural newspaper publishers and editors have told us will be most valuable to their readers and advertisers -- births, deaths, land transactions, crime reports and health inspections.

Crime reports are particularly interesting. We know that people love police blotters, but also have real concerns about the safety of victims and the fairness of the criminal process. We know that state and federal agencies collect crime information in digital formats, but it's old and aggregated so it no longer has news value by the time it reaches that place in the information food chain.

To properly gauge the state of digital police records, we have to go to the city and county level. So our first step was to try to find or create a comprehensive list of every law enforcement agency in North Carolina that might generate incident or arrest reports. Thanks to a great report that a state agency submitted to the legislature earlier this year, we have the names of 569 police agencies.

From there, we're in the process of tracking down the website addresses of each agency to examine whether they publish incident and arrest reports there. (We will publish that list shortly, and may ask for your help filling in the blanks.)

Taking an 'Element' State of Mind

The bad news is that there's no indication we'll find a single agency that produces reports in a GeoRSS feed. The good news is that most police departments in the state appear to use a relatively standardized paper form to record police incidents and arrests.

In most cases, we can at least get those pieces of paper. But we've already run into cases in which police departments are unwilling to turn over standard incident reports without first heavily redacting them with misused citation of the state's open records law.

We're interested in the financial viability of OpenBlock, and paper records raise the cost. We'd have to pay people or recruit reliable volunteers to gather the paper records, scan them, and upload them to a DocumentCloud service that could use the layout of the page to extract editorially meaningful elements such as the date, time, location, and description of each document. That becomes almost impossible if we run into handwritten paper reports, which would force us to re-key the documents using local volunteers or perhaps something like Mechanical Turk.

For our census, it is not going to be enough to report that police records are online or offline, or that they are digital or not digital. We really need to be able to describe the format, location and timeliness of each data element. Taking a look at the website for the Winston-Salem Police Department, gives a good idea why we have to get more granular than the "documents state of mind" of traditional investigative reporters.

Winston-Salem publishes to its site what amounts to an index of incident and arrest reports. Each record includes the date, time, "type," case number, "primary offense" and "location." But for incident reports, it also links to a fuller record that provides information that's important for readers and reporters who want to determine the relative news value of each event -- data elements such as whether a weapon was used; the name, age, race and gender of the victim; whether drugs and alcohol were involved; whether anyone was injured; the amount of time the crime went unreported; and descriptions of the items that were stolen.

But missing from even those fuller records are data elements that would be useful for journalists who want to report trends and patterns rather than simple events. Some of the data is omitted with claims of too vague "information security purposes" and other data is omitted because of technical limitations of the departments' digital records management systems.

Each element brings with it a different cost of transforming it into a complete and current digital public record.

The variety of formats that our initial tests have already turned up seem to be limited. We've come across PDFs on the web, Word documents delivered daily via email, HTML tables, CSV file on the web, CD and via email, and the Mac-proof SNP filetype courtesy of Microsoft Access.

Most of these digital formats are created for police departments by one of three vendors that have a corner on 95 percent of the market. But we also know that 15 percent of the state's police agencies -- covering 1 percent of the population -- maintain no digital records.

police records play a key role

Police records are far from the most important -- and have proven throughout the history of this and other similar applications to be the hardest to get. But they play a key role in determining the viability of OpenBlock at rural papers. When compared to other interesting public records such as real estate or health inspections, there are simply more police reports that come out more often than other record types. Volume and frequency drive most common measures of audience engagement such as time on site and return visits.

OpenBlock is a hungry animal, and we've got to find a way to help rural papers feed it without going broke. That's the whole point of our census.

As we set off on our survey, we'll report findings and failings here. We're beginning to imagine some interesting things we'll be able to measure once we have a fuller picture of the state of records in North Carolina.

In the meantime, let me know what your experiences have been gathering digital public records at the state, county and city level. Share your experiences with @OpenRural Twitter and I'll re-tweet them. I've got lots to learn from you as well.

This is an America where people still read and trust the local newspaper, where print advertising hasn't completely migrated online. But it's also an America where reporters keep their crime database in a 300-page Word document. It's a place where the inspections department doesn't have a live XML feed, and where they may even be a little reluctant to give you paper copies if they don't recognize your face. And it's a place where a good Python developer is darn hard to find.

OpenBlock Rural is going to help these rural newspapers get ahead of the oncoming wave of digital interlopers by lowering the cost of deploying OpenBlock and using it as a tool to engage younger audiences, as well as increase advertising revenue.

Ultimately, we need to help figure out how to give rural North Carolinians -- about a third of the state's 9 million people -- the news and information they need to take advantage of life's opportunities and to participate fully in our system of self-government. This is but one of the ways that the project's goals overlap nicely with the Knight Commission on the Information Needs of Communities in a Democracy.

For us and our focus on OpenBlock, that's going to mean finding a way -- or, more likely, many ways -- to acquire, organize and produce relevant government data. While getting data into OpenBlock, and publishing it in a way that makes sense for rural areas, may have some unique technical hurdles, I don't expect the technology of scraping a site in Washington, D.C., to be much different than scraping a site in Washington, N.C.

how to produce high-quality public records

The real challenge is going to be to open government data inexpensively. Small newspaper staffs do not have access to $100-per-hour programmers. Done well, our project will show both rural journalists and county governments a way to produce high-quality -- more in a later post on what that means -- public records. My expectation is that we'll be able to recycle one or two efficient methods of getting data into the OpenBlock application. For example, just three vendors supply the record management systems to 95 percent of all police agencies in the state. So rather than finding 500 different ways to gather incident and arrest reports, we should be able to come up with three templates.

An inventory of digital public records will be one of the first things you will see from this project. Eliza Kern, a senior journalism major at the University of North Carolina and the student leader of ReeseNews.org, is already hard at work on a research project that will eventually yield a website with the locations and descriptions of local digital public records in North Carolina, as well as a report in which we're going to put a dollar figure on what it will cost private publishers to acquire and convert records.

Whatever the cost of records management, it's going to need to come in far below the amount of revenue we aim to help rural newspapers generate from this product. As I'll describe later, OpenBlock is but one piece of a digital revenue strategy for rural newspapers that my colleague Penny Muse Abernathy, the Knight chair in Digital Media Economics here at UNC, has developed with the help of her undergraduate and graduate students. (Read more about it in the textbook and workbook.) As part of this Knight News Challenge grant, she will be training the ad sales staff at our partner newspapers on how to create and sell opportunities for local advertisers to sponsor this application.

With revenues outpacing the operational costs of each rural OpenBlock installation, profits will be available to produce analytical reporting that builds and sustains an informed community. The data we free up for use in this project should allow both journalists and their readers to ask themselves questions that are vital to economic and cultural development -- how are we doing compared to other communities like us, and are we heading in the right direction?

Using hard data to answer hard questions

And just like those communities, we're going to use hard data to find the answers to difficult questions we'll face throughout this project.

Questions such as:

• How does OpenBlock deal with a rural county that has 17 local governments, including a tribal government?
• What geographies are important in sparsely populated rural communities, and how do we display them in a meaningful way?
• In communities where everyone knows everyone, does "public information" take on a different meaning when it gets published online?
• Can a relevant dataset foster technical innovation and STEM education in rural communities?
• What role will broadband penetration and mobile Internet play in the user experience of rural OpenBlock?
• How can UNC's Journalism & Mass Communication students continue to be cross-discliplinary leaders in editorial product development?

I'm looking forward to your questions as well. Send them my way at @rtburg or @openrural, and watch this space as we learn and share.