It turns out only part of that quote is accurate, but you'll never know which one for sure! This is why, before I can graduate from MIT, I have to create an automated bullshit detector. The basic premise is that we, as readers, are inherently lazy. It isn't just that we'll believe almost anything. (Remember that time in 1938 when we believed aliens were invading the planet just because someone on the radio said so? Yeah. That happened.) The real problem is that we'll often believe what we want to believe (or disbelieve what we don't want to believe).

It's hard to blame us. Just look at the amount of information flying around every which way. Who has time to think carefully about everything? Not me. This is why I'm working on a tool called Truth Goggles that will help hone our critical abilities -- and help us identify pieces of information that are worth inspecting a little more before deciding how it fits into our world views.

Thesis Goggles

When I wrote "before I can graduate from MIT" earlier in this post I wasn't lying; I have decided to pursue Truth Goggles for my thesis. I'm definitely not the first person to explore this problem space, but there's lots of room to contribute. New technology has opened up new possibilities; needs have become clearer; and there is a wide variety of possible solutions and unanswered questions just sitting around waiting to be explored.

In November, I presented the idea to the Media Lab community using the following slides:

The feedback I got was mixed, but what can you expect from a day called "Crit Day," which is short for "Critically Injure Pride, Hopes, and Dreams of Graduating Day." Here were the main questions asked:

This doesn't seem like it will scale considering Politifact only has a few thousand fact-checked claims. Why aren't you using the crowd to fact check?

My time at MIT will be spent focusing on the interface and user interaction rather than the generation and aggregation of source information. There are enough difficult questions surrounding the interaction layer. I don't think it is worth complicating things further by trying to create a crowd-based journalism platform (which is essentially what crowdsourced fact checking amounts to).

Isn't this just a mashup of technologies and data sets? How is what you are doing novel?

It's true that I'm not inventing new algorithms. I'm applying existing algorithms in novel ways. Credibility layers aren't robust right now, and they come with their own sets of interesting questions in terms of user experience and system design. My contribution will be to frame those questions, answer some of them, create a prototype, and test that prototype. This won't be as trivial as just throwing more information on a screen and calling it a day -- the interface has to be designed with care.

Do you expect to incorporate primary source data?

My initial prototype probably won't pull from sources other than Politifact and other fact-checking services, but I will definitely be thinking about ways to use other sources of data. Primary source content will eventually help with information scalability since raw footage and raw data could help computers find potentially dubious claims (and help readers make determinations about those claims).

Bullshit, This is Clearly Science Fiction

There are a lot of hard questions lurking behind corners here. In fact, most of them aren't even trying to hide; they're just sitting obnoxiously in the middle of the room. Some are technical. Some are philosophical. But all of them need to be addressed intelligently for something like Truth Goggles to actually have a chance of working. I'll rattle off a few of them.

• Who determines the truth? Journalists? Experts? Crowds? Individuals? Algorithms?
• Sometimes there is a right answer, and sometimes there is room for debate. Can you tell which is which? How do you reflect the difference?
• How does the tool account for bias in sources?
• How does the tool account for bias in users?
• Will the system actually know enough to be regularly useful?
• This could easily just make consumers more lazy. How do you prevent that?
• What happens when the tool is wrong?
• How will this change the way people produce content?
• Where do journalists fit into the picture?

As I've pondered these questions, I've come to the following absolute conclusion: Credibility layers need to empower critical ability. I've also decided that it's OK for the system to make mistakes, but it is never allowed to lie. This means the interface should be less focused on telling the reader what to think and much more honed in on reminding (and helping) the reader to think at times when thinking is most important.

I've also come up with a list of weaker claims to throw out there for discussion:

• Credibility layers don't have to speak to everyone, but they need to empower the open-minded.
• Journalists are our best bet for deep analysis and identifying truth that requires lots of time and effort (e.g., investigation and concept synthesis).
• Algorithms are our best bet for identifying contextual evidence (e.g., data, trends, and sources of sound bytes).
• Mobs can't be trusted to decide what is true and false, but they are the key to figuring out what is worth thinking about.

Over the coming months, I'll be cranking out interfaces, prototypes, and eventually some good old-fashioned, boring academic papers about this idea. In the meantime, if you're interested in Truth Goggles, I'll be trying to post updates as regularly as possible on my blog, on Twitter (@slifty), and eventually on the newly registered truthgoggl.es.

This post has been cross posted on slifty.com

Since it looks like I'm going to be one of the lucky ducks who gets to hack away on my proposed idea in Berlin this September, I wanted to share it here. I would love feedback, of course, but also if you know anybody who might be interested in incorporating their content or using the platform, I would love to get in touch.

Part 1: Introduction

Scraping, Slicing, and Scrubbing C-SPAN

How do you get from a TV channel to a rich video archive, and how do you get there automatically? The goal of my project, called ATTN-SPAN, is to convert C-SPAN into a series of overlapping video segments that are identified in terms of state, politician, topic, party, action, and legislative item. Some of this is straightforward, and some of it might be impossible, but here's an overview of the planned nuts and bolts:

1. DirecTV offers TV content in a format that's easy to record digitally, and VLC is a free tool that can do that recording. Combine the two, and we can download C-SPAN streams into individual files that are primed and ready for analysis.
2. Once a video file is in our clutches, we can use VLC once again to separate out the video from the Closed Captioning transcript.
3. Now we have a transcript and a raw video file. Next, we register all of this information (in a database) so that we can look it all up later, and then convert the video file into streaming-friendly formats and store it alongside the original recording.
4. C-SPAN consistently shows a graphic on the bottom of the screen that says who is talking, their state, their party, and what is being debated. By using a technique called Optical Character Recognition (OCR) we can pull this text out of the video image. Once pulled, we can add that to our database so that we can access all of this information for any moment in the video.
5. At this point, we have most of the information we need, but there's still room for fine-tuning. We can use audio levels and the closed-captioning transcripts to try to identify moments of inactivity, normal dialogue, and heated dialogue.

These steps are enough to split up and categorize C-SPAN footage into an organized video database, but there are still more ways to flag special moments in the footage. For example, we may want to identify changes in speaker emotion in order to give our algorithms the ability to craft more engaging episodes. This is possible through the work of Affective Computing at the MIT Media Lab, a group which has developed several tools that perform emotional analysis using facial recognition.

We may also want to identify specific legislative action (e.g., "calling a vote"). This could be accomplished by looking for key words in the transcript (e.g. "call a vote") and possibly through common patterns in the audio signal. (Maybe there are identifiable sounds, such as a gavel hitting the table.) Both of these concepts require additional research.

Creating a Profile and Constructing an Episode

If video events are the building blocks, then viewer interests are the glue. The creation of a personalized episode requires two things: a user account and a context. The user account provides general information like where you live, what issues you have identified as important, and (if you are willing to connect with Twitter or Facebook) what issues your circles have been discussing lately.

The context comes from time and cyberspace. Every night, after Congress closes their gates, your profile is used to create a short, rich video experience designed to contain as much relevant content from that day as possible. At this point, you might get an email begging you to watch, or maybe you log in on your own because you're addicted to badges and points and you want as much ATTN-SPAN karma as you can get.

There's another way to access this content though, and that is through the websites you visit anyway. Imagine if you could read an article about the national debt on The New York Times (or in a chain email) and actually see quotes from your own senators in the report. What if you could supplement the national report with a video widget that lets you browse what your House members had to say when they controlled the floor during the debt debates?

From a technical perspective, this isn't that far-fetched. Truth Goggles, one of my other projects, is a bookmarklet that will analyze the web page you are viewing, fact-check it, and rewrite the content to highlight truths and lies. This impossible feat is fairly similar to what I'm proposing here.

Adding Rich Information

Once an episode is pieced together, we can look up the information surrounding the video to know who is talking and what they're talking about. What else can be added, and how do we get it? Existing APIs (application programming interfaces) offer some good options:

• Contact Information - Thanks to the Sunlight Labs Congress API, it's possible to get the contact information for any member of Congress on the fly. Thanks to VoIP services, it's possible to create web-based hooks to call those people with the click of a button.
• Campaign Contributions - The New York Times offers a Campaign Finance API which can help you understand where the person onscreen gets his or her money.
• Voting Records - The New York Times also offers a Congress API that will make it possible to know vote outcomes from related bills as well as information about the active speaker's voting records.
• Truth and Lie Identification - My Truth Goggles project can be easily adapted to work with snippets from video transcripts. This will allow ATTN-SPAN to take advantage of fact-checking services like PolitiFact and NewsTrust.

This is a good start, but I would also like to show links to related news coverage and create socially driven events based on community sentiment (for instance, to track moments that caused people to get upset or happy). This won't come for free, but it should be accessible given the right interface design.

Part 3: A Note to the Newsies

So that's the idea and the plan. What's the value?

It seems plausible that ATTN-SPAN, a system that analyzes primary source footage and pulls out any content that's related to a particular beat, could be useful as a reporter's tool, but what about your subscribers? ATTN-SPAN can augment an individual article so that it hits everybody close to home. Suddenly, one article becomes as effective as two dozen. Moving past text, for larger organizations with a significant amount of video footage, ATTN-SPAN can be tweaked to use your programming instead of (or in addition to) C-SPAN.

At this point, I have to warn you that this isn't the first, nor will it be the last, project to work with C-SPAN. A 2003 demo out of the Media Lab used C-SPAN as one of several sources of information in a platform aimed to provide citizens with Total Government Awareness. Metavid, the platform I used in my initial prototype, already makes C-SPAN more accessible by enabling searches and filters. The list surely goes on.

So why is this a more powerful project? Well, the real goal of ATTN-SPAN isn't to get more people watching C-SPAN. In fact, I tricked you: This project isn't about government awareness at all. It's actually part of an effort to make indisputable fact ("blunt reality" and "primary source footage") a more prominent part of the media experience without requiring additional effort from the audience.

Newsrooms do an amazing job of reporting events and providing insight, but for deeper stories, there simply isn't enough time or money to cover everybody's niche without going beyond the average person's attention span.

Thus ends my pitch.

The code for both prototypes mentioned in this post can be found on GitHub: ATTN-SPAN and Truth Goggles. Please forgive any dirty hacks. I would be thrilled if anybody wants to offer suggestions or even collaborate. On that note, please get in touch on Twitter @slifty. This proposal was originally posted on slifty.com.

Instead I witnessed a far more situation-based analysis of what incentivizes action that concluded with a simple, summarizing message: The only thing technology can do is amplify a movement; to instigate actual change you need people on the ground.

Hearing this summary launched dreams of Anonymous dancing through my head, and I almost jumped out of my seat to exclaim O RLY? but as I thought about it, I realized that my gut's push-back was misdirected. The panel was correct, just not as universally as they tried to imply.

Chris Faulkner, a Republican political consultant, and Yesenia Sanchez, an organizer for the Illinois Coalition for Immigrant and Refugee Rights, were organizing rallies, facilitating communities of people with similar values, and swaying political opinions. Both of their efforts required feet on the ground, but this doesn't represent all movements.

What circumstances make digital tools a sufficient means to an activist's end?

The Ground

After the session, I spoke with Charlie DeTar, a fellow student at MIT who has done many great projects under the umbrella of civic media. He agreed with the panelists' conclusion and has for some time. He cited the Green Revolution from 2009 where a bunch of Westerners managed to use up some extra electrons to power green diodes. The real battle took place in Iran -- the online movement just helped spread the messages.

Even in success stories with incredible digital representation, the ground movement is often equally great. The 2008 Barack Obama campaign, lauded for its use of social networking and digital tools, was also made up of spectacular grassroots community organizing. For Faulkner, that ground movement is what really separated Obama from Sen. John McCain. He sees online versus on-foot as the difference between creating supporters (those who will vote) and fans (those who will act). McCain had supporters. Obama had fans.

The Cloud

Despite what this panel may have implied, online-centric movements exist in full force. They are easy to overlook when thinking about physical communities because they're often focused on digital concepts such as services, information flow, or technology products. They could be inspired by the Facebook Terms of Service, for instance, which at times have garnered public outcry, push-back, and even some sort of happy ending.

Another mainstream example can be found in the countless events surrounding WikiLeaks, and even the WikiLeaks themselves. The battles waged online through "hacktivism," document releases, and online campaigns to spread information put targeted pressure on real-world organizations. This situation resonated beyond cyberspace into the real world because the issues at stake concerned real-world governments, conflicts and organizations.

There are also plenty of niche issues with online movements behind them. I would venture to say that every active online community has a core who are advocating change to their groups in some form or another. For instance, there is a reasonably successful push among web developers to prevent the perpetuation of W3Schools, a programming knowledge base, which many believe teaches poor best practices in the name of simplification. Gamers have similarly passionate pushes, as any StarCraft 2 player would see after one minute of looking at the Blizzard forums.

Several of these examples don't sound like political movements or revolutions because they aren't. Nonetheless, they are arguably movements, and unlike their locative counterparts they could have legs that never touch the ground. In fact, some would make no sense whatsoever offline.

Threading the Needle

Clearly there are examples of both extremes, but we don't live in binaries. A movement's tension between digital and physical embodiment is reflected by its details. When you're organizing or analyzing a movement, whether it be digital or physical, you should consider the following:

• The Goals: Do you want new legislation or the election of a representative? Is it a push for free and open information flow or the change of a digital standard? Movements should be based in the places that they will affect.
• The Community Audience: Is the community Springfield or is it the disenfranchised member base of the Huffington Post? You need to speak to your audience so they will pay attention and in terms that will inspire them.
• The Outlet for Change: How will a community member have an impact once they join your cause, and what will they be excited to do? Spread information? Run an application? March? Sit-in? This will likely indicate where your focus should be.

If you look at movements through these lenses it should become clearer why they evolved as they did and why certain actions worked more effectively than others. In the cases where the lines blur, as they did with Anonymous vs. Scientology, and then once again with WikiLeaks, you can see that there were conflicting answers to those three characterizing questions. These are the cases where the most useful lessons can be learned, but for that very reason they are the most difficult to analyze.

Neither digital tools nor grassroots efforts will be a consistently dominant force for change. Both categories of technique play different roles in different circumstances. With that in mind, activists shouldn't use generalized advice to decide when a technology makes sense; rather, that advice should just be used to fuel brainstorming. If you have a vision for the future, it's up to you to figure out which messages and mediums will resonate most within your community.

It turns out that the use of digital tools in crisis situations is a concept with rich communities and plenty of solid examples. Within the Media Lab there are a handful of projects designed to assist in crisis situations and outside of the Lab there are hundreds more. Because I just started exploring the area, I want to share my initial observations along with a quick description of the projects I'm most familiar with.

Framing the Challenge

I always thought that social systems were complicated, but I will never complain again. The issues surrounding a crisis tool are almost unreal. Take a minute to think of the types of challenges that a relief technology project needs to consider up front, before even targeting a particular problem. Here are a few examples to get your brain running:

• Limited technology access -- The primary stakeholders of a relief technology are those living in a disaster area. Maybe the area is a developing country with no digital infrastructure or a more developed country whose digital infrastructure just got annihilated. Maybe people have the Internet, maybe they just have cell phones, or maybe they really have nothing on the ground. Nothing is a given here.
• Shifting impact windows -- What are the phases of disaster relief, how long do they last, and what changes between them? I like the general structure presented in this document [PDF]. It starts with an immediate response, which focuses on things like saving lives, cleaning up, and tending to basic human needs. This is followed by mid-term planning, which involves getting the basics back such as temporary housing and lifeline utilities. Finally the long term reconstruction phase begins. Each of these phases has drastically different needs, requirements, and timelines.
• Global-scale requirements -- National disaster relief is a problem that even government-scale organizational structures struggle to deal with. What are the core needs and how can they be met? Who is being impacted? Who is going to participate and how? What are the implications of your solution? What is already being done and how can you fit in effectively? How many different cultures are going to be using your tool? What languages are involved? These are all vital questions which have to be thought through.
• Lives are at stake -- Behind this entire process is an ultimate fact: These technologies are dealing with matters of life and death. If an organization relies on a tool for some portion of its operations and the tool fails there could be very real and serious consequences.

What exactly do these issues mean from a system design standpoint? How do these concerns end up shaping a project? Hopefully some examples can help illustrate that.

Example 1: Konbit

Greg Elliott and Aaron Zinman, two students at the MIT Media Lab, noticed a major problem with the way reconstruction efforts were being approached after the 2010 Haiti earthquake. The issue was simple: Haiti lacked the information infrastructure needed to effectively identify local skill sets at scale and hire Haitians to assist in the rebuilding tasks.

For instance, instead of hiring a local plumber or electrician, someone might be flown in from the United States to get the job done. Plumbers, bricklayers, drivers, nurses, and translators all live within the crisis areas, but without a way for foreign NGOs to easily discover them and their skill sets they simply aren't hired. To make the problem more challenging, access to the Internet in Haiti is uncommon, meaning a purely web-based solution would offer no help at all. Additionally, 50% of the country is illiterate, so even SMS-based solutions are not appropriate.

They created Konbit to address this problem. It provides a phone-based interface that allows Haitians to register their skills and life experiences via an automated interview process. The interviews are then translated and categorized, resulting in an online searchable directory that employers and NGOs can use to discover local workers. They have more than 3,000 workers ready to be hired right now and are looking for NGOs and employers who can use their database.

Example 2: Ushahidi

While Konbit focuses on the rebuilding phase of a disaster, Ushahidi, a 2008 Knight News Challenge winner, has proven how powerful an information-mapping platform can be in the immediate response to a crisis. Within days of the tsunami in Japan, an instance of the platform was set up to track reports and needs from the ground. This particular map (see above) has an aggregation of reports with labels such as "Wanted!" "Disaster Area," and "Available Service."

As those of you who are familiar with the Ushahidi platform already know, it is brilliant because the open tools are general enough to be easily and quickly adapted for use in new situations. Information transfer based on geography is going to be needed in any crisis situation and many non-crisis situations as well. This helps separate the technology from the context, which means that at the very least a general information flow can be quickly set up almost immediately after disaster strikes.

Example 3: Grassroots Mapping

One of the projects that has come out of the Center for Future Civic Media is a set of tools and techniques for community-driven maps called Grassroots Mapping. The tools allow individuals to create high resolution maps using what boils down to a kite and a camera. Unlike Konbit and Ushahidi, this project is much more focused on the documentation of geographic change.

The project had immediate application after the oil spill in the Gulf of Mexico in 2010, where thousands of miles of coastline were contaminated with oil over a period of several weeks. Satellite imagery gave a sense of the damage, but grassroots maps made it possible to create high resolution maps of the damage, which are now part of the public record for anyone to view and use.

Designing for a Need

I want to wrap up the post with a story about my own miniature attempt to contribute to the world of relief technology. As reactors were beginning to overheat in Japan, I heard someone comment on a desire to better understand what was actually needed and how they could help. That night I threw together a quick Google Maps mash-up with the hope to make it easy for people to help log needs and support organizations on the ground:

The next day I learned about the Ushahidi instance and put my project on temporary hold; a few days of rushed hacking wasn't going to save Japan and I needed some giant shoulders to stand on. One week later, it seems that the original need I was approaching -- making it possible for American donors to understand how and where they could help -- is still not being met. Ushahidi has the information buried inside of its maps but the interface is simply not designed for that purpose and the reports are in Japanese.

I want to tap into Ushahidi by creating a layer which can frame the information for charitable supporters rather than for NGOs and survivors. The goal is a system that turns information into action by helping people with resources understand where needs are being met, who is meeting them, and how they can help. In the meantime, though, I would love to hear about any type of relief technology that you have seen which stood out as successful or unique.

I want to share a little bit about how the Lab works from a student's perspective, along with some first impressions from my first semester. It should be worthwhile for anyone interested in media labs. For everyone else I'll be sure to touch on where civic and community media fit into the operation.

The Lab: A Newcomer's Guide

If you don't know much about the Lab, here is my go-to description: imagine Charlie and the Chocolate Factory. Replace some of the Oompa Loompas with grad students (the rest are robots), and most of the candy wonders with technological ones. This isn't as far off as you might think; we even have a glass elevator.

Now that you have the big picture, I'll explain some of the inner workings.

The lab is organized into entities called research groups, which accept students. Each group has its own focus, and is led by a faculty member. Group sizes vary, but as of this writing there are about 24 groups and 139 students in the Lab, so you can do the math.

The groups' focuses fall across a wide spectrum. For example, New Media Medicine aims to improve the way healthcare is practiced around the world, while Opera of the Future is redefining music for the modern age. My group, Information Ecology, hopes to incorporate interactions with digital information more naturally into our day-to-day lives.

All of this research is funded by a consortium of sponsors. These companies help foot the bill and in return they get VIP treatment and licenses to any IP generated during the time of their sponsorship. Hey Washington Post, where are you? Or other major news organizations, for that matter.

Every year there are two huge celebrations called sponsor weeks in which all of the students and most of the faculty hustle bustle without sleep to prepare all of their demos and show off everything that the Lab has been working on since the last get-together. There is no cramming involved at all, I swear...

In addition to being researchers, everyone is still a student. Masters students take five classes over two years. The courses can be from anywhere in MIT, although many first year students start with ones from Media Lab. The same faculty members that lead the research groups lead Media Lab courses.

The best courses are often lottery-based. For instance, How to Make Almost Anything is in incredibly high demand because after taking it you know how to make almost anything.

The Center for Future Civic Media

About five years ago, the Knight Foundation gave a sizable grant to the Media Lab. The grant funded a "center" -- namely, the Center for Future Civic Media, a safe haven for anyone interested in pursuing projects related to information and physical community.

Centers are different from research groups because they don't accept new students; instead they sneakily lure current students into their clutches. There are a few centers besides C4FCM, such as the Center for Future Storytelling and the now defunct Center for Future Banking. They all provide direct support for research that fits into their theme.

The C4FCM leverages the Media Lab in a way that research groups can't because it has potential access to everyone. It can attack a problem from dozens of angles at once. There is also a new research group that starts fresh next year called Civic Media, so really they have it all going for them.

For obvious reasons I have a second home in the Center.

So, is it any good?

Earlier this semester I found myself in trouble. I was working on my composites project for How to Make Almost Anything -- I was making a fabricated pet rock. A silly project, maybe, but I had to make something out of composite and I wasn't prepared to make an airplane. That isn't the point. The point is that I needed googley eyes, and it was 3:00 in the morning.

I sent an email to msgs, the Lab-wide mailing list, with few expectations. Within five minutes I had half a dozen replies from people who were still awake, still working, and had access to a stash of eyes that I could use. What a place!

The plethora of available eyes at 3:00 a.m. reflects one of the most important characteristics of the Media Lab: An almost universal appreciation for fun. This spirit makes the Lab one of a kind, and without it people would have a much harder time breaking away and trying new things. They definitely wouldn't work as hard to attempt the impossible -- you need to have fun if you're going to do something as stupid as that.

Before I started my time here I was warned by several students not to fall into the all-too-common trap of putting too much energy into projects that are just silly, goofy, and don't have real impact on the world. So far I have been too busy learning to sink much time into projects at all, but I understand the temptation.

There are many merits to this place -- it has more thought diversity, skill sets, and resources than you can shake a stick at -- but what sets it apart is the need for that warning. It is the fine line that everyone here walks. To do the best work you have to think like a kid living in a crazy person's body, but you can't forget your calling.

Oh, and the other thing that separates it from other institutions is Food Cam.

I'm sure you know what I mean. Sometimes people only feel like hanging out with the "core." Or maybe someone has decided that they like the group, but can't stand a few of its members which causes a rift. The dynamics of even a small group can drastically shift with a single addition or removal. More often than not, butting out and letting things unfold on their own is the only safe bet.

In my experience, there is no such thing as a "one size fits all" group, even among friends. The ideal shape and nature of a group changes depending on the context and whose perspective you are considering. If we want social media to evolve into a more meaningful form of dynamic group interaction, fluidly defined communities need be seriously thought through.

Let the Pondering Begin

A community doesn't have to be social to have personalized borders -- physical communities and informational communities have their own unique flavors of fluidity. The more we can understand and reflect individual views of the interactions we host, the more relevant and engaging our systems will be.

• Location-Based Communities
  

  Geopolitical regions are nice, but not all locative information fits this kind of organizational model. Unless we're talking about government or a large-scale event, I'm far more likely to care about the happenings, thoughts, and commentary of my block and immediate neighborhood than the collective buzz of my entire city.

  I'm guessing the same could be said for my neighbor, and my neighbor's neighbor, and so on -- at some point that path ends up outside of my "care zone." Each person along the way has a slightly different circle of interest. A lot of those circles overlap with mine, and that overlap defines a fluid physical community -- my neighborhood.

• Knowledge-Based Communities
  

  As a programmer, there is no question that I learn from my peers. I grow by discovering the knowledge that PHP developers, SQL gurus, and other specialized members of the traditional "developer community" have to share. At the same time, however, I benefit hugely from interactions with journalists, designers, managers, entrepreneurs, and academics. Together, these people define a personal knowledge community just for me.

  Knowledge-based communities often surround a common interest, but they go a lot deeper than that. A person's ideal information pool needs to hone in on their specific interests, but should also incorporate the more elusive "second degree" interests (e.g. technology and entrepreneurship; snowboarding and meteorology) in the name of breadth.

• Network-Based Communities
  

  There is a field of Mathematics called graph theory. Despite the name, graph theory has nothing to do with the charts you learned in high school. In this case, graphs are "mathematical objects" which, believe it or not, are pretty simple (think connect the dots from back in kindergarten). They are used to represent relationships between objects through "edges" (lines) and "vertices" (dots). Graphs are a great way to represent groups: the group members are vertices and the relationships between those members are edges.

  Below is an example of a small graph which represents one person's group-based community (Person A). Maybe it depicts person A's best buds, where the edges represent strong friendships, or maybe it's a map of his workplace where the edges represent people who often talk to one another.

  

  Although everybody involved shares a common group identity, the graph tells a completely different story depending on whose eyes you look through. For instance, person C tends to hang out with A and B. Meanwhile A, D, E, and F form a tight knit group, but A and F also share a close mutual friend.

  In the real world, these graphs would be far larger, far more complex, and would vary far more from person to person. Suddenly Facebook and Twitter's tendency to bypass community and stick to networks makes a lot more sense.

The Tale of Clipt

Last year my friend Erek Alper and I submitted an application to the Knight News Challenge for a project called Clipt. The idea was a "digital conversation platform that hosts discussions grounded in bite-sized 'clips' of audio, video, image, and text that have been captured online."

Clips and posts would be submitted by users, tagged to locations and topics, and linked to each other. This content would then be displayed as interlinked bubbles in scalable, fluid interfaces called "clipboards" (see below for one of our early mockups to get a basic visual). These user-defined views of the system's global content base were created in terms of:

• Location: Adding a location to a clipboard was simply a matter of placing a pin in a map. The clipboard then contained content associated with an area around the pin, and more proximal content (thoughts and clips created by locals or tagged to a location) would get more exposure.

• Topic: Setting topics to a clipboard would narrow down what content was displayed. As the user added topics, the clipboard would become more and more relevant to a specific set of interests.

• Group: Users would be able to define a clipboard so that it would only show conversations involving specific people. Conversations involving more than one of those people would be more likely to appear on the board.

A piece of content that met all of the clipboard's conditions wouldn't necessarily be displayed if there were way too many matches. Instead, the interface allowed users to move along the "relevance gradient" -- zooming in to increase the focus (which would do things like "increase the required number of conversing users," or "decrease the physical radius of interest"), and zooming out to get see more general content.

Clipboards and their ability to capture relevance were our answer to the "fluid group" problem. The hope was that by defining filters for clips and thoughts in terms of location, topic, and group, our users would be able to follow and create tapestries of community conversation without resorting to rigid categorizations and boundaries.

Despite Knight's rejection, I still think the idea has great potential. That being said, I realize there are plenty of other ways to achieve personalized community. If you have any bright ideas of your own, please share a musing below!

That post sparked a lot of interesting comments, and it's clear that online community is something people care about. Within the comments I noticed two distinct camps: People who found the article through Facebook, and people who found the article through Twitter. Let's just say that I was surprised to learn that folks on Facebook are very defensive. (Twitter users at least thought the ideas were interesting, even if they didn't agree!).

Since I still feel that the current state of social media has barely scratched the surface of community-driven conversation and information sharing, I want to continue the conversation. This time, though, I want to begin the process of designing a community system that can satisfy my thirst for group conversation!

The Paradox of Digital Community

Community is all about shared experience and common identity, but the participatory web is all about personalization.

Why? One reason is because people won't participate if they don't feel connected to the content or conversation. Another is that without personalization, information overload kicks in and the experience becomes unmanageable and overwhelming.

It is no surprise that the most engaging and useful social systems try very hard to present individually-targeted content, resulting in unique experiences for each user. Unfortunately, however, this is exactly why it is difficult to find true community on these systems. In other words, lack of community isn't a fault of sites like Facebook and Twitter; it is just a result of the paradox.

Designing for Participation

I can think of three broad tactics used by participatory systems to try and manage the balance between noise reduction and social interaction:

1. Focus on content: Conversation is organized around specific pieces of information, such as news articles, blog posts, and other media. This solves the noise problem because personalization can happen at the content level, while the interactions within each piece of content can be universal.
2. Focus on niche: The system is set up around a specific organization, place, or topic. The experience is largely the same for all of the users, but the scope is narrow enough that only those personally interested will choose to participate.
3. Focus on users: Users define their experience on the site by making it about themselves, their networks, and their friends. This results in personalization with some inherently shared content. There are usually a few additional mechanisms that support a broader shared experience (e.g. retweets or public walls).

Each of these methods can lead to engaging social systems that host meaningful content, but they all have major problems that limit their community potential.

A content focus restricts group interaction to commentary and debate surrounding the content. Niche sites exist in relative isolation, meaning the participants and conversations generally come from within the niche and lack global perspective. Individual-centric systems create experiences that are as diverse as the individuals themselves (e.g. no two experiences are alike); and although some content is shared, it is often done so indirectly or among loosely connected, ad-hoc groups.

Community meets Personalized Mass Media

I said that I wanted to start designing something with this post. In particular, I'm looking for a way to do for community what social networks have done for the individual. As I just noted, however, it isn't enough to just host a niche. This system needs to be able to support networks of interconnected communities. This might sound like a social network, but it isn't -- social networks are made of interconnected individuals.

Below are a few ideas that I'm going to explore in future posts. The hope is that, once fully thought through, these will provide a starting point for the system I want to create:

• Fluid communities: Real communities rarely have rigid boundaries and static definition. Groups of people develop through complex relationships, and often have fuzzy borders.
• User profiles that are defined by participation: In real communities people aren't defined by profiles -- they're defined by actions. Online identities shouldn't just reflect a person's activity in the system; they should defined by that activity.
• Shared content: Information in this system has to be able to exist in multiple contexts at once. This will allow independent communities to exist independently without forcing them to exist in isolation.
• Individual influence: Whether it's users controlling their individual experiences or local communities making collective decisions about their groups' content, people need to be able to shape the information around them.

Please let me know if you have any other ideas that you feel would be important in a community network, either via a comment below, or via Twitter (@slifty).

Don't get me wrong, I wasn't crushed; I've never been an Apple fan-boy. My iPod Touch is gathering dust right now because I refused to cave into Apple's $10 milk-the-early-adopter patches (yes, even the one that enabled Apps).

Starting today, however, things will be different. I'm joining the Cult of Mac. I'm in the market for a Mini, and I'm going to buy an iPad. But there's a catch. I'm not doing this as a consumer; I'm doing it as a developer. The iPad can save the news industry, and I want to help.

The Simpler Days of Killer Apps

It seems like only yesterday when you required the fabled "killer app" in order to unleash the full potential of a technology. At some point, Apple decided that instead of worrying about spending time making these things themselves, they would just launch an App Store and let everyone else do the work.

Apple released the iPad without a killer app. This was a problem because we didn't just expect one -- we need one. We need this tablet to redefine the way people access media content in order to make newspapers and magazines viable in the digital age.

It's hard to believe Apple didn't know about our needs, and yet we are sitting here without a media revolution...Or are we? Maybe they just assumed that people would recognize the inherent potential of a lightweight, attractive, networked platform that comes with a relatively inexpensive 10-inch touch screen and an automatic critical mass. When you think about it that way, the iPad might not be so horrible after all.

The Potential of the iPad

Before thinking about what a newspaper-saving application might look like, here are four reasons why the iPad should be exciting for the news and publishing industries:

1. It is a giant iPod Touch: People say this like it's a bad thing, but for anybody who likes iPhone/iPod Touch apps, consider that user interfaces for the iPad don't have to fit on something the size of an index card. In the land of touch screens, extra space means more content, more features, and more potential.
2. Touching is better than clicking: Tablets aren't a new concept, but existing tablets tended to run awkward ports of mouse-based operating systems. Since Apple created the precedent for successful touch application platforms, I can only assume that this will be a ridiculously engaging way of consuming media.
3. It will be quick: By porting the iPhone OS to the iPad, Apple ensured the iPad would have the screen real estate of a laptop without its lengthy boot time. This might not sound like much, but instant gratification means you will be far more likely to use it for short periods of time (just like you might with a smartphone). You will pick it up at the breakfast table while sipping on OJ, sneak in some time right before going to sleep, and use it for a few minutes while waiting for the train.
4. There is an established distribution mechanism: Because the iPad uses the App Store, it will be easy to get our revolutionary applications out into the world.

What Would a Killer App Look Like?

Let's pretend that the iPad (or a similar tablet) will catch on in the next year. How can the news industry take advantage of this as an opportunity to revitalize itself? What would the solution have to do?

Below are a few things I want to see:

• A singular distribution channel. I want an iTunes for print. It is far more powerful to have one full-featured application that can serve a thousand newspapers than it is to have a thousand simple applications that host individual newspapers.
• Publications selling information again. No more of this "giving out your content for free" nonsense. Sell your daily paper through this platform for 10 cents an issue. This will be a viable business decision for two reasons: The convenience of the medium, and the insane richness of the consumer experience.
• Far more than just text. What the New York Times showed off at the Apple media event was nice, but we need to go a few steps further. Details will have to wait for future posts, but I'm generally talking about tools for contextualization, crowd empowerment, rich media, and personalization.
• Effective and powerful content creation tools. There will be two sides to this system: the consumer facing side, and the publication facing side. Publishers need to be able to create these dynamic digital documents quickly and easily.

Even with just these four attributes one can start to see how this might help local news organizations. If done right, people would buy the newspaper again. They would add the iPad to their morning routine and flip through through the media-rich pages at their leisure.

The reason this would happen is because this system will provide a more powerful experience than anything that exists today. The tablet interface will enable things that nobody would have dreamed of doing on a laptop or smart phone. The application would be accessible because of the consistent consumer experience that comes from a common platform (in the same way that people around the world read the newspaper in the same way 50 years ago).

I wasn't kidding about wanting to dream up, design, prototype, and launch something for the iPad that will save publications; but I can't do that without getting some publications on board. If there are any organizations or individuals out there that are interested in seeking the Holy Grail, contact me (@slifty on Twitter).

Why the disparity? Simple: I care more about community than myself.

I'm sure you've heard people talk about the ego-centric nature of today's social media, which tend to focus on one-to-one and one-to-many communication. Not only does the spotlight on the individual create an unappealing blend of "often boring" and "always noisy," but it also makes it essentially impossible to facilitate real community. In fact, even the systems that are designed for groups leave much to be desired.

We all want to inform, and form, our communities using today's digital tools, but how is this possible if the proper tools don't exist? If there isn't anything out there that can host community on the Internet without sacrificing something important, then maybe it's time to invent something new. I've been thinking about where community stands at this stage of the digital era, and what these new tools, or tool, might look like. Here are some of my thoughts.

Charting Digital Media

I'll try to simplify things by visualizing the current state of social media in terms of "focus" and "scope." (See image below.) The location of each icon on the spectrum is subjective, so don't ruffle your feathers if, for instance, you think Twitter should be closer to the "group" side. The point is to get a sense of where existing services might fall, and start thinking about the costs and benefits of each quadrant.

Focus (Individual vs. Group) describes the type of social interaction users engage in on the system. Is the tool's functionality geared toward private conversations or group discussions? Is content sharing ego-driven, or is there a focus on discussion? For me, if the system is primarily designed for one-to-one or one-to-many communications, then it is individual focused.

Scope (Niche vs. Global) explains the type of people found on the system. Is the site universally attractive, or is there a well defined target audience? Will users tend to find information thanks to common interest, or will they be exposed to a wide range of perspectives?

This setup creates a framework for thinking about online services. Here is what I was thinking about when I tried applying it (why I put the icons where I did):

• Social networking sites (Facebook, MySpace, and LinkedIn) allow global users to contact one another directly, and create a detailed digital identity. These services provide effective one-to-one communication tools, such as private messages or wall posts, but the group-oriented features often feel shallow and impersonal.
• News media repositories (Digg, Reddit) let groups share and discover content through collective intelligence. They provide a space for many-to-many conversation, but tend to aim at a global audience, since they rely on network effects to achieve a critical mass.
• Personal media publishers (YouTube, Flickr, Twitter, blogging platforms) make it easy for users to get their message to the world, and therefore focus on one-to-many communication.
• Discussion Platforms (Ning, PHPbb, and niche community websites) facilitate communication by creating a space for groups to use. This makes it possible for niche communities to function, though they do so with an element of isolation from the global community.

Looking for Trends

There are a few traits to consider within this framework:

The Noise to Information Ratio: How difficult is it for a user of the system to get the information they want? Intuition tells me that global services have more content, but they also have a better chance of getting enough users to support collective intelligence -- and collective intelligence can help the system route information more efficiently. Niche services, however, have less content and a more specific audience, so there will be less noise in the system.

The Value of Contribution: A participatory system can only exist if it has users contributing content. It seems to me that it is much easier for a user to "freeload" on the activity of others in group-oriented sites by lurking in the shadows. On sites with an individual focus, the user won't get nearly the same experience if they don't interact in some way.

There are also some unscientific generalizations to be made about the four quadrants.

• Global-and-Individual (upper left) leads to popularity. There is a lot of interest in being able to share your voice to the world, and these sites do exactly that. These are the places online where an individual could become a superstar, or at the very least feel important. If I were a psychiatrist, I'd probably be able to make an argument that people flock to these sites because they secretly like themselves a lot, but I'm not, so I won't.
• Niche-and-Individual (lower left) promotes personal relevance. If a user chooses to participate in a niche-content system, they presumably belong to that niche (or aspire to). If the interactions are individual-focused, they are probably applicable to the individuals involved. This adds up to a system where most of the messages are naturally relevant to the people that see them.
• Global-and-Group (upper right) creates and organizes knowledge. There is something to be said about the crowd's ability to organize information. When you have a global user base behaving as a collective, there is huge potential for the creation and organization of knowledge.
• Niche-and-Group (lower right) facilitates community. Community requires group interaction with an underlying common identity. These sites provide space for exactly that.

Dreams of the Future

Community tools exist, but they are drastically underpowered. The systems lack the popularity of Facebook, the societal potential of Wikipedia, and the personal relevance of email. As a result, they are drowned out by the far more successful alternatives that I outlined above.

To change this, we need something that can:

1. Host niche communities without isolating them from the rest of the world.
2. Give individuals a chance to shine without letting their egos dominate the content.
3. Attract enough people to drive collective intelligence, while maintaining the level of granularity needed to provide a truly personalized experience.

That isn't too much to ask for... right? I personally believe that these systems will be the key to meeting community information needs. As such, I believe this is the direction that news organizations need to move if they want to maintain/reclaim their role as community informer.

October 12 was a day of high emotion; it was finally time to thrive under pressure. I got home from work, rushed to my friend's house, and cracked open my laptop. The goal was to brainstorm like crazy, write up some solid project descriptions, and submit as many Knight News Challenge grant applications as possible over the three days I had left. Thank goodness fate had a better plan: the deadline was extended.

Now that we all have another two months, I'm going to take a few steps back and try to combine my formal education in information systems with my Knight-sponsored crash course of journalism's ongoing transformation and the lessons I've learned from three years of applications. I hope that the resulting guide will help you come up with some good ideas of your own. Remember, there is no limit to the number of applications you can submit. (Note: I won a 2007 Knight News Challenge grant for $15,000 to blog about
"connecting people, content and community.")

Well-designed solutions require well-understood problems. When brainstorming ideas for your News Challenge application, the first step is to write down a long list of complaints. (This also works if you already know what you want to do.) What do you want to change? What isn't working well today? What isn't happening that should? This list will give you a starting point, and will make sure that the ideas you come up with will have a real purpose.

Complaints might start from your own personal experience, but you need to expand from there because the true gems come from a fusion of imagination and research. Here are some ways to gather information:

• Talk to people. If you are looking for new problems, ask people you know what they would change. If you are looking to better understand one that you have already identified, ask if the problem exists for them and why. Their insights could provide the spark you need to turn a potentially good opportunity into a great one.
• Pay attention to buzz. Read what other people are saying about your issues. I can guarantee that as you read this there are members of the blogosphere discussing their own trials and tribulations with new media. This will help you get a general understanding of how people fit into the way things work, where they see opportunities for improvement, and which direction the crowd is moving.
• Know the current process. You can't change something without knowing what it is you are changing. Even if you plan to completely redefine the status quo, you need to appreciate and learn from the way things work now. There will always be something worth incorporating or maintaining.
• Explore the cutting edge. What are the front-runners doing, and what problems are they addressing? The cutting edge is known for being risky because nobody is sure of the best solution. Look at the problems that they are addressing and add those to your list. With any luck, you can think of a better idea.

The more you understand the tools available to you, the more effective you will be at finding creative applications for them. Ideally, you want to get to the point where you can have an intelligent conversation with a programmer, but for now it is enough to just have higher-level knowledge.

For each technology you think you might be able to use, figure out:

• How it works. While you don't have to be able to program, you really should have a general idea of how the magic happens. So long as you know what the tool is called, you will be able to find an accessible guide. Just remember that Google is your friend here.
• What it does. Tools tend to have an intended purpose, although often it's a very broad one. Be sure to understand what that purpose is. You can start by looking at its website and see what its creators say. Also try to find out which existing sites have used it, and examine what they have done.
• What it could do. Once you get a basic understanding of a tool, you can start to get creative and think of ways to use it that its creators never would have thought of. This task is all on your shoulders, but you can always scour tech blogs like TechCrunch or SlashDot to find examples of how people can push the limits.

If the technology is at all popular there is probably a community surrounding it. Once you find it, create an account and join the party; there will be people willing to help you learn.

You have a list of tools and a list of goals, now it is simply a matter of creative application: find out how to achieve those goals with the technologies available. Brainstorm as many solutions as you can for each problem, and be sure to dream a little bit here.

For each solution you need to be able to explain:

1. What it would do and how it would be used.
2. How it could fit in and what it could change.
3. How it would incorporate technology and/or people.
4. What assumptions would have to be met and how you would meet them.

You will probably find yourself coming up with new tools, new processes, or (more often) a combination of the two. If you're having trouble, try looking at how people are solving problems in completely different fields. Maybe you can learn from their work. Just remember that you don't need to know how everything fits together just yet.

This is when you descend from the land of the theory and optimism and take a close look at the world. You have some ideas already thanks to your list of solutions, but there are plenty of others to be found; plus, not all of what you have will work. What looks promising? Which ones can you cross off the list?

Some things to think about:

• Who are you dealing with? You need to understand your stakeholders (i.e. the people who would be impacted by your project). You want your solution to provide them so much value that they will be willing to donate time -- and maybe even money. At the very least, they need to be willing to try it.
• What is your competition? There will be direct and indirect competition, but you need to know about both. This will help you differentiate your idea from what is already out there, and it can also force you to further develop your solution into something even better.
• What resonates personally? You are going to need to explain why you are the person to take this idea and make it a reality, so figure out what you bring to the table and make sure you can get excited about it. It won't be enough to say, "I thought of it!"

To get funding from the Knight Foundation you need to be able to convince the world that what you have is a genuine opportunity. It should have the potential to redefine landscapes.

Take your most promising solutions and try to envision their implementations. What is going on behind the curtains? What kind of synergy can you create? Even though you are getting more concrete, don't get bogged down in unimportant details -- you are still brainstorming here.

Think in terms of process (how things get done), objects (e.g. "news article," "user contributed question," or "media clip") and user roles (e.g. "journalist," "editor," "consumer," or "judge"). How will the objects interact? How will each role fit into the system? Refine this line of thought with your previously researched understanding of your stakeholders, the current state of affairs, and the technologies available.

By now you've probably conjured up something spectacular, and filling out that first round application will be a breeze. With any luck you will have enough inspired thought to submit more than one. Good luck!

Last week I visited Carnegie Mellon University's website for the first time as an alumnus. The front page, often dedicated to highlighting faculty work, had a picture of an iPhone screen displaying brightly colored data visualizations. I didn't have to look past the first two words of the title -- "Citizen Scientists" -- before I knew that it would be worth my time to keep reading.

The article described how Eric Paulos, an assistant professor in the Human-Computer Interaction Institute, is equipping "everyday mobile devices" with sensors used to collect reliable scientific data. The point of all this effort is to create "a new generation of 'citizen scientists,' connected both to the environment and each other." Just in case that quote isn't clear enough: the study might as well have been funded by The Knight Foundation.

What's interesting about this project is that members of the scientific community, a professional group that arguably maintains higher standards for verification than journalism, are trying to harness the crowd in the same way that we are. In fact, they are actually going out of their way to do it. This demands an explanation!

At the end of July, an amateur astronomer found a giant hole on Jupiter. Some people were surprised that the ocean-sized crater was discovered by a hobbyist instead of a professional. I, however, wasn't shocked. My reaction could be the result of a personal perception that astronomy is relatively accessible to "the common man." Or maybe it's because looking to the sky is such a primal thing to do. Either way, the subsequent media dialogue exposed me to several new spins on the "amateur vs. professional" debate.

A few days later, I saw a follow-up story on Digg about another amateur observation. This time someone found a bright spot on Venus. It was admittedly less interesting than the news from earlier in the week. I scanned the comments anyway and saw a highly rated one that asked, "Why is everything first discovered by amateur astronomers? What the heck are we paying the professional astronomers for?"

Using my recently gained knowledge of the issue, I replied and pointed out that professionals have bigger and better things to do; it doesn't make sense for a PhD to use a million-dollar telescope to look at something that a hobbyist could view using a thousand-dollar one, especially when there is so much of the universe left to unlock.

Once again, there seems to be a lesson here for journalism.

It's important to know where amateur and citizen scientists fit in relation to professional ones. Here's how I define these three broad classes of scientist:

• Professional Scientists - These people make a living from science. They have the expertise, the patience, and the resources to handle the big stuff.

• Amateur Scientists - Folks who tackle science as a hobby. They enjoy participating and have varied levels of knowledge. They also have the capacity to make direct contributions to the scientific community because they're exploring the same reality as everyone else (regardless of what quantum physicists and philosophers might say).

• Citizen Scientists - These individuals are equipped to contribute to science when they are empowered by tools and networks. They aren't out conducting experiments or learning methods, but they are generally willing, with the help of professionals, to provide crowd power and reap the benefits of the resulting information.

Here's the interesting part: between the story of the amateur astronomers and the vision of Eric Paulos, all three types of scientist have beautifully compatible relationships.

Professionals can safely focus on daunting tasks, knowing that amateurs are ready and willing to take on the smaller stuff (like keeping tabs on Jupiter). The community standards are clear and ultimately bound by cold hard observable fact, so amateurs can make meaningful contributions without diluting the knowledge base. Meanwhile, citizens are being empowered by professionals to help the scientific cause in a way that informs individuals and improves their lives.

Now it's time to take the leap back into the land of journalism. If you buy my claim that scientists and journalists all care about informational integrity and the quest for truth, then several things can be extrapolated:

1. If professional journalists take the lead by clearly defining expectations, explaining best practices, and implementing an accessible infrastructure, then amateurs can contribute without disrupting the industry.
2. If amateur journalists do a good job of covering a smaller scope of topics or areas (e.g. the hyperlocal), then professionals can focus on the deeper, otherwise inaccessible issues.
3. Professional journalists are responsible for creating and maintaining the citizen network if they want it to meet their standards.
4. Citizen networks need more than a host. In order to reach full potential, they need to be explicitly empowered through tools and guidance.
5. A symbiotic relationship between the professional, the amateur, and the crowd is not just possible, it's socially optimal.

And there we have it: If the journalism industry can create an infrastructure that allows amateurs to contribute reliable information, then professionals will be able to dedicate more resources to epic reporting. If local papers can find the capacity to set up and empower meaningful citizen networks, they will establish a major foothold in the evolving domains of community and information. Man, science is useful.

Now we just need to define the standards (and explain how to meet them), create that infrastructure, and set up those networks.

iPhone image via Carnegie Mellon University.

But wait! It's a trick question, and not just because there are countless examples of all classes of reporting from both mainstream and creek media. The trick is epistemological: The existence or non-existence of these qualities on either side is practically meaningless if nobody can prove they exist to the audience.

What makes this trick important is that a person who doesn't believe in the benefits of mainstream media probably won't do much to support it, and I worry that too many people don't. That's the bad news. The good news is that technology can help newspapers, and all other deserving entities, earn trust among a generation that includes skeptics with access to a world of information.

Some Non-Representative Opinions

During the conference at MIT last month I decided to ask some back-at-home friends how much they value professional journalism. The results aren't exactly uplifting, but obviously they don't mean much -- I doubt Gallup would back my informal "methodology." I'm also betting a lot of the people I asked were thinking of network news when voting (one person commented "I voted for 5, but would vote for 1 if Wolf Blitzer [were] not included in that category.")

The reason I mention this private poll is because it reflects a small-but-still-too-large sentiment of mistrust toward the mainstream media that I've noticed among peers and Internet brethren. I'm not old enough to know if this is new or if it's been around since the dawn of time, although I assume the latter. The truth is, I also find myself finishing articles only to wonder what wasn't said; and why shouldn't I?

Fox News shows how easy it is for a media source to shape people's understanding of reality on a massive scale. Goodness knows I don't want to become one of those sheeple things I hear so much about.

The main issue is that most mainstream media is lumped together with political pundits, polarized reporting, editorial echo chambers, and infotainment-driven news cycles. Is it fair? Maybe, maybe not, but either way it's something that must be considered: In the eyes of some, The Watchdog has become The Man.

Defend Your Credibility

Clearly a business model or distribution channel that prevents free riders would be nice, and providing so much value that everyone starts subscribing again is also a great plan, but what about all the people out there who would vote for option #4 in that poll? It seems to me that since professional journalists are doing what the industry says they do anyway, it's time to use technology to leverage that effort and eliminate any room for doubt.

Here are some thought stubs about ways to show the audience why they can trust a specific news source. I may explore them in more detail in future posts.

• Make your source documents public - I was so excited when the Obama Administration made it easy to read all the official documents on whitehouse.gov, until I realized I didn't have 5 hours a day to read them. What I want is the ability to read a reporter's analysis and have the means to check their work and form opinions of my own. (Word on the street is that somebody awarded a grant to develop a technology that could help with this effort...)

• Provide Timelines and Notes - How much time was spent on this article? When did the research begin? What contextual factors influenced its development? Did anything get cut out that might still be worth showing in a footnote? The more information you can provide about the creation process for the audience the better.

• Share Author and Editor Information - I know that disclaimers are provided when appropriate, but the more information about the people who shaped the analysis, the better. What else has this author written? For who? Where does the editor fall on political issues? Maybe some of this is none of my business, but then again, maybe it is.

• Reference the Past - If you know the context of a new story, chances are you documented the context in the past, and maybe you even have some archived reports about it. Link to it. Even better, make a technology that allows for line item comments on old articles so that people can understand how it fits in. There is a lot of potential here.

• Track your News Trends - What if I could see a visual summary of the topics covered by a news organization over time? Then I could know for sure how much was spent on crime, entertainment, world news, politics, etc. Ideally this would also show me what the specific trend was about - did CNN spend more time on Michael Jackson than Iran? I don't really know for sure, but I'd like to.

Now that I look at it again, this might be the sort of added value that actually matters anyway. It differentiates those with time and resources from those without it while bettering the quality of the product. Sounds like a 2-birds-for-1-stone situation!

I hate to think that a newspaper can't take advantage of free, open source, low hanging fruit simply because it would create another system that has to be taught and maintained! At the same time, though, I very much appreciate the position of the incredibly stretched-thin tech guys. This post is about better understanding the "why nots" that discourage newspapers from adopting new technology and trying to figure out if there is any hope of getting around them.

The Barriers

In hobby-land, there aren't too many reasons not to play with technology. Nerdy people set up -- or program from scratch -- wikis, forums, and brand new systems just for fun all the time. Their peers are generally savvy enough to use the stuff without instruction and, although terribly sad, a "datapocalypse" won't cost anyone their job. That's probably why cutting edge is often discovered in the garage -- garage dwellers have much less to lose.

Newspapers need to innovate the same way garages do, or at the very least reap the benefits of innovation, so why is it so hard for them to do so?

• Cost of Setup -- I've mentioned legacy systems before; here is a place where they can constrain. If there is any chance that the new won't play nicely with the old, then systems administrators have to take extra precautions lest they break something that's being used. Worst of all, it's always possible something will go wrong anyway.

• Cost of Implementation -- Do databases or servers need to be set up? Does software have to be installed onto multiple computers? Do accounts have to be created for every user? What settings should be tweaked? Even if the software is free, getting it ready to use by a whole team of people might not be easy to do.

• Cost of Maintenance -- I'm told nothing works perfectly, although I hate to believe it. When things that employees use break, someone has to take the time to fix them. Every new tool hosted in-house is another thing that could go wrong and take a day to fix.

• Cost of Backup -- In the words of Tim Dunham, the CTO at the Post-Gazette, if a pet system becomes mission critical, it has to be treated like a mission critical system. This means that if a wiki containing organizational knowledge is set up and relied upon, it can't ever go down and the data can't ever be lost. Having a backup and recovery plan in place becomes essential, and that takes resources.

• Training and Learning Curves -- After hearing some horror stories about even the most minor software changes causing confusion, it seems reasonable to expect trouble when pushing people to use something completely different from what they're used to. How will everyone learn about it? Will they have to be taught? Is there documentation? Is the interface straightforward? And the answers to these questions will probably lead to more work.

• Functional Overlap -- It is possible that the need that will be addressed by new software can already be partially addressed by the software that is already set up? In some cases this means that the new idea really isn't worth the effort, but in others this line of thought might just be an inner excuse to avoid costs and risks associated with adopting new technology. Either way, you don't want to maintain two systems for the same task because it will create miscommunication and general confusion.

Some Workarounds

The above list may seem depressingly long, but never fear! There are ways to fake flexibility and nimbly try new things. Although I can't speak from experience, I'm going to throw a few ideas out there -- take 'em or leave 'em.

• Start Small -- Since the dawn of time, people have found ways to lessen the blow when dealing with large scale projects: developers make prototypes, web applications have closed betas, and cavemen probably made miniature wheels before trying full-sized ones. Not everything needs to be launched full-featured and full-scale up front. Pick a small group of people to try the new internal project first, or set up a smaller portion of the feature set, or just use default settings instead of spending hours tweaking to perfection. Doing these things will give you time to work out kinks, get feedback, and figure out how/if the new service might be used before you spin your wheels.

• Try a Web Service Instead -- Using an external product is a risky idea for many reasons, and I wouldn't suggest doing it for anything that involves trade secrets, information storage, or truly vital processes. But if it is possible to use a service someone already provides for free or for cheap, it's worth giving it a shot. The positive is that you can probably get it up and running in less than an hour and you don't have to worry about maintenance. The negative is that you lose control and increase risks -- what if the service dies off? What if someone steals data?

• Create an Experimental Environment -- You don't need much computing power to host an internal service. Heck, my 9-year-old laptop could probably serve a wiki or chat server without much trouble. Set up a place where techies can try new things without being worried about important tools breaking or data being lost.

• Run an Inventory and Cut out the Fat -- One thing that nobody wants is technological bloat -- i.e. having more services than are needed. It makes things confusing for everyone and creates unnecessary maintenance overhead. Take some time to go over the services currently available and what needs they address; try to find opportunities for improvement, consolidation, or system retirement. If this means you need to temporarily move backwards to open up resources for new opportunities, so be it.

• Centralized Documentation -- Having all documentation, guides, and general commentary in a central location is an incredibly useful way to share technical advice. If done right this will lower the time spent answering common questions and provide a nice way to communicate new information as new tools are launched. I would recommend a wiki for this, but there are definitely other ways.

• Automate and Streamline -- Menial tasks in the tech room are probably already automated. For instance, hopefully backups aren't being done each night by hand. If a chore is taking up too much time, try to automate it so that the whiz kids can work on bigger and better things. It is also worth thinking of ways to do this flexibly to save time down the road when additional steps need to be added.

Final Thoughts

The missing section of this post is a list of tools to try -- I'm hoping that you guys and gals could help me fix that in the comments. The point, though, is that newspapers need to have the freedom to try new things in a way that doesn't add much to their technical overhead. Regularly incorporating technology to improve day-to-day operations is incredibly important for the future of reporting.

It is also important to keep in mind that not everything attempted will work, and not everything will get used; just remember that if shots aren't taken, nothing gets hit. The risk of an individual project failing -- once again, many will -- isn't so dire if the setup costs are low. This is why starting small is so important -- it allows you to throw a lot at the wall and see what sticks.

Seriously, though, any suggestions, success stories, or tool ideas would be greatly appreciated!

The Spirit of Innovation Lives Strong

I have to start by commenting on the number of good and, maybe more importantly, innovative ideas that are being considered and pursued at the Post-Gazette. From day one it was exciting to see the plans that were already in motion. Put bluntly: many of the people at the Post-Gazette get it. They understand what the Internet is about and the potential that it brings for them and the communities they serve. Better yet, they even have some great homegrown ideas about to how to take advantage of that understanding.

I'm betting that they aren't the only ones, and if that bet is any good then neither the Post-Gazette nor the industry is suffering from a shortage of creativity, an inability to understand the potential of new technologies, or unwillingness to try new things. Yet if you look at the web sites of a lot of these papers, (yep, even the PP-G's), you would probably think that my previous claims of support were just an embarrassing attempt to kiss up.

You'll have to trust me when I say I'm telling the truth... Now the question is: what is preventing the changes from happening?

Some Major Disadvantages

Everyone reading this knows that newspapers are in a pretty uncomfortable position. Even after ignoring fiscal concerns they are under pressure to add substantial new features along with new processes and they can't afford to wait until precedents are set (since by then it could be too late). To make matters worse, they have to publish a paper every day.

In other words, they face almost all of the problems that tech startups have to tackle with a few added bonuses:

• Minimized Flexibility - Assuming the org already has a web presence it isn't working with a clean slate. The existing "legacy" system - be it in an in-house creation or some kind of purchased software - represents a past investment and provides vital functionality. The problem is that it also constrains improvement. Realistically this means that any new stuff will probably include awkward workarounds, an inability to integrate smoothly, or blatant loss of functionality.

• Doubly Limited Resources - From what I hear, software development is costly when you don't do it as a hobby, especially when you are trying to be cutting edge. This is restrictive enough, but it becomes even worse when these costs, combined with a tight budget, force shortcuts. For instance, having to buy existing software that only gets you part way there (and can't be easily extended), or creating one-time widgets instead of being able to spend extra time to make something reusable.

• Newspaper Overhead - It's easy for me to forget that papers don't exist to innovate; they exist to report the news and in doing so serve their community. As a result, tech people have to spend a lot of time on more mundane day-to-day support tasks instead of designing and developing big changes. Constantly looming newsroom deadlines also make it difficult to fully incorporate some aspects of good software design that require stakeholder (i.e. non-programmer) participation, such as focus groups, feedback sessions, and user tests.

Is There Hope?

Yes, yes, a thousand times yes. Despite these hurdles and the ones I'm not mentioning, papers actually have some pretty epic things going for them. For the same reason they have legacy systems, they also have legacy assets. Here are some things that money can't buy:

• Established Audience - I was told that the phrase "If you build it they will come" doesn't apply to the Internet. This is absolutely true for Dan the Programmer, but if you are a mainstream Newspaper you have the power to direct readers to check out new features. You get that all important shot at injecting life into a new-born system. In other words, if you build it they will come (that doesn't mean they will stay, of course).

• Community Allies - If a paper has played a positive role in their community for any significant period of time, it is not a lonely island. Local groups are out there that would love to join in on a mutually beneficial, innovative relationship. This means that if there is a new community service that the paper wants to provide, it can surely get some help.

• Content and Knowledge - Content may not sell, but it certainly adds value. Furthermore, papers have a tacit and deep knowledge of the communities they serve; something that stupid Facebook will never be able to snag. Since "know thy audience" is just as much a tenant for software design as it is for journalism, this is quite an advantage.

• A Very Dedicated Work Force - Newspapers are one of the few kinds of organizations that are literally awake 24/7. This speaks to the extent that the people there care about the paper's mission. Since the technological success of the paper is now a part of that mission, they presumably care about that too. Reporters and editors may not be able to program (which is perfectly fine, I might add), but that doesn't mean that they can't be involved in the improvement of the paper's online presence.

Some Final Thoughts

I'm curious to know what other disadvantages and advantages people see for papers - be it from experience or speculation. Please throw your $0.02 in the comments!

I'll wrap up with some random pieces of general advice, in no particular order:

1. Get feedback from everyone. If any significant development is made without actively soliciting feedback from everyone on staff, before, during, AND after implementation, then the paper is missing out on hugely important insights.
2. A great idea can fail miserably with a bad interface. You might have the best idea ever for a participatory system, but if the implemented U.I. isn't easy to use or is hard to find, then it will never reach full potential.
3. Find someone that has an eye for good on-screen visual design and do lots of user tests before releasing. This role could probably even be filled by a few current employees instead of an outside hire. Just remember, even if you think the product looks and feels amazing, chances are it is unattractive and difficult to use.
4. If you invest in out of the box software, make sure it can be easily extended to do what you want it to do. If it can't then you are doomed to either come short of your goals or spend more time finding a work around than it would have taken to code everything from scratch.
5. Open source doesn't mean inferior. I'll say it again: open source doesn't mean inferior. Part of the reason for this is that open source products often have awesome support and development communities by definition.
6. Don't let programming languages or similar concerns prevent you from adopting a better technology. Make your techies learn something new. The week or so that it takes for them to ride the learning curve will cost far less than a bad decision.

The Million Dollar Questions
The first thing you learn in an I.S. class at Carnegie Mellon is that before you can effectively design a system you need to understand three things:

• The People - Who will be affected by this system? What are their needs and desires? What are their current roles?
• The Process - How do things work now? What can we do better? What do we want to do? What has to be done? What are our limitations?
• The Technology - What technologies can we work with? What can they do naturally? What can we push them to do?

These are the high level questions you should be asking when planning to do anything technology related, and you should have the answers long before a dime is spent on programming or design. If you dart into a project carelessly and/or wildly flailing your arms then chances are high that you will end up with an expensive failure.

Six Steps for Success
Addressing those questions isn't easy; in fact, I'm sure it is much like writing an investigative news report. You enter the process with a high level concept and you need to dig down into the details, adapting your vision along the way. The key, though, is that it is much cheaper to adapt before there is code to change. With that in mind, here are the steps that will help you get ready to develop.

1. Identify stakeholders - Stakeholders are "people who will be affected by the project or can influence it but who are not directly involved with doing the project work." This might be readers, reporters, editors, community members, etc. The list will be different for every project and you will probably add more as the idea develops. Thinking in terms of stakeholders makes it easier to keep the needs and backgrounds of different kinds of users in mind, which in turn makes it easier to design an information system that will be helpful to everyone.
2. Gather requirements - Requirements are "statements that identify a necessary attribute, capability, characteristic, or quality of a system in order for it to have value and utility to a user." In other words, what does your system have to do? This includes functional requirements (e.g. users need to be able to upload and tag photos) but there are also non-functional requirements (e.g. the website needs to process all requests in less than 5 seconds). You'll have your own ideas, but never assume that you have all the answers; after interviewing stakeholders you will realize that their needs are different from what you would have expected. This is, by far, the most important part of a project's lifecycle since it will shape just about every decision; unfortunately, it is also the easiest to botch simply because people aren't good at clearly explaining what they want.
3. Understand constraints - Constraints are "restrictions on the degree of freedom you have in providing a solution." How much money can you spend? Do you have a strict deadline? Can you hire people with new skill sets? Is your website already using a particular piece of technology that you will have to keep supporting? All of these are limitations which need to be considered while deciding on a final solution.
4. Research technologies - There might be something out there that can help you do what you want to do; in fact, there might be something out there that does what you want to do. Each project will have its own unique technology needs based on the requirements and constraints you identified, and researching the landscape will enable you to select the best solution for your special case. This can save a lot of time and money down the road, so please, do your homework and avoid the kool-aided urge to skip this step and just jump to the latest buzz-on-the-street tech! Some things to consider while looking at a technology: What needs of yours will it meet? Is it overkill? How new is it? Is there a support community built around it? What are people saying about it? What are developers saying about it? What other sites are using it? How are they using it? Will you have to hire or train people?
5. Brainstorm solutions - By now you know what you need to do and you know the tools that are out there. Figure out all the different paths you can take to get where you want to be without violating any constraints. For each item on your list come up with pros, cons, and (if possible) estimates for the amount of time and money the solution would cost. For instance, when creating a brand new website one solution might be to hire an external firm that develops Drupal systems. Another might be to hire Ruby programmers and create the site in house using Ruby on Rails.
6. Pick the best solution - You're going to have to choose one to run with. Before you do, though, minimize unanswered questions. It is very inexpensive to learn more early on - it is much worse to realize 6 months down the line that you made the wrong choice and could have finished the job with half the price or in half the time. Do you know that the technology can do what you want it to do? Are you sure your staff can learn the skills required? Will your stakeholders actually approve of this solution and use the darn thing? To help this effort it might be helpful to do a prototype, which is a quick, cheap version of your system designed to answer your questions and, in project management jargon, "mitigate risks."

Once you have done all this you still have plenty of work to do before anyone should start churning out code. This should get you started, though, and it will make it much easier to talk with developers and contractors down the line. Now, go hire an Information Systems student so you don't have to battle through this brave new world alone!