Idea Lab

ScraperWiki Lets You Make Magic Out of Web Data

by Nicola Hughes
2011 Knight News Challenge Winner

There's a wonderful magic wand that every member of a digital newsroom wants to get their hands on. Take control and you can work wonders, untangle the world wide web of information, and even decrease your workload to fit in that extra cup of coffee. "What is this wand?" you ask, and "How can I get my hands on it?" It's the wondrous API (application programming interface). At ScraperWiki, we provide the tools to custom fit your wand to your magical purpose. Learn a couple of incantations in either Ruby, Python or PHP and you can concoct an API of...

• January 25, 2012
• comments

ScraperWiki: Wrapped Up in Time for Christmas

by Nicola Hughes
2011 Knight News Challenge Winner

Come the holidays, we all like to do ourselves up -- a new frock for the party season, or a post-Movember shave. We all like to look our best in preparation for the Christmas glut. This extravagance now extends to the web. But instead of adding a bit of snowfall, ScraperWiki has driven that further mile and added a whole host of UX features to our site! ScraperWiki is a developer platform that aims to liberate data from the web, build upon this information to make useful applications, and get journalists and developers working together in a wiki-like fashion. There...

• December 14, 2011
• comments

With Tools, Tables and Tours, ScraperWiki Wants to Liberate Data

by Nicola Hughes
2011 Knight News Challenge Winner

As part of the Knight News Challenge entry, we at ScraperWiki said we would roll out Journalism Data Camps across the U.S. We had done what we called "Hacks and Hackers Hack Day" events across the U.K. and Ireland, bringing journalists and coders together. This happened at the same time as HacksHackers in the U.S. -- great minds and whatnot! Now we're scaling up when it comes to exploring the data prospects of the new world. We are heading across the U.S. on a data liberation front. But where do we start, and where do we go? Well, firstly...

• November 7, 2011
• comments

How Data Can Become an Evergreen Source for Newsrooms

by Nicola Hughes
2011 Knight News Challenge Winner

Newsrooms don't fear too much news. They fear not enough news. With news on demand 24/7, the stream of information that journalists work with is becoming the commodity upon which they rely -- which is why "evergreen" stories are becoming a staple for the modern newsroom. What they need now are evergreen news sources. So how can data be an evergreen news source? Traditionally, data was hard to work with. It had to be collected, cleaned, organized, and once the effort was made to produce something consumable, it was left to stagnate and rot over time. With ScraperWiki, we've structured...

• October 7, 2011
• comments

ScraperWiki Digs Up Dirty Data So You Don't Have To

by Nicola Hughes
2011 Knight News Challenge Winner

The best journalism comes from digging. Not phoning up press officers but speaking with those in the know. Not seeking comments from experts but going out onto the streets. The real stories, the scoops and the breakthroughs don't come prepackaged. So why limit data-driven journalism to the relatively few sources of clean, pre-packaged and nicely delivered data? This is where ScraperWiki comes in. ScraperWiki is a developer platform that aims to liberate data from the web, build upon this information to make useful applications, and get journalists and developers working together in true HacksHackers fashion! WHAT WE DO We let...

• September 19, 2011
• comments