Overview & Initial Experience

Karen Tracey, Colin Copeland

• What it is?
• Our experience with it
• Extensions inspired by our experience
• Future

• "Hyper-local news" application framework
• Example news types
  • police incident reports
  • property transactions
  • restaurant inspections
  • business registrations
• Open source project
  • Web site: http://openblockproject.org/
  • Code: https://github.com/openplans/openblock
  • Mailing list: http://groups.google.com/group/ebcode/

• Sites with a local focus,
• which can benefit from automated news item generation,
• where news item sources already exist online,
• in some user-unfriendly, but regular, fashion

• Adrian Holovaty founded a web startup, EveryBlock, with a team of six
• March 2007: Won Knight News Challenge program grant
• Jan. 2008: Site launch with Chicago, New York, San Francisco
• June 2008: Charlotte and Philadelphia added
• Aug. 2009: Acquired by MSNBC
• Today in 16 cities, 3 more coming soon

• July 2009: the EveryBlock team open sourced core functionality on Google Code
• 7 tarballs available at http://code.google.com/p/ebcode/
• And then what happened?

• June 2010: Knight Foundation launches OpenBlock Initiative grant
  • OpenPlans: streamline and extend OpenBlock over 2 years
  • The Columbia Daily Tribune: install, test, and add new features in the context of a smaller newspaper
  • The Boston Globe: install, test, and add new features in the context of a larger newspaper

• Easier to install and setup
• Consolidated codebase into 4 core modules
  • ebdata
  • ebpub
  • obadmin
  • obdemo
• Current (v1.2) and previous versions are installable via PyPI
• Code: https://github.com/openplans/openblock
• Demo site: http://demo.openblockproject.org/
• Documentation: http://openblockproject.org/docs/

• Primary News Models
  • Schema: description of a particular data set, like "Restaurant Inspection"
  • NewsItem: individual piece of news associated with a schema
• Primary Geocoder Models
  • Street: a street with a unique name
  • Intersection: a point representing the meeting of two streets
  • Block: segment of a single street between two intersecting streets

• June 2011: OpenRural funded by a three-year Knight News Challenge grant
• Ryan Thornburg, professor at School of Journalism and Mass Communication at UNC
• Caktus is helping develop and deploy OpenRural for these NC communities

• Shapefiles contain location data and metadata
  • Census (Tiger)
  • County
  • State
• How to measure accuracy & completeness?
  • Columbus County GIS has addresses file
  • ~38,000 valid addresses in the county

• Multiple "cities"
  • Supported by OpenBlock, but not "default"
  • Different urlpatterns single- vs. multi-city
  • Multi-city urlpatterns include "city slug"
• Unincorporated areas
  • Lots of space not in any town/city
  • These places need names to be navigable
  • Can use census "county subdivision" names
  • ...but these are not meaningful to residents

• Advantages
  • Code already exists in OpenBlock to use these files
  • Generalizable to other NC counties
• Disadvantages
  • Incomplete/incorrect data
  • 70% success rate geocoding ~38,000 Columbus County addresses

• Advantages
  • More complete/accurate
  • ~38,000 address geocode success rate improved to 93%
• Disadvantages
  • Custom code to load this data (custom BlockImporter)
  • Not generalizable to other counties
  • This data not available for all counties

• Geocoding is a hard problem to solve
• String parsing
  • number
  • predir
  • street name
  • street type
  • postdir
• Streets can have multiple names (misspellings table can help)
• 3rd-party geocoder fallback?

• Scripts that extract information from online data sources
• The process is conceptually simple:
  • Download some data from the web
  • Create one or more NewsItems whose fields are populated with that data
  • Save the NewsItem(s) to the database
• The grunt work is in extracting the data you need
• Scrapers sometimes require more than a single data source
  • CSV/Excel/Navy DIF
  • Shapefile
  • Download multiple files and stitch them together locally

• Corporation Filings: scraped from the NC Secretary of State website
• Restaurant Inspections: scraped from large Crystal Report exports from the NC Department of Health and Human Services
• Property Transactions: scraped from the Columbus County Tax and GIS offices
• Geocoded News Articles: scraped from whiteville.com
• Notably missing: police incident reports

• Automated fabric server provisioning and deployment. Testable with vagrant.
• Using Celery and RabbitMQ for asynchronous tasks (scrapers and maintenance tasks)
• Modified fork for OpenBlock that includes staticfiles changes.
• Production runs nginx and gunicorn on a small Amazon EC2 instance.
• Most issues fixed on OpenBlock core are pushed back to the official repository
• Everything is completely open source
• https://github.com/openrural

• How successful was the geocoder?
• How many news items were added?
• Why is my scraper failing to run?
• Why did this address fail to geocode? How can I correct it?

from openrural.data_dashboard.scrapers import DashboardMixin
from openrural.retrieval.base.scraperwiki import ScraperWikiScraper

class CorporationsScraper(DashboardMixin, ScraperWikiScraper):

    # scraper settings
    logname = 'corporations'
    schema_slugs = ('corporations',)

• Improving the geocoder is tough and, therefore, expensive
  • Possibly fallback to 3rd party geocoder
• Web UI code is hard to use and extend
  • JavaScript libraries for interacting with slippy maps have come a long way
  • Rewrite would make our lives easier in the future
• Sustainability as we scale
  • Would it be more efficient to build a single system to power all counties?
  • In our case, each OpenBlock install will be very similar

• OpenBlock has largely been developed through grant funding
• Paul Winkler of OpenPlans has been very helpful and active in the community
• However, Knight funding has ended and OpenPlans is no longer actively working on the project
• Future of the community is unknown
• OpenBlock needs an organic online community to survive
• If you're interested in OpenBlock, come speak to us!

• Slides: http://caktus.github.com/openblock-djangocon-talk
• Karen Tracey - @km_tracey
• Colin Copeland - @copelco