Follow the instructions in the Python Quick Start Guide to install Homebrew, Git, PostGIS, Python 3.3+ and virtualenv.
mkvirtualenv scrapers-ca --python=`which python3`
git clone git://github.com/opencivicdata/scrapers-ca.git
cd scrapers-ca
pip install -r requirements.txt
Initialize the database:
createdb pupa
psql pupa -c "CREATE EXTENSION postgis;"
pupa dbinit ca
pupa update ca_ab_edmonton
To run only the scraping step and skip the import step add the --scrape switch:
pupa update --scrape ca_ab_edmonton
For documentation on the pupa command:
pupa -h
For documentation on the update subcommand:
pupa update -h
Find division identifiers using the Open Civic Data Division Identifier (OCD-ID) Viewer or by browsing the list of identifiers. In most cases, a municipality will have a division identifier with a type ID of csd. Create a scraper with:
pupa init ca_on_toronto
Read the Pupa documentation or an existing scraper's code.
Avoid using the XPath string() function unless the expression is known to not have matches on some pages. Otherwise, scrapers may continue to run without error despite failing to find a match. A comment like # can be empty or # allow string() should accompany the use of string().
Use the get_email and get_phone helpers as much as possible.
In late 2014/early 2015, we disabled some single-jurisdiction scrapers to lower maintenance costs, some of which have been re-enabled, and disabled all multi-jurisdiction scrapers, because Pupa didn't support them. The disabled scrapers are in disabled/.
Make the code style consistent:
flake8
Check whether any non-authoritative CSVs are likely to be stale:
invoke stale
Check module names, class names, classification, division_name, name and url in __init.py__ files:
invoke tidy
Check sources are credited and assertions are made:
invoke sources_and_assertions
Check jurisdiction URLs (look for Delete COUNCIL_PAGE or Missing COUNCIL_PAGE instructions):
invoke council_pages
Update the OCD-IDs:
curl -O https://raw.githubusercontent.com/opencivicdata/ocd-division-ids/master/identifiers/country-ca.csv
Scraper code rarely undergoes code review. The focus is on the quality of the data.
This repository is on GitHub: https://github.com/opencivicdata/scrapers-ca, where your contributions, forks, bug reports, feature requests, and feedback are greatly welcomed.
Copyright (c) 2013 Open North Inc., released under the MIT license