Skip to content

kaijchang/USNews-College-Scraper

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

US News College Scraper

Really simple and modifiable scraper for US News' College Rankings.

The included data.csv should have all the data you need, but you can modify the FIELDS variable in main.py to get other desired fields or remove unwanted fields.

Default fields:

FIELDS = [
    'institution.displayName',
    'institution.schoolType',
    'institution.aliasNames',
    'institution.state',
    'institution.city',
    'institution.zip',
    'institution.region',
    'institution.isPublic',
    'institution.institutionalControl',
    'ranking.displayRank',
    'ranking.sortRank',
    'ranking.isTied',
    'searchData.actAvg.rawValue',
    'searchData.percentReceivingAid.rawValue',
    'searchData.acceptanceRate.rawValue',
    'searchData.tuition.rawValue',
    'searchData.hsGpaAvg.rawValue',
    'searchData.engineeringRepScore.rawValue',
    'searchData.parentRank.rawValue',
    'searchData.enrollment.rawValue',
    'searchData.businessRepScore.rawValue',
    'searchData.satAvg.rawValue',
    'searchData.costAfterAid.rawValue',
    'searchData.testAvgs.displayValue.0.value',
    'searchData.testAvgs.displayValue.1.value'
]

Look at the items array in the data field of the search results (https://www.usnews.com/best-colleges/api/search?_sort=schoolName&_sortDirection=asc&_page=1) to look at the structure to find the fields.

If the DETAILED flag in main.py is set to true, the scraper will also scrape the page for each individual school for more data instead of just the aggregated search pages. This makes the scraping process about 10x slower, but adds the DETAIL_FIELDS.

The included data-detailed.csv was run with DETAILED on.

Detail fields:

DETAIL_FIELDS = [
    'School Type',
    'Year Founded',
    'Religious Affiliation',
    'Academic Calendar',
    'Setting',
    '2018 Endowment',
    'School Website'
]

About

US News College Rankings Scraper

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages