Skip to content
forked from binux/pyspider

A Powerful Spider(Web Crawler) System in Python.

License

Notifications You must be signed in to change notification settings

jn7163/pyspider

This branch is 337 commits behind binux/pyspider:master.

Folders and files

NameName
Last commit message
Last commit date

Latest commit

aa7beea · Sep 23, 2016
Mar 6, 2014
Aug 29, 2016
Sep 20, 2016
Sep 20, 2016
Sep 30, 2015
Jan 18, 2015
Sep 9, 2015
Sep 13, 2016
May 24, 2016
Nov 16, 2014
Jan 28, 2015
Jan 23, 2016
Jul 10, 2016
Jun 22, 2016
Nov 24, 2014
Sep 23, 2016
Mar 6, 2016

Repository files navigation

pyspider Build Status Coverage Status Try

A Powerful Spider(Web Crawler) System in Python. TRY IT NOW!

Tutorial: http://docs.pyspider.org/en/latest/tutorial/
Documentation: http://docs.pyspider.org/
Release notes: https://github.com/binux/pyspider/releases

Sample Code

from pyspider.libs.base_handler import *


class Handler(BaseHandler):
    crawl_config = {
    }

    @every(minutes=24 * 60)
    def on_start(self):
        self.crawl('http://scrapy.org/', callback=self.index_page)

    @config(age=10 * 24 * 60 * 60)
    def index_page(self, response):
        for each in response.doc('a[href^="http"]').items():
            self.crawl(each.attr.href, callback=self.detail_page)

    def detail_page(self, response):
        return {
            "url": response.url,
            "title": response.doc('title').text(),
        }

Demo

Installation

Quickstart: http://docs.pyspider.org/en/latest/Quickstart/

Contribute

TODO

v0.4.0

  • local mode, load script from file.
  • works as a framework (all components running in one process, no threads)
  • redis
  • shell mode like scrapy shell
  • a visual scraping interface like portia

more

  • edit script with vim via WebDAV

License

Licensed under the Apache License, Version 2.0

About

A Powerful Spider(Web Crawler) System in Python.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Python 85.1%
  • JavaScript 8.1%
  • HTML 3.9%
  • CSS 1.9%
  • Lua 1.0%