Skip to content

stevenhanson/pyspider

This branch is 205 commits behind binux/pyspider:master.

Folders and files

NameName
Last commit message
Last commit date

Latest commit

3fccfab · Mar 9, 2019
Apr 18, 2017
Mar 6, 2014
Apr 5, 2018
Feb 24, 2019
Feb 24, 2019
Sep 30, 2015
Jan 18, 2015
Mar 15, 2018
Feb 24, 2019
Feb 14, 2019
Nov 16, 2014
Jan 28, 2015
Jun 15, 2017
Jul 10, 2016
Mar 15, 2018
Nov 24, 2014
Feb 24, 2019
Jan 16, 2017

Repository files navigation

pyspider Build Status Coverage Status Try

A Powerful Spider(Web Crawler) System in Python. TRY IT NOW!

Tutorial: http://docs.pyspider.org/en/latest/tutorial/
Documentation: http://docs.pyspider.org/
Release notes: https://github.com/binux/pyspider/releases

Sample Code

from pyspider.libs.base_handler import *


class Handler(BaseHandler):
    crawl_config = {
    }

    @every(minutes=24 * 60)
    def on_start(self):
        self.crawl('http://scrapy.org/', callback=self.index_page)

    @config(age=10 * 24 * 60 * 60)
    def index_page(self, response):
        for each in response.doc('a[href^="http"]').items():
            self.crawl(each.attr.href, callback=self.detail_page)

    def detail_page(self, response):
        return {
            "url": response.url,
            "title": response.doc('title').text(),
        }

Demo

Installation

WARNING: WebUI is open to the public by default, it can be used to execute any command which may harm your system. Please use it in an internal network or enable need-auth for webui.

Quickstart: http://docs.pyspider.org/en/latest/Quickstart/

Contribute

TODO

v0.4.0

  • a visual scraping interface like portia

License

Licensed under the Apache License, Version 2.0

About

A Powerful Spider(Web Crawler) System in Python.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Python 85.1%
  • JavaScript 8.3%
  • HTML 3.7%
  • CSS 1.8%
  • Other 1.1%