Skip to content

zofuthan/pyspider

This branch is 646 commits behind binux/pyspider:master.

Folders and files

NameName
Last commit message
Last commit date

Latest commit

09cb765 · May 31, 2015
Mar 6, 2014
May 24, 2015
May 31, 2015
May 27, 2015
Jan 18, 2015
Mar 16, 2014
May 24, 2015
May 27, 2015
Nov 16, 2014
Jan 28, 2015
May 22, 2015
Jan 3, 2015
May 22, 2015
Nov 24, 2014
May 22, 2015
May 24, 2015

Repository files navigation

pyspider Build Status Coverage Status Try

A Powerful Spider(Web Crawler) System in Python. TRY IT NOW!

  • Write script in Python
  • Powerful WebUI with script editor, task monitor, project manager and result viewer
  • MySQL, MongoDB, Redis, SQLite, PostgreSQL with SQLAlchemy as database backend
  • RabbitMQ, Beanstalk, Redis and Kombu as message queue
  • Task priority, retry, periodical, recrawl by age, etc...
  • Distributed architecture, Crawl Javascript pages, Python 2&3, etc...

Documentation: http://docs.pyspider.org/
Tutorial: http://docs.pyspider.org/en/latest/tutorial/

Sample Code

from pyspider.libs.base_handler import *


class Handler(BaseHandler):
    crawl_config = {
    }

    @every(minutes=24 * 60)
    def on_start(self):
        self.crawl('http://scrapy.org/', callback=self.index_page)

    @config(age=10 * 24 * 60 * 60)
    def index_page(self, response):
        for each in response.doc('a[href^="http"]').items():
            self.crawl(each.attr.href, callback=self.detail_page)

    def detail_page(self, response):
        return {
            "url": response.url,
            "title": response.doc('title').text(),
        }

Demo

Installation

Quickstart: http://docs.pyspider.org/en/latest/Quickstart/

Contribute

TODO

v0.4.0

  • local mode, load script from file.
  • works as a framework (all components running in one process, no threads)
  • redis
  • shell mode like scrapy shell
  • a visual scraping interface like portia

more

License

Licensed under the Apache License, Version 2.0

About

A Powerful Spider(Web Crawler) System in Python.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Python 81.9%
  • JavaScript 9.0%
  • HTML 4.6%
  • CSS 4.5%