Skip to content

Commit

Permalink
Merge pull request gocolly#470 from imthaghost/master
Browse files Browse the repository at this point in the history
[update] add project to README.md
  • Loading branch information
asciimoo authored Apr 17, 2020
2 parents 1cd6840 + 29c9f07 commit 6f8e581
Showing 1 changed file with 23 additions and 31 deletions.
54 changes: 23 additions & 31 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,21 +14,19 @@ With Colly you can easily extract structured data from websites, which can be us
[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fgocolly%2Fcolly.svg?type=shield)](https://app.fossa.io/projects/git%2Bgithub.com%2Fgocolly%2Fcolly?ref=badge_shield)
[![Twitter URL](https://img.shields.io/badge/twitter-follow-green.svg)](https://twitter.com/gocolly)


## Features

* Clean API
* Fast (>1k request/sec on a single core)
* Manages request delays and maximum concurrency per domain
* Automatic cookie and session handling
* Sync/async/parallel scraping
* Caching
* Automatic encoding of non-unicode responses
* Robots.txt support
* Distributed scraping
* Configuration via environment variables
* Extensions

- Clean API
- Fast (>1k request/sec on a single core)
- Manages request delays and maximum concurrency per domain
- Automatic cookie and session handling
- Sync/async/parallel scraping
- Caching
- Automatic encoding of non-unicode responses
- Robots.txt support
- Distributed scraping
- Configuration via environment variables
- Extensions

## Example

Expand All @@ -51,33 +49,31 @@ func main() {

See [examples folder](https://github.com/gocolly/colly/tree/master/_examples) for more detailed examples.


## Installation

```
go get -u github.com/gocolly/colly/v2/...
```


## Bugs

Bugs or suggestions? Visit the [issue tracker](https://github.com/gocolly/colly/issues) or join `#colly` on freenode


## Other Projects Using Colly

Below is a list of public, open source projects that use Colly:

* [greenpeace/check-my-pages](https://github.com/greenpeace/check-my-pages) Scraping script to test the Spanish Greenpeace web archive.
* [altsab/gowap](https://github.com/altsab/gowap) Wappalyzer implementation in Go.
* [jesuiscamille/goquotes](https://github.com/jesuiscamille/goquotes) A quotes scrapper, making your day a little better!
* [jivesearch/jivesearch](https://github.com/jivesearch/jivesearch) A search engine that doesn't track you.
* [Leagify/colly-draft-prospects](https://github.com/Leagify/colly-draft-prospects) A scraper for future NFL Draft prospects.
* [lucasepe/go-ps4](https://github.com/lucasepe/go-ps4) Search playstation store for your favorite PS4 games using the command line.
* [yringler/inside-chassidus-scraper](https://github.com/yringler/inside-chassidus-scraper) Scrapes Rabbi Paltiel's web site for lesson metadata.
* [gamedb/gamedb](https://github.com/gamedb/gamedb) A database of Steam games.
* [lawzava/scrape](https://github.com/lawzava/scrape) CLI for email scraping from any website.
* [eureka101v/WeiboSpiderGo](https://github.com/eureka101v/WeiboSpiderGo) A sina weibo(chinese twitter) scrapper
- [greenpeace/check-my-pages](https://github.com/greenpeace/check-my-pages) Scraping script to test the Spanish Greenpeace web archive.
- [altsab/gowap](https://github.com/altsab/gowap) Wappalyzer implementation in Go.
- [jesuiscamille/goquotes](https://github.com/jesuiscamille/goquotes) A quotes scrapper, making your day a little better!
- [jivesearch/jivesearch](https://github.com/jivesearch/jivesearch) A search engine that doesn't track you.
- [Leagify/colly-draft-prospects](https://github.com/Leagify/colly-draft-prospects) A scraper for future NFL Draft prospects.
- [lucasepe/go-ps4](https://github.com/lucasepe/go-ps4) Search playstation store for your favorite PS4 games using the command line.
- [yringler/inside-chassidus-scraper](https://github.com/yringler/inside-chassidus-scraper) Scrapes Rabbi Paltiel's web site for lesson metadata.
- [gamedb/gamedb](https://github.com/gamedb/gamedb) A database of Steam games.
- [lawzava/scrape](https://github.com/lawzava/scrape) CLI for email scraping from any website.
- [eureka101v/WeiboSpiderGo](https://github.com/eureka101v/WeiboSpiderGo) A sina weibo(chinese twitter) scrapper
- [imthaghost/goclone](https://github.com/imthaghost/goclone) Clone websites to your computer within seconds.

If you are using Colly in a project please send a pull request to add it to the list.

Expand All @@ -86,14 +82,12 @@ If you are using Colly in a project please send a pull request to add it to the
This project exists thanks to all the people who contribute. [[Contribute]](CONTRIBUTING.md).
<a href="https://github.com/gocolly/colly/graphs/contributors"><img src="https://opencollective.com/colly/contributors.svg?width=890" /></a>


## Backers

Thank you to all our backers! 🙏 [[Become a backer](https://opencollective.com/colly#backer)]

<a href="https://opencollective.com/colly#backers" target="_blank"><img src="https://opencollective.com/colly/backers.svg?width=890"></a>


## Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [[Become a sponsor](https://opencollective.com/colly#sponsor)]
Expand All @@ -109,8 +103,6 @@ Support this project by becoming a sponsor. Your logo will show up here with a l
<a href="https://opencollective.com/colly/sponsor/8/website" target="_blank"><img src="https://opencollective.com/colly/sponsor/8/avatar.svg"></a>
<a href="https://opencollective.com/colly/sponsor/9/website" target="_blank"><img src="https://opencollective.com/colly/sponsor/9/avatar.svg"></a>




## License

[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fgocolly%2Fcolly.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2Fgocolly%2Fcolly?ref=badge_large)

0 comments on commit 6f8e581

Please sign in to comment.