Skip to content
forked from andeya/pholcus

[Crawler for Golang] Pholcus is a distributed, high concurrency and powerful web crawler software.

License

Notifications You must be signed in to change notification settings

moregatest/pholcus

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

延伸自 https://github.com/henrylee2cn/pholcus

變更紀錄

phantom的改善

官方版本的phantom模式js是固定無法隨抓取規則變更 我的版本可以透過SetTemp添加客製js

					req := &request.Request{
						Url:          initUrl,
						Rule:         "keywds",
						ConnTimeout:  -1,
						Reloadable:   true,
						DownloaderID: 1,
					}
					req.Prepare()
					req.SetTemp("js", cjs)
					ctx.AddQueue(req)

About

[Crawler for Golang] Pholcus is a distributed, high concurrency and powerful web crawler software.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Go 98.6%
  • Roff 1.3%
  • Other 0.1%