Uncategorized
- mxrch/darkshot - Lightshot scraper on steroids with OCR
- devanshbatham/ParamSpider - Mining parameters from dark corners of Web Archives
- Nekmo/dirhunt - Find web directories without bruteforce - 有个index of自动识别功能
- fcavallarin/htcap - a web application scanner able to crawl single page application (SPA) in a recursive manner by intercepting ajax calls and DOM changes
- GerbenJavado/LinkFinder - A python script that finds endpoints in JavaScript files - 支持burpsuite导出的文件
- nahamsec/JSParser - A python 2.7 script using Tornado and JSBeautifier to parse relative URLs from JavaScript files. Useful for easily discovering AJAX requests when performing security research or bug bounty hunting
- OWASP zaproxy - AJAX Spider site
- danielmiessler/RobotsDisallowed - A harvest of the Disallowed directories from the robots.txt files of the world's top websites
- milo2012/pathbrute - Pathbrute is a DirB/Dirbuster type of tool designed to brute force directories and files names on web/application servers
- maurosoria/dirsearche - Web path scanner
- s0md3v/Photon - Incredibly fast crawler which extracts urls, emails, files, website accounts and much more
- si9int/cc.py - Extracting URLs of a specific target based on the results of "commoncrawl.org"
- jordanpotti/CloudScraper - a Tool to spider and scrape targets in search of cloud resources
- facert/awesome-spider - 爬虫集合
- Anorov/cloudflare-scrape - A Python module to bypass Cloudflare's anti-bot page
Chrome headless
- ultrafunkamsterdam/undetected-chromedriver - Optimized Selenium Chromedriver patch which does not trigger anti-bot services like Distill Network / Imperva / DataDome / Botprotect.io Automatically downloads the driver binary and patches it
- 0Kee-Team/crawlergo - A powerful dynamic crawler for web vulnerability scanners - 2020.6停更
- knownsec/LSpider - 一个为被动扫描器定制的前端爬虫 - 有个seebug文章说明
- yujiosaka/headless-chrome-crawler - Distributed crawler powered by Headless Chrome
- ring04h/papers - WEB2.0启发式爬虫实战-猪猪侠-20180616 - 关闭xss auditor + 自定义函数检测dom xss/拦截请求以防止意外跳转/劫持XHR/监控DOM变化/表单自动填充/隐身模式启动
- myvyang/chromium_for_spider - 为漏扫动态爬虫定制的浏览器
- adieuadieu/serverless-chrome - Run headless Chrome/Chromium on AWS Lambda
- jsoverson/hackium - Hackium is a CLI tool, a browser, and a platform for analyzing and manipulating web sites
- starlabs: You Talking To Me - chrome/firefox等驱动漏洞利用，符合一定条件可以RCE
Puppeteer