Overview

rod is a High-level Devtools driver directly based on DevTools Protocol. It's designed for web automation and scraping. rod also tries to expose low-level interfaces to users, so that whenever a function is missing users can easily send control requests to the browser directly.

Features

• Fluent interface design to reduce verbose code
• Chained context design, intuitive to timeout or cancel the long-running task
• Debugging friendly, auto input tracing, remote monitoring headless browser
• Thread-safe for all operations
• Two styles of error handling to choose.
• Automatically find or download browser
• No external dependencies, CI tested on Linux, Mac, and Windows
• High-level helpers like WaitStable, WaitRequestIdle, HijackRequests, GetResource, etc
• Two-step WaitEvent design, never miss an event
• Correctly handles nested iframes
• No zombie browser process after the crash (how it works)

Examples

You can find examples from here or here.

For more detailed examples, please search the unit tests. Such as the usage of method HandleAuth, you can search all the *_test.go files that contain HandleAuth or HandleAuthE, for example, use Github online search in repository. You can also search the GitHub issues, they contain a lot of usage examples too.

Here is a comparison of the examples between rod and chromedp.

If you have questions, please raise an issue or join the chat room.

How it works

Here's the common start process of rod:

1. Try to connect to a Devtools endpoint (WebSocket), if not found try to launch a local browser, if still not found try to download one, then connect again. The lib to handle it is launcher.

2. Use the JSON-RPC to talk to the Devtools endpoint to control the browser. The lib handles it is cdp.

3. The type definitions of the JSON-RPC are in lib proto.

4. To control a specific page, rod will first inject a js helper script to it. rod uses it to query and manipulate the page content. The js lib is in assets.

FAQ

TOC

• Why functions don't return error values
• How to use rod with docker
• Why there is always an "about:blank" page
• Does it support other browsers like Firefox or Edge
• Why is it called rod
• How to contribute
• How versioning is handled
• Why another puppeteer like lib

Q: Why functions don't return error values

Please read the E suffixed function family.

Q: How to use rod with docker

To let rod work with docker is very easy:

1. Run the rod image docker run -p 9222:9222 rodorg/rod

2. Open another terminal and run a go program like this example

The rod image can dynamically launch a browser for each remote driver with customizable browser flags. It's tuned for screenshots and fonts among popular natural languages. You can easily load balance requests to the cluster of this image, each container can create multiple browser instances at the same time.

Q: Why there is always an "about:blank" page

It's an issue of the browser itself. If we enable the --no-first-run flag and we don't create a blank page, it will create a hello page which will consume more power.

Q: Does it support other browsers like Firefox or Edge

rod should work with any browser that supports DevTools Protocol.

• Microsoft Edge can pass all the unit tests.
• Firefox is supporting this protocol.
• Safari doesn't have any plan to support it yet.
• IE won't support it.

Q: Why is it called rod

rod is related to puppetry, see rod Puppet. So we are the puppeteer, the browser is the puppet, we use the rod to control the puppet. So in this sense, puppeteer.js sounds strange, we are controlling a puppeteer?

Q: How to contribute

Please check this doc.

Q: How versioning is handled

Semver is used.

Before v1.0.0 whenever the second section changed, such as v0.1.0 to v0.2.0, there must be some public API changes, such as changes of function names or parameter types. If only the last section changed, no public API will be changed.

You can use the Github's release comparison to see the automated changelog, for example, compare v0.44.2 with v0.44.0.

Q: Why another puppeteer like lib

There are a lot of great projects, but no one is perfect, choose the best one that fits your needs is important.

• chromedp

  With chromedp, you have to use their verbose DSL like tasks to handle the main logic, because chromedp uses several wrappers to handle execution with context and options which makes it very hard to understand their code when bugs happen. The DSL like wrapper also makes the Go type useless when tracking issues.

  It's painful to use chromedp to deal with iframes, this ticket is still open after years.

  chromedp doesn't support shadow DOM.

  When a crash happens, chromedp will leave the zombie browser process on Windows and Mac.

  For more comparison you can check here. If you compare the logic example between rod and chromedp, you will find out how much simpler rod is. Besides, rod has more high-level helpers like WaitRequestIdle, GetDownloadFile, HijackRequests, etc.

• selenium

  Selenium is based on webdriver protocol which has much less functions compare to devtools protocol. Such as it can't handle closed shadow DOM. No way to save page as PDF. No support for tools like Profiler or Performance, etc.

  Harder to set up and maintain because of extra dependencies like a browser driver.

  Though selenium sells itself for better cross-browser support, it's usually very hard to make it work for all major browsers.

  There are plenty of articles about "selenium vs puppeteer", you can treat rod as the Golang version of puppeteer.

• puppeteer

  With Puppeteer, you have to handle promise/async/await a lot. It requires a deep understanding of how promises works which are usually painful for QA to write automation tests. End to end tests usually requires a lot of sync operations to simulate human inputs, because Puppeteer is based on Nodejs all control signals it sends to the browser will be async calls, so it's unfriendly for QA from the beginning.

  Rod will only enable domain events when they are needed, puppeteer will always enable all the domains which will consume a lot of resources when driving a remote browser.

• cypress

  Cypress is very limited, for closed shadow dom or cross-domain iframes it's almost unusable. Read their limitation doc for more details.

  If you want to cooperate with us to create a testing focused framework base on rod to overcome the limitation of cypress, please contact us.