forked from vercel/turborepo
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
add benchmark documentation and blog post (vercel#2492)
Co-authored-by: Anthony Shew <[email protected]> Co-authored-by: Jared Palmer <[email protected]> Co-authored-by: Maia Teegarden <[email protected]> Co-authored-by: Will Binns-Smith <[email protected]> Co-authored-by: Alex Kirszenberg <[email protected]> Co-authored-by: Matt Pocock <[email protected]>
- Loading branch information
7 people
authored
Oct 31, 2022
1 parent
e89ffd6
commit f5ce4c9
Showing
28 changed files
with
2,526 additions
and
93 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| <!-- | ||
| IMPORTANT: This document is linked to from https://turbo.build | ||
| DO NOT rename this file without updating the link in docs/pages/pack/docs/benchmarks.mdx | ||
| --> | ||
|
|
||
| # Benchmarking Turbopack | ||
|
|
||
| The simplest way to run Turbopack's benchmark suite is with the command `cargo bench -p next-dev`. This will benchmark Turbopack's Next.js development server in a variety of scenarios and it's what we use to track Turbopack's performance over time. | ||
|
|
||
| `cargo bench -p next-dev` accepts different options via environment variables. To vary the number of modules tested, set `TURBOPACK_BENCH_COUNTS`. For example, to test against an app with 5,000 modules instead of the default 1,000, run | ||
|
|
||
| ```sh | ||
| TURBOPACK_BENCH_COUNTS=5000 cargo bench -p next-dev | ||
| ``` | ||
|
|
||
| ## Benchmarking Turbopack against other bundlers | ||
|
|
||
| The benchmark numbers we share on [the Turbopack website](https://turbo.build/pack) are informed by running Turbopack's benchmark suite against Turbopack and other bundlers. These are run in a controlled environment prior to being published. We use the `bench_startup` and `bench_hmr_to_eval` benchmarks currently (see below). | ||
|
|
||
| To run Turbopack benchmarks against other bundlers, run: | ||
|
|
||
| ```sh | ||
| TURBOPACK_BENCH_BUNDLERS=all cargo bench -p next-dev | ||
| ``` | ||
|
|
||
| and optionally filter the benchmarks run to specific bundlers, such as: | ||
|
|
||
| ```sh | ||
| TURBOPACK_BENCH_BUNDLERS=all cargo bench -p next-dev -- "hmr_to_eval/(Turbopack CSR|Vite)" | ||
| ``` | ||
|
|
||
| **Note**: The Turbopack benchmark suite includes a mix of server-side rendered and client-only rendered examples -- these are reflected in "CSR" or "SSR" in the benchmark name. Turbopack supports both, while some other bundlers only support client-rendered examples. Take that into account when comparing CSR results against SSR. | ||
|
|
||
| **Hint**: These benchmarks take a long time to complete, since they try to capture at least 10 samples for every scenario. There is a `TURBOPACK_BENCH_PROGRESS=1` env var to show values while the benchmarks are running. | ||
|
|
||
| ## Benchmark Suite scenarios | ||
|
|
||
| The benchmark suite runs Turbopack and other bundlers in a variety of scenarios. The tests use a real headless browser and perform a variety of common scenarios in web development, and wait for results to be reflected in the page. | ||
|
|
||
| - **bench_startup:** Time from startup (without cache) until the app is rendered in the browser (it doesn't have to be interactive/hydrated for this.) | ||
| - **bench_hydration:** Time from startup (without cache) until the app is interactive in the browser (it needs to be hydrated for that.) This metric is not captured for CSR since the first render is interactive. | ||
| - **bench_hmr_to_eval:** Time from changing a file until the new code is evaluated in the browser. Evaluating the code does not mean the change is visible to the user yet. For instance, when a React component changes, it need to be re-rendered in the browser. This mostly measures the time spent computing the update in the bundler itself and sending it to the client. | ||
| - **bench_hmr_to_commit:** Time from changing a file until the change is reflected in the browser. We are using a `useEffect` hook within a React component to measure the time it takes for the updated React component to be committed to the DOM. This is a good measure of the end to end performance perceived by the user. | ||
| - **bench_startup_cache:** Time from startup with persistent cache until the app is rendered in the browser (it doesn't have to be interactive/hydrated for this.). Turbopack doesn't include a persistent cache yet. (This benchmark is disabled by default and can be enabled with `TURBOPACK_BENCH_CACHED=1`) | ||
| - **bench_hydration:** Time from startup with persistent cache until the app is interactive in the browser (it needs to be hydrated for that.) This metric is not captured for CSR since the first render is interactive. Turbopack doesn't include a persistent cache yet. (This benchmark is disabled by default and can be enabled with `TURBOPACK_BENCH_CACHED=1`) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,80 +1,54 @@ | ||
| { | ||
| "cold": { | ||
| "1000": { | ||
| "next13": 0.868, | ||
| "vite": 4.8, | ||
| "next12": 3.4, | ||
| "next11": 7.7 | ||
| "next13": 1.1252525, | ||
| "vite": 4.76572914, | ||
| "next12": 3.4092260955, | ||
| "next11": 7.73633697 | ||
| }, | ||
| "5000": { | ||
| "next13": 3.0, | ||
| "vite": 19.2, | ||
| "next12": 10.7, | ||
| "next11": 24.8 | ||
| "next13": 3.55898876, | ||
| "vite": 19.18001276, | ||
| "next12": 10.66688052, | ||
| "next11": 24.78618103 | ||
| }, | ||
| "10000": { | ||
| "next13": 6.1, | ||
| "vite": 37.2, | ||
| "next12": 20.1, | ||
| "next11": 49.4 | ||
| "next13": 7.4692269, | ||
| "vite": 37.2017471, | ||
| "next12": 20.05620067, | ||
| "next11": 49.37418728 | ||
| }, | ||
| "30000": { | ||
| "next13": 20.3, | ||
| "vite": 109.5, | ||
| "next12": 76.6, | ||
| "next11": 176.2 | ||
| "next13": 22.3009127, | ||
| "vite": 109.52734199, | ||
| "next12": 76.55617539, | ||
| "next11": 176.18900858 | ||
| } | ||
| }, | ||
| "file_change": { | ||
| "1000": { | ||
| "next13": 0.015, | ||
| "vite": 0.087, | ||
| "next12": 0.134, | ||
| "next11": 0.201 | ||
| "next13": 0.01528548, | ||
| "vite": 0.08654017, | ||
| "next12": 0.13393639, | ||
| "next11": 0.27323013 | ||
| }, | ||
| "5000": { | ||
| "next13": 0.027, | ||
| "vite": 0.093, | ||
| "next12": 0.386, | ||
| "next11": 0.85 | ||
| "next13": 0.0165313, | ||
| "vite": 0.09254822, | ||
| "next12": 0.41555998, | ||
| "next11": 0.84880069 | ||
| }, | ||
| "10000": { | ||
| "next13": 0.015, | ||
| "vite": 0.11, | ||
| "next12": 1.067, | ||
| "next11": 1.714 | ||
| "next13": 0.01575056, | ||
| "vite": 0.11038802, | ||
| "next12": 1.06711049, | ||
| "next11": 1.71389651 | ||
| }, | ||
| "30000": { | ||
| "next13": 0.015, | ||
| "vite": 0.256, | ||
| "next12": 3.4, | ||
| "next11": 6.9 | ||
| } | ||
| }, | ||
| "from_cache": { | ||
| "1000": { | ||
| "next13": 0.868, | ||
| "vite": 2.9, | ||
| "next12": 2.2, | ||
| "next11": 4.3 | ||
| }, | ||
| "5000": { | ||
| "next13": 3.0, | ||
| "vite": 11.5, | ||
| "next12": 7.2, | ||
| "next11": 12.9 | ||
| }, | ||
| "10000": { | ||
| "next13": 6.1, | ||
| "vite": 22.1, | ||
| "next12": 13.0, | ||
| "next11": 27.4 | ||
| }, | ||
| "30000": { | ||
| "next13": 20.3, | ||
| "vite": 66.3, | ||
| "next12": 42.5, | ||
| "next11": 31.8 | ||
| "next13": 0.01513124, | ||
| "vite": 0.25617544, | ||
| "next12": 5.39863267, | ||
| "next11": 6.94377699 | ||
| } | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.