feat: introduce the new reporter API

[sheremet-va]

Description

THIS IS A DRAFT.

interface Reporter {
  /**
   * This method is called when the error happens outside of the test.
   *
   * Possible errors:
   * - unhandled error thrown during the test run
   * - error thrown during collection phase when importing a module (global throw)
   * - error thrown during collection phase when calling `describe` callback
   * - error thrown inside `beforeAll`/`afterAll` hooks since it can't be attributed to a specific test
   */
  onError(error: SerialisedError, suite?: TestSuite): void

  /**
   * This method is called when all tests inside a single file were collected.
   * The `file` object contains `children()` method with all suites and tests.
   * Tasks won't have `result` property yet.
   *
   * **Note:** This method can be called in parallel if multiple files are collected at the same time (`--single-worker` is used)
   */
  onFileCollected(file: TestModule): void

  /**
   * This method is called when the runner is ready to start running a test. The `result` will always be `undefined`.
   * **Note:** If test is marked as `todo` or `skip`, this method will still be called.
   */
  onTestPrepare(test: TestCase): void
  /**
   * This method is called when the test finished running. It will always have a `result` property.
   */
  onTestFinished(test: TestCase): void
  /**
   * This method is called only after the test has finished running and the `result.type` is `failed`.
   */
  onTestFailed(test: TestCase): void

  /**
   * This method is called when user logs something to the console.
   * The order of logs is not guaranteed.
   */
  onConsoleLog(type: 'stderr' | 'stdout', log: UserConsoleLog): void

  /**
   * This is called when all tests have finished running.
   */
  onTestRunFinished(files: TestModule[], errors: SerialisedError[], reason: 'passed' | 'failed' | 'timedout' | 'interrupted'): void

  /**
   * Coverage is performed for all files that were executed.
   * This is called only if `--coverage` flag is used.
   */
  onCoverage(summary: any): void
}

The current lifecycle ("run tests" has its own lifecycle):

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

• It's really useful if your PR references an issue where it is discussed ahead of time. If the feature is substantial or introduces breaking changes without a discussion, PR might be closed.
• Ideally, include a test that fails without this PR but passes with it.
• Please, don't make changes to pnpm-lock.yaml unless you introduce a new test example.

Tests

• Run the tests with pnpm test:ci.

Documentation

• If you introduce new functionality, document it. You can run documentation with pnpm run docs command.

Changesets

• Changes in changelog are generated from PR name. Please, make sure that it explains your changes in an understandable manner. Please, prefix changeset messages with feat:, fix:, perf:, docs:, or chore:.

[netlify]

✅ Deploy Preview for vitest-dev ready!

Name	Link
🔨 Latest commit	38b967c
🔍 Latest deploy log	https://app.netlify.com/sites/vitest-dev/deploys/6759ab044481a60008a31c02
😎 Deploy Preview	https://deploy-preview-7069--vitest-dev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[AriPerkkio]

I can start helping out with this. Did you already have an idea how this should be integrated into the core.ts and RPC calls?

[sheremet-va]

I wanted to have parsing logic here, and use higher level methods in core (this.testRun.start(specs)) and low lever in rpc calls (like onTaskUpdate)

This is also the place where we call reporter methods related to the test run (onTestRunStart, but not onWatcherStart).

The inspiration for method names is https://vshaxe.github.io/vscode-extern/vscode/TestRun.html

[sheremet-va]

I also think that proposed events are not enough, we will probably need to add more

[sheremet-va]

I also added statistics to the reported tasks, so we can modify them in place

[AriPerkkio]

Proposition for the runTests phase here:

• onTestFileQueued(file: TestModule) - collection of module is about to start
• onTestFileStart(file: TestModule) - module is about to run
  • onHookStart(...) - before* hooks is called
  • onTestStart(test: TestCase) - a single test/it is about to start
  • onTestFinish / onTestFail
  • onHookEnd(...) - after* hook is called
• onTestFileFinish(file: TestModule) - all tests in the module finished running

Important part here is that this order must be guaranteed. Currently onTaskUpdate can skip certain phases if tests run fast enough. We need to add some state in module that converts onTaskUpdate into these hooks.

[AriPerkkio]

Could we use vitepress-plugin-mermaid for this? 🤔

https://emersonbottero.github.io/vitepress-plugin-mermaid/guide/styles.html

[sheremet-va]

Maybe when the lifecycle is finalised

[sheremet-va]

• onHookStart(...) - before* hooks is called

if we report suite hooks (beforeAll/beforeEach), should we report onSuiteStart?

• onTestFinish / onTestFail

I think now it's better to have a single hook (onTestFinish) and check the result there - I updated the TestCase to always return a result() (now return pending if no result is set)

• onTestFile*

Should be onTestModule*. The idea with the name is that I expect us to implement support for virtual modules.