[v1][stats][1/n] Add RequestStatsUpdate and RequestStats types

[rickyyx]

This PR adds the basic data structures to capture request stats change and their lifecycle transitions.

Intended usage in follow-up PRs

• The RequestStatsUpdate would be recorded on both the engine core as well as the frontend engine (async multiprocess architecture) process as request goes through various stages (i.e. input processing, prefill, decode, finished, etc). If it's single process architecture, they would be on the same process.
• Frontend engine would periodically poll the request updates, and materialize them into RequestStats for each request, and build a loggable Stats object each iteration.

See full prototype here in #10651

Addresses #10582

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[WoosukKwon]

@robertgshaw2-neuralmagic Do you have bandwidth to review this PR by any chance?

[robertgshaw2-neuralmagic]

@robertgshaw2-neuralmagic Do you have bandwidth to review this PR by any chance?

Yeah

[robertgshaw2-neuralmagic]

Generally looks really good. My thoughts are:

• Preemption: I do not think that the way we are handling preemption is consistent with what we should be exposing to users (see comments in the code for why)

• Detokenization / Finished: I think that tracking these data are a bit complicated since they are not in the EngineCore and the Detokenizer runs concurrently with the EngineCore. I am curious to get your thoughts around from which object's POV we should consider a request "finished" and how/if we should include detokenization time in the TPOT metrics given the V1 design.

[rickyyx]

Generally looks really good. My thoughts are:

• Preemption: I do not think that the way we are handling preemption is consistent with what we should be exposing to users (see comments in the code for why)
• Detokenization / Finished: I think that tracking these data are a bit complicated since they are not in the EngineCore and the Detokenizer runs concurrently with the EngineCore. I am curious to get your thoughts around from which object's POV we should consider a request "finished" and how/if we should include detokenization time in the TPOT metrics given the V1 design.

Thanks for the great review @robertgshaw2-neuralmagic!
For preemption:

1. model forward time /model execute time should be accumulative across preemptions as you pointed out.
2. same for output token times.

For detokenization stage and finish case:

1. I think it might be better to mark a request from the PoV of the EngineCore only by sending the finish reason in the abort request RPC.
2. For detokenization time, this is a great point, TPOT should include detokenization time actually, and being recorded on the detokenizer.

[rickyyx]

Updates:

• Preemption should not clear some of the fields (e.g. output tokens ts)
• Add an explicit FINISH update to be used by caller (i.e. detokenizer)
• Clarify the running states into PREFILLING and DECODING more explicitly.

[comaniac]

As we are adding more supports to caching (MM preprocessor cache, prefix caching, and potentially embedding caching), it would be useful to track cache status in metrics, so would like to bring the metric system to v1 asap if possible.

[robertgshaw2-neuralmagic]

xxx