[V1] [2/n] Logging and Metrics - OutputProcessor Abstraction
#11973
Conversation
Signed-off-by: [email protected] <[email protected]>
robertgshaw2-neuralmagic
requested review from
WoosukKwon,
njhill,
ywang96,
comaniac and
alexm-neuralmagic
as code owners
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
robertgshaw2-neuralmagic
changed the title
OutputProcessor Abstraction
| @@ -189,85 +178,3 @@ def _get_next_output_text(self, finished: bool, delta: bool) -> str: | |||
| self._last_output_text_offset = length | |||
| return self.output_text[last_offset:length] | |||
| return "" | |||
|
|
|||
|
|
|||
| class Detokenizer: | |||
There was a problem hiding this comment.
This is now called OutputProcessor, since we will do more than "just" detokenization in the step() function
| @@ -233,57 +225,46 @@ async def generate( | |||
| await self.abort(request_id) | |||
| raise | |||
|
|
|||
| def _process_request_outputs(self, request_outputs: List[RequestOutput]): | |||
There was a problem hiding this comment.
This logic is moved into OutputProcessor.process_outputs() loop
| queue=queue, | ||
| ) | ||
|
|
||
|
|
There was a problem hiding this comment.
NOTE: this was previously called Detokenizer
| @@ -59,9 +59,6 @@ def __init__( | |||
| lora_config=vllm_config.lora_config) | |||
| self.tokenizer.ping() | |||
|
|
|||
| # Request streams (map of request_id -> queue). | |||
There was a problem hiding this comment.
NOTE: these queues are held in OutputProcessor
| If you need to touch every element of the batch, implement a | ||
| method called XXXClass.update_from_output() to be called | ||
| within the loop below. For examples, see: | ||
| * IterationStats.update_from_output() |
There was a problem hiding this comment.
This is a great abstraction IMO.
nit: I wonder if we also want to make it more explicit by having something like a OutputHandler protocol that takes in the engine core output + maybe a current request state?
There was a problem hiding this comment.
I will add a comment suggesting that we do this once we add RequestStats. I want to keep flexibility while we are in the development stage
SUMMARY:
Detokenizer>>OutputProcessor.XXXClass.update_from_output+ be called inOutputProcessor.process_outputsloop.self._process_request_outputsinto this loop (previously this was a separate loop inoutput_handler)IterationStats.update_from_output()to this loopNOTES:
Detokenizerin a separate process hurts performance ([V1] [7/N] API Server: Multiprocessing Detokenizer [ DO NOT MERGE ] #11636). So feel confident that adding all of this to a single loop is the right approach.TODO: