Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core] Prefill Only Tokens Without KV Cache in Batch Requests (Disagg Prefill) #12285

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

[Core] Prefill Only Tokens Without KV Cache in Batch Requests (Disagg Prefill) #12285

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
2b78a04
Only perform prefiling on requests without KV cache in a batch
Shaoting-Feng Jan 21, 2025
5151c51
Fix request id typo
Shaoting-Feng Jan 24, 2025
624b93c
Revert "Fix request id typo"
Shaoting-Feng Jan 24, 2025
f16a324
Fix request idx
Shaoting-Feng Jan 24, 2025
8561ee5
Fix request id typo
Shaoting-Feng Jan 24, 2025
e3d841e
Fix unresolved conflicts
Shaoting-Feng Jan 24, 2025
7549568
Change logger level
Shaoting-Feng Jan 24, 2025
b8fb040
Rerun unit test
Shaoting-Feng Jan 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Rerun unit test 
Signed-off-by: Shaoting Feng <[email protected]>
  • Loading branch information
@Shaoting-Feng
Shaoting-Feng committed Jan 24, 2025
commit b8fb040280f834a9e67852b692946629a9304958
3 changes: 1 addition & 2 deletions vllm/distributed/kv_transfer/kv_connector/simple_connector.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -390,8 +390,7 @@ def build_partial_prefill_input(
rebuilt_slot_mapping.append(new_slot_mapping)
rebuilt_max_query_len = max(q_len, rebuilt_max_query_len)
last_query_start_loc += q_len
rebuilt_query_start_loc.append(
last_query_start_loc) # start with 0
rebuilt_query_start_loc.append(last_query_start_loc)
rebuilt_context_lens_tensor.append(num_computed_token)

# recover `block_table`
Expand Down
Loading