Support cuda graph for LoRA

[Qiaolin-Yu]

Motivation

Close #3282

Modifications

It seems that there are no inherent compatibility issues between CUDA Graph and LoRA, so I simply enabled the configuration.

Benchmarking Result

I run these commands to launch the server.

# Triton backend
python benchmark/lora/launch_server.py --max-loras-per-batch 4 --lora-backend triton

# Base model without lora
# python benchmark/lora/launch_server.py --base-only

Then run this command to request test from client:

python benchmark/lora/lora_bench.py

Benchmark Configs

• base model: meta-llama/Llama-2-7b-hf
• lora adapter: winddude/wizardLM-LlaMA-LoRA-7B
• GPU: Nvidia A100
• maximum number of serving loras: 4
• number of requests: 50
• input length: uniform random distribution on [1, 1024]
• output length: uniform random distribution on [1, 128]

Here are the results.

Backend	Enable Cuda Graph	Total Throughput (tok/s)	Mean E2E Latency (ms)
Triton	True	3518.19	4528.47
Triton	False	2010.20	7590.18
No LoRA	True	4275.06	3760.48
No LoRA	False	3073.95	5009.22

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.