Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit number of sequences #220

Merged
merged 11 commits into from
Feb 23, 2024
Merged

Limit number of sequences #220

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
29cac68
Add max_num_seq
yelite Feb 21, 2024
549861a
add to engine
yelite Feb 21, 2024
4cf54ea
Limit max_num_seq when grows the batch
yelite Feb 22, 2024
a96e915
Add max num seq to args
yelite Feb 22, 2024
1fd2b2e
Add max_num_seq_per_sequence
yelite Feb 23, 2024
634a869
Expose gpu memory utilization to engine config
yelite Feb 22, 2024
48af6d5
Fix dataclass
yelite Feb 22, 2024
38ba3e3
Remove the default value of gpu_memory_utilization
yelite Feb 23, 2024
320a9ba
Merge branch 'expose-gpu-util' into fix-memory-profiling
yelite Feb 23, 2024
0b574aa
Fix params
yelite Feb 23, 2024
cd4f1c8
Apply to torch model
yelite Feb 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix dataclass
  • Loading branch information
@yelite
yelite committed Feb 23, 2024
commit 48af6d5b263b26b86750bc650d136188bfd316ab
2 changes: 1 addition & 1 deletion serve/mlc_serve/engine/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ class MLCServeEngineConfig:
init_timeout: int = 120
model_type: str = "tvm" # "tvm", "torch"
num_shards: Optional[int] = None # Need to be specified for if model_type is "torch"
gpu_memory_utilization=0.9
gpu_memory_utilization: float = 0.9

@classmethod
def _from_json(config_cls, json_obj: Dict[Any, Any]):
Expand Down