UnboundLocalError: local variable 'pass_loras' referenced before assignment

[ezra-ch]

Loading model: /home/peko/models/capy34b
Traceback (most recent call last):
  File "/home/peko/tabbyAPI/main.py", line 289, in <module>
    for (module, modules) in load_status:
  File "/home/peko/tabbyAPI/model.py", line 187, in load_gen
    self.model.forward(input_ids, cache = self.cache, preprocess_only = True)
  File "/home/peko/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/peko/.local/lib/python3.10/site-packages/exllamav2/model.py", line 582, in forward
    r, ls = self._forward(input_ids = input_ids[:, chunk_begin : chunk_end],
  File "/home/peko/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/peko/.local/lib/python3.10/site-packages/exllamav2/model.py", line 655, in _forward
    x = module.forward(x, cache = cache, attn_mask = attn_mask, past_len = past_len, loras = loras)
  File "/home/peko/.local/lib/python3.10/site-packages/exllamav2/attn.py", line 478, in forward
    pass_loras,
UnboundLocalError: local variable 'pass_loras' referenced before assignment
peko@sdfsdfI:~/tabbyAPI$ ls  /home/peko/models/capy34b
README.md          generation_config.json             output-00003-of-00003.safetensors  tokenizer.model
added_tokens.json  output-00001-of-00003.safetensors  special_tokens_map.json            tokenizer_config.json
config.json        output-00002-of-00003.safetensors  tokenizer.json

[ezra-ch]

low_mem: True in the config was making that error, made it false and it loads

[bdashore3]

Hi there! Thanks for opening an issue. The low_mem option is experimental and has resulted in an error like that. If you'd like to save on vram, use 8-bit cache (cache_mode: FP8).

[ezra-ch]

Hey, thanks for the repo, its really great.

A non related question but, just wondering about the streaming function. Im unsure how to receive text streams on silly tavern from the API. Are there any flags I must use?

[bdashore3]

I'd recommend asking questions like these on the Discord Server (make sure to get the Tabby role when joining). Issues is moreso for bug reports and feature requests.