Load models from sub-directories

[zappityzap]

Checklist:

• I have read the Contributing guidelines.

• Support loading models from sub-directories
• Allow following symlinks in models directory
• Add .modelfile to excluded extensions

This PR enables organizing models into folders, and symlinking to other locations. I use this to keep one copy of models in a local folder that can be shared among all UIs.

[TheLounger]

This setup currently works:

Typical model folder:

Besides me liking this setup (1 model = 1 directory), I'm pretty sure it's required for llamacpp_HF due to tokenizer files being required. However, this PR makes things really messy:

It also seems to fail on some (non-GGUF) models, getting the path wrong or something.

  File "...\webui\modules\ui_model_menu.py", line 232, in load_model_wrapper
    shared.model, shared.tokenizer = load_model(selected_model, loader)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\webui\modules\models.py", line 93, in load_model
    output = load_func_map[loader](model_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\webui\modules\models.py", line 315, in ExLlamav2_HF_loader
    return Exllamav2HF.from_pretrained(model_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\webui\modules\exllamav2_hf.py", line 175, in from_pretrained
    config.prepare()
  File "...\webui\installer_files\env\Lib\site-packages\exllamav2\config.py", line 175, in prepare
    assert os.path.exists(self.model_config), "Can't find " + self.model_config
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Can't find models\8B-EXL2-5B__Llama-3-Lumimaid-v0.1-8K\output.safetensors\config.json

  File "...\webui\modules\ui_model_menu.py", line 232, in load_model_wrapper
    shared.model, shared.tokenizer = load_model(selected_model, loader)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\webui\modules\models.py", line 93, in load_model
    output = load_func_map[loader](model_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\webui\modules\models.py", line 155, in huggingface_loader
    config = AutoConfig.from_pretrained(path_to_model, trust_remote_code=shared.args.trust_remote_code)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\webui\installer_files\env\Lib\site-packages\transformers\models\auto\configuration_auto.py", line 1017, in from_pretrained
    config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\webui\installer_files\env\Lib\site-packages\transformers\configuration_utils.py", line 574, in get_config_dict
    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\webui\installer_files\env\Lib\site-packages\transformers\configuration_utils.py", line 672, in _get_config_dict
    raise EnvironmentError(
OSError: It looks like the config file at 'models\0.125b-FP16__galactica\model.safetensors' is not a valid JSON file.

I really hope these are easy fixes because I very much welcome this PR, more organization options are always welcome.

[zappityzap]

Thanks for taking the time to test. I was unaware how the HF folders worked, as I've only been using the single GGUF files with llamacpp. I'll have to spend some more time understanding how that part works.

[jfmherokiller]

This would be beneficial to my unusual setup where I use lmstudio to manage the models and I have them symlinked in the needed directories under textgen.

[oobabooga]

Thanks for the PR. I understand that lmstudio uses nested directories, but for simplicity, I prefer to store models directly under models/.