Add new model (upstage/solar-pro-preview-instruct)

[remy-rec]

We have released a new model and would like to add it to the HELM leaderboard. We have confirmed that it works well on the local leaderboard.
If a merge is possible, could you let me know by when the merge will be completed?
You can check our model's performance at: https://www.upstage.ai/products/solar-pro-preview.

model_deployments.yaml

  - name: upstage/solar-pro-preview-instruct
    model_name: upstage/solar-pro-preview-instruct
    tokenizer_name: upstage/solar-pro-preview-instruct
    max_sequence_length: 4096
    client_spec:
      class_name: "helm.clients.huggingface_client.HuggingFaceClient"
      args:
        torch_dtype: auto
        trust_remote_code: true

model_metadata.yaml

  - name: upstage/solar-pro-preview-instruct
    display_name: solar-pro-preview-instruct (22B)
    description: solar-pro-preview-instruct (22B) is the most intelligent LLM on a single GPU ([blog](https://www.upstage.ai/products/solar-pro-preview))
    creator_organization_name: Upstage
    access: open
    num_parameters: 22000000000
    release_date: 2024-09-11
    tags: [TEXT_MODEL_TAG, LIMITED_FUNCTIONALITY_TEXT_MODEL_TAG]

tokenizer_configs.yaml

  # Upstage
  - name: upstage/solar-pro-preview-instruct
    tokenizer_spec:
      class_name: "helm.tokenizers.huggingface_tokenizer.HuggingFaceTokenizer"
      args:
        trust_remote_code: true
    end_of_text_token: "<|im_end|>"
    prefix_token: "<|startoftext|>"

[yifanmai]

Thanks for the suggestion! I'll discuss with the team internally and figure out what we want to do here.

[nayeon-upstage]

Hi @yifanmai ! I am Nayeon form Upstage. I respectfully request your team's consideration to include Solar in the leaderboard. Please let me know if you require further information or �tasks. Thank you!

[yifanmai]

Hi @nayeon-upstage, I've added Solar to the queue of models to add to the leaderboard. Would you already have results from internal runs that you could share?

[nayeon-upstage]

Hello @yifanmai ! Thank you so much for your attention.Here is the results of solar-pro-preview-instruct(22b) uploaded to the local server through captures! It's a 22b model, but its performance is good compared to larger models. The file is compressed and exceeds 2G. If you need it as a file, I can send it to you via email. (please note that the latest template is not released, so it's a past bench LEADERBOARD_VERSION=v0.3.0)

[yifanmai]

Thanks for the results! Please hold on sending the complete files. I am currently doing some evaluation runs and hopefully we can use those results.

Also, I may be slow to respond over the next week due to the US Thanksgiving holiday.

[nayeon-upstage]

Thank you for the running evaluation. The requested model was the Open model, Solar Pro Preview on HuggingFace. Our Solar Pro has been released today. Would it be possible to evaluate with this new model too?

Although Solar Pro is a closed model, everyone can use it as an API on our console. Do you have any guidance on this adding model other than using HuggingFace or Together AI? We will provide the credits for evaluation with our API endpoint. The tokenizer will be released as well.

Hope you have a wonderful holiday!

[yifanmai]

Hi @nayeon-upstage, congrats on the model launch!

Yes, we can evaluate Solar Pro through the Upstage API. Could you contact me by email at my git commit email address regarding providing credits for evaluation?

[yifanmai]

@nayeon-upstage I see that Solar Pro access is free until January - can I run the evaluation using this free access?

[nayeon-upstage]

@yifanmai Yes please! Thank you 🤗

[yifanmai]

I've added Upstage Solar Pro to the Lite and MMLU leaderboards. It does very well compared to models of the same size.

Thanks again for your help on this!

[nayeon-upstage]

Thanks so much! This is the best news I've heard all year! I'm so grateful for everything you've done.

Have a great holiday! 🎄🎄