Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FR]: Bar chart (categorical) score metric e.g. "low", "medium", "high" #1436

Open
gustavhartz opened this issue Mar 3, 2025 · 2 comments
Open
Assignees
Labels
enhancement New feature or request

Comments

@gustavhartz
Copy link

Proposal summary

Image

Make it possible for evaluation metrics that have categorical/text values

Motivation

What problem are you trying to solve: key approach within the company I currently work at is evaluation of LLM's with categorical rankings: e.g. low, medium, high, level of hallucination

**How are you currently solving this problem?:**Currently I generate plots based on the data manually after a run

What are the benefits of this feature?: This would make evaluation of model performance a lot easier

@gustavhartz gustavhartz added the enhancement New feature or request label Mar 3, 2025
@aadereiko aadereiko self-assigned this Mar 3, 2025
@aadereiko
Copy link
Collaborator

Hey @gustavhartz! A great idea! Thanks for reporting it. I am going to create a ticket internally so our team can take it from there.

OPIK-1150 - for our tracking

@gustavhartz
Copy link
Author

@aadereiko sounds good! Plotting experiment works reasonably well with Weights & Biases, so might be a bit of inspiration to be found there.

Image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants