-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feature: AI generated evaluations #407
base: main
Are you sure you want to change the base?
Conversation
a4a69f7
to
3394d11
Compare
Helps people get started with evaluations!
3394d11
to
367501a
Compare
return ( | ||
<DocumentContext.Provider value={document}> | ||
<DocumentContext.Provider value={document || fallbackDocument}> | ||
{children} | ||
</DocumentContext.Provider> | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this makes this hook work properly with frontend state, finally
input.type === 'number' | ||
? EvaluationResultableType.Number | ||
: EvaluationResultableType.Boolean, | ||
detail: input.type === 'number' ? input.metadata : undefined, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All this could be validated inside zod so this checks are not needed. refine
is what you're looking for
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
const allowedTypesSchema = z.enum([
EvaluationResultableType.Number,
EvaluationResultableType.Boolean,
]);
const inputSchema = z
.object({
name: z.string(),
prompt: z.string(),
type: allowedTypesSchema, // Using the restricted enum values
metadata: z
.object({
range: z.object({
from: z.number(),
to: z.number(),
}),
})
.optional(),
})
.refine((data) => {
// If type is 'Number', metadata must be present
if (data.type === EvaluationResultableType.Number && !data.metadata) {
return false;
}
return true;
}, {
message: "Metadata is required when type is 'Number'",
path: ['metadata'],
})
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also instead of metadata call it configDetail
|
||
const suggestedEvaluations = res.object[0] as SuggestedEvaluation[] | ||
const suggestedEvaluation = res.object | ||
console.log('suggested eval: ', suggestedEvaluation) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔥
const { execute: generateEvaluation, isPending: isGenerating } = | ||
useLatitudeAction(generateSuggestedEvaluationsAction, { | ||
onSuccess: () => { | ||
// do nothing |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No need no?
) | ||
} | ||
|
||
export default TypewriterText |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is an atom. Please keep having things organized
description: string | ||
eval_name: string | ||
eval_type: 'number' | 'boolean' | ||
eval_prompt: string |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why snake case?
const evaluation = result[0]! | ||
|
||
// If projectId and documentUuid are provided, connect the evaluation | ||
if (projectId && documentUuid) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hombre, ya que estas
NEXT_PUBLIC_POSTHOG_KEY: '', | ||
NEXT_PUBLIC_POSTHOG_HOST: '', | ||
DATASET_GENERATOR_WORKSPACE_APIKEY: | ||
'e2c21df6-4e2d-4703-9b65-5f4dee0add23', // fake |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
:ecce_homo:
I really hate all this not working by default dev setup
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also we should unifiy under 1 unique project Copilot
all prompts used for suggestions and copilots to simplify the setup and the amount of env vars
Helps people get started with evaluations!