feedback

[ShiboSoftwareDev]

hello @mattpocock,
I'm Shibo from tscircuit, great work so far.
We used evalite to create a prompt benchmark that would help us find the best prompt for AI to make the most accurate electronic circuits, and I have some questions:

• Can we make tests in the same file run asynchronously?
• Can we get the average score to be displayed in the webpage?
• Can we have a history of the previous attempts, their prompts and results be displayed somewhere on the webpage?

In the current implementation we ask an initial prompt where the AI is asked to create a circuit, the circuit is then validated and passed to the test phase as input, follow up questions will then be asked about the circuit generated previously.

Here's the PR where we integrated evalite into our repo: tscircuit/prompt-benchmarks#9
screenshots:

[mattpocock]

Hey @ShiboSoftwareDev,

This is tremendously exciting, fantastic stuff.

1. By asynchronously, do you mean at the same time? If so, yes - that seems like a sensible default.
2. Yes, I'll add it.
3. Yes, this is in the works.

[mattpocock]

Note that the 'status symbols' do a lot of the heavy lifting of the metadata you proposed above. If you notice a dip, you can scan down to see the changes in the scores.

[ShiboSoftwareDev]

If you plan to implement a graph of the entire suite can we have a file-level metadata?

[mattpocock]

file-level metadata

Could you elaborate on what you mean by this?

[seveibar]

I think what Shibo means is when we design a new prompt, we want to see the actual AI response from the past. For example:

1. We test the system prompt: "Here's how tscircuit works, every root element is a board..."
2. We see how it performed on every user prompt: ${systemPrompt} Please design an LED lamp, ${systemPrompt} Please design an arduino-powered eink display
3. We get results like this:

Prompt	Score
LED Lamp	100%
eink Display	50%

4. We notice that the AI, with the eink display system prompt, is failing to put the prop width and height on <board />
5. We revise the system prompt "Here's how tscircuit works, every root element is a board with width and height..."
6. We get results like this:

Prompt	Score
LED Lamp	50%
eink Display	100%

7. We would now like to view the "LED Lamp" prompt to see why it regressed, how the AI originally responded and why it's wrong now (a big mystery!)

[mattpocock]

@seveibar This is great, thanks - I can put something together for this.