{"error":"Tool \"get_weather\" has not been added"}

[jambudipa]

My app stopped working and of course I blamed myself.

But returning to the realtime console default app, I am receiving this error when asking about the weather anywhere

{"error":"Tool "get_weather" has not been added"}

...even though the messages subsequently look fine:

function call output
{"latitude":51.5,"longitude":-0.120000124,"generationtime_ms":0.01895427703857422,"utc_offset_seconds":0,"timezone":"GMT","timezone_abbreviation":"GMT","elevation":16,"current_units":{"time":"iso8601","interval":"seconds","temperature_2m":"°C","wind_speed_10m":"km/h"},"current":{"time":"2024-10-04T20:30","interval":900,"temperature_2m":10.8,"wind_speed_10m":5.4}}

assistant
It seems I'm having a bit of trouble fetching the weather right now. Is there anything else I can help you with?

But it claims not to be able to understand the response. My own app failed in a similar way, "there appears to be a problem."

It's been like this for several hours, but nothing on any status page I can find.

[Stevenic]

This error happens when a tool throws an exception... Took me a bit to figure out as well. Likely you're getting an error from the weather API call.

[jambudipa]

No, I don't think that's the case. Even if I replace the return with a simple JSON, and no other code, it still claims the tool has not been added – yet immediately produces the expected response of the tool.

[jambudipa]

Ok @Stevenic , so I wrapped my tool function body in try / catch and, even though the catch is never hit, it works now. Who knows 🤷🏻‍♂️

[jeanmayorga]

Seems like the first time the client doesn't wait fetch to be fulfilled, but the second time you already have the answer

user
What's the water in New York?

function call
get_weather({"lat":40.7128,"lng":-74.006,"location":"New York"})

function call output
{"error":"Tool \"get_weather\" has not been added"}

function call output
{"latitude":40.710335,"longitude":-73.99309,"generationtime_ms":0.026941299438476562,"utc_offset_seconds":0,"timezone":"GMT","timezone_abbreviation":"GMT","elevation":32,"current_units":{"time":"iso8601","interval":"seconds","temperature_2m":"°C","wind_speed_10m":"km/h"},"current":{"time":"2024-10-04T22:00","interval":900,"temperature_2m":19.8,"wind_speed_10m":14.6}}

assistant
(truncated)

user
What's the weather in New York?

assistant
The current temperature in New York is 19.8°C, with wind speeds of 14.6 km/h. How else can I assist you?

[Stevenic]

It also appears to happen if you don't return a value from your handler. You need to always return a value... I created this helper:

    async _onTool(args, handler) {
        try {
            return await handler(args) ?? { success: true };
        } catch (error) {
            console.error('Error handling tool:', error);
            return { error: error.message };
        }
    }

[Stevenic]

I found another issue with tools... If you're running the relay server both the client and the relay server try to run the tool. The issue is the relay server doesn't have a handler to call so it always generates the `Tool "xyz" has not been added" error. I filed an issue with the real time library because its in that code:

openai/openai-realtime-api-beta#14

[jambudipa]

Very thorough ☺️ given me some pointers, thanks.

[Stevenic]

@jambudipa thanks... The race condition between tools and the relay server is likely the most pressing thing to fix @khorwood-openai as that's going to give people fits. I have manual tool calling working now with the changes I pointed out and after tweaking the server_vad threshold things are starting to work better...

[jambudipa]

I mean I realised yesterday just how bloody expensive this API is. Producing audio is $0.24 per minute 😳

Had a look around, that is not much more expensive than many other text to speech solutions though.

[Stevenic]

Had a look around, that is not much more expensive than many other text to speech solutions though.

I did the same investigation. They're actually $0.06 cheaper then Eleven Labs. It's still super pricey.

[jambudipa]

Oh, I looked at them this morning. Must have miscalculated because I thought they were cheaper.

I mean it's a great idea, but at the moment prohibitively expensive. My plan was to build a memorisation aid, but honestly memorising a page of text like this would cost about $5.

[Stevenic]

Must have miscalculated because I thought they were cheaper.

Funny it looks like they just adjusted their pricing lol... I was going off the pricing of their $99/mo pro account but looks like they just dropped that from $0.30/minute to $0.24/minute. I wonder why???

If you go with a higher tier subscription it looks like you can get Eleven Labs down to $0.10/minute but you'll have to signup for their Business (Annual) subscription which is only $15,840 per year.

[Stevenic]

My plan was to build a memorisation aid, but honestly memorising a page of text like this would cost about $5.

You can still build that you just to leverage tools to help keep your cost in check. Basically you want to use the RT API to control the flow of the conversation and not do the work of actually memorizing the page content via RAG. Have the agent call a tool that then uses gpt-4o-mini to extract memories from the current page.

[jambudipa]

Really it's the cost of the audio...

For the moment, I can't think I will be using audio out with the Realtime API. Wildly expensive. I might buy a yacht instead.

[bjsllc]

I've posted the solution here and I've also given credit to those that assisted with the bug. You guys are the real heroes!!!!!
https://www.youtube.com/playlist?list=PLkNKgOpMwm0uvvtrnFn1hYrllN-d95FVV

[projectabel]

Had a look around, that is not much more expensive than many other text to speech solutions though.

I did the same investigation. They're actually $0.06 cheaper then Eleven Labs. It's still super pricey.

That's not true. If you calculate based on a meaningful conversation greater than a few minutes with half a dozen to a dozen exchanges, you're looking at close to $0.75/min (all up). This is if you don't make any efforts to reduce the audio output token carryover dramas.

Without VAD, circa $0.41/min
With VAD but with context truncation (varying degrees of quality loss): $0.27/min (that's the best I've been able to get to).