Local REST API web server TLS

[LukeDuffy98]

Basic issue is we need to have https on the local REST API server it seems, as we cannot bypass it.

Playground works fine, using onnx works as well. The REST API does not.

Following the basic Get Started from here : https://learn.microsoft.com/en-us/windows/ai/toolkit/toolkit-getting-started?tabs=csharp#download-a-model-from-the-catalog

Using c# as the language

When connecting to the built in server : http://localhost:5272/v1/chat/completions

Bearer token authentication is not permitted for non TLS protected (https) endpoints.

Error Message is :
Unhandled exception. System.AggregateException: Retry failed after 2 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (No connection could be made because the target machine actively refused it. (localhost:5272)) (Bearer token authentication is not permitted for non TLS protected (https) endpoints.)
---> Azure.RequestFailedException: No connection could be made because the target machine actively refused it. (localhost:5272)
---> System.Net.Http.HttpRequestException: No connection could be made because the target machine actively refused it. (localhost:5272)
---> System.Net.Sockets.SocketException (10061): No connection could be made because the target machine actively refused it.
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at Azure.Core.Pipeline.HttpClientTransport.ProcessSyncOrAsync(HttpMessage message, Boolean async) --- End of inner exception stack trace --- at Azure.Core.Pipeline.HttpClientTransport.ProcessSyncOrAsync(HttpMessage message, Boolean async) at Azure.Core.Pipeline.HttpPipelineTransportPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.HttpPipelineExtensions.ProcessMessageAsync(HttpPipeline pipeline, HttpMessage message, RequestContext requestContext, CancellationToken cancellationToken) at Azure.AI.OpenAI.OpenAIClient.GetChatCompletionsStreamingAsync(ChatCompletionsOptions chatCompletionsOptions, CancellationToken cancellationToken) at Program.<Main>$(String[] args) in G:\TestApp\AIToolkitConsumer\AITKConsumer\Program.cs:line 22 at Program.<Main>(String[] args) ---> (Inner Exception #1) System.InvalidOperationException: Bearer token authentication is not permitted for non TLS protected (https) endpoints. at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)<---

Any ideas on how to work around it ?

[microsoft-github-policy-service]

Thank you for contacting us! Any issue or feedback from you is quite important to us. We will do our best to fully respond to your issue as soon as possible. Sometimes additional investigations may be needed, we will usually get back to you within 2 days by adding comments to this issue. Please stay tuned.

[LukeDuffy98]

So it seems to be one of those misleading errors.

I shut down everything overnight and restated today and had a different error, that I could at least work with. The issue seems to be something to do with streaming.

With a working copy, I was able to send multiple requests initially, before the HTTPS error appeared again. Closing VS Code and starting again allowed requests to be sent again. I can also replicate the issue by having a DeploymentName value that does not exist.

I have amended the code as per below with non streaming that works, and streaming that fails still.

// THIS WORKS
Console.WriteLine(client.GetChatCompletions(options).Value.Choices[0].Message.Content);

StreamingResponse streamingChatResponse
= await client.GetChatCompletionsStreamingAsync(options);

// THIS CAUSES ERROR
await foreach (StreamingChatCompletionsUpdate chatChunk in streamingChatResponse)
{
Console.Write(chatChunk.ContentUpdate);
}

[a1exwang]

Hi @LukeDuffy98, thanks for the feedback.

It seems there are 2 issues here:

1. The HTTPS error you mentioned is misleading and seems originated from No connection could be made because the target machine actively refused it. (localhost:5272) in your log. This could be because the local inference server is not started yet. To start it, you need to select a model in Playground. Could you try that?

2. For the error related to streaming, could you attach the detailed error messages or call stacks.

[LukeDuffy98]

Hi Alex

Is this what you need ?

PS G:\TestApp\AIToolkitConsumer\AITKConsumer> dotnet run
Overriding request URI to http://localhost:5272/v1/chat/completions
Unhandled exception. System.InvalidOperationException: The requested operation requires an element of type 'Object', but the target element has type 'Null'.
at System.Text.Json.ThrowHelper.ThrowJsonElementWrongTypeException(JsonTokenType expectedType, JsonTokenType actualType)
at System.Text.Json.JsonElement.EnumerateObject()
at Azure.AI.OpenAI.StreamingChatCompletionsUpdate.StreamingDeltaData.DeserializeStreamingDeltaData(JsonElement element, ModelReaderWriterOptions _)
at Azure.AI.OpenAI.StreamingChatCompletionsUpdate.StreamingChoiceData.DeserializeStreamingChoiceData(JsonElement element, ModelReaderWriterOptions _)
at Azure.AI.OpenAI.StreamingChatCompletionsUpdate.DeserializeStreamingChatCompletionsUpdates(JsonElement element)
at Azure.Core.Sse.SseAsyncEnumerator1.EnumerateFromSseStream(Stream stream, Func2 multiElementDeserializer, CancellationToken cancellationToken)+MoveNext()
at Azure.Core.Sse.SseAsyncEnumerator1.EnumerateFromSseStream(Stream stream, Func2 multiElementDeserializer, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at Program.

$(String[] args) in G:\TestApp\AIToolkitConsumer\AITKConsumer\Program.cs:line 31
at Program.$(String[] args) in G:\TestApp\AIToolkitConsumer\AITKConsumer\Program.cs:line 31
at Program.(String[] args)
PS G:\TestApp\AIToolkitConsumer\AITKConsumer>

Here is the program.cs file for refrence
`// Program.cs
using Azure.AI.OpenAI;

Uri localhostUri = new("http://localhost:5272/v1/chat/completions");

OpenAIClientOptions clientOptions = new();
clientOptions.AddPolicy(
new OverrideRequestUriPolicy(localhostUri),
Azure.Core.HttpPipelinePosition.BeforeTransport);
OpenAIClient client = new(openAIApiKey: "unused", clientOptions);

ChatCompletionsOptions options = new()
{
DeploymentName = "Phi-3-mini-128k-directml-int4-awq-block-128-onnx",
MaxTokens = 5000,
Messages =
{
new ChatRequestSystemMessage("You are a helpful assistant. Be brief and succinct."),
new ChatRequestUserMessage( "Explain in detail the following point : Describe the value of Microsoft Power Apps to build applications"),//"What is the golden ratio?"),
}
};

//Console.WriteLine(client.GetChatCompletions(options).Value.Choices[0].Message.Content);

StreamingResponse streamingChatResponse
= await client.GetChatCompletionsStreamingAsync(options);

await foreach (StreamingChatCompletionsUpdate chatChunk in streamingChatResponse)
{
Console.Write(chatChunk.ContentUpdate);
} `