chore(weave): custom provider model built in object backend #3841

jwlee64 · 2025-03-04T20:30:27Z

Description

Adds 3 built in objects
provider
provider_model
LLM_model

runs the code gen to generate the frontend types

adds handling in clickhouse_trace_server_batched and llm_completion for custom models

Diagram of how the new objects interact

video of it working (for this i spun up a server on my local that just passes stuff through to openai) caveat for this to work with litellm, the apis need to be the same as openai

Screen.Recording.2025-03-04.at.12.29.16.PM.mov

core branch with frontend - core/custom-beta

Testing

How was this PR tested?

Summary by CodeRabbit

New Features
- Introduced new schemas for language model configurations, enhancing the flexibility and structure of model parameters.
- Added support for multiple model modes, including options for 'chat' and 'completion'.
- Enhanced capabilities for managing provider information and model parameters, improving integration with custom providers.
Bug Fixes
- Improved error handling for API interactions and custom provider logic, ensuring a more reliable user experience.
Refactor
- Streamlined the logic for processing model requests and handling API keys, enhancing clarity and consistency.

circle-job-mirror · 2025-03-04T20:30:29Z

Preview this PR with FeatureBee: https://beta.wandb.ai/?betaVersion=689eac05076100269778c89a4a497f494b8792b6

coderabbitai · 2025-03-04T20:35:47Z

Walkthrough

This pull request introduces several new Zod schemas and TypeScript types for built-in object classes related to language model configurations and providers. It updates error handling and control flow in the trace server by distinguishing between standard and custom models. New classes and enumerations are added in separate files for LLM models and providers, and these classes are registered in the global registry. Additionally, the custom provider flow is enhanced by updating the API call parameters and including a dedicated function to extract provider details.

Changes

File(s)	Change Summary
`weave-js/.../generatedBuiltinObjectClasses.zod.ts`	Added new Zod schemas and inferred types for `ReturnType`, `ModelMode`, `ModelParams`, `Provider`, `ProviderModel`, and `LlmModel`; registered in the builtin object class registry.
`weave/trace_server/clickhouse_trace_server_batched.py`	Updated `completions_create` to check model type (standard vs custom) with improved error handling and clarity by using keyword arguments.
`weave/trace_server/interface/builtin_object_classes/builtin_object_registry.py`	Imported and registered new classes: `Provider`, `LLMModel`, and `ProviderModel`.
`weave/trace_server/interface/builtin_object_classes/{llm_model.py, provider.py}`	New files added: `llm_model.py` (defines `ModelMode`, `ModelParams`, `ProviderModel`, `LLMModel`) and `provider.py` (defines `ReturnType` enum and `Provider` class).
`weave/trace_server/llm_completion.py`	Enhanced `lite_llm_completion` to accept additional parameters (`base_url`, `extra_headers`, `return_type`), updated custom provider logic, and added `get_custom_provider_info` function with detailed error handling.

Sequence Diagram(s)

sequenceDiagram
    participant C as Client Request
    participant CS as ClickHouseTraceServer
    participant CP as CustomProviderService
    participant LLM as lite_llm_completion

    C->>CS: Send completions_create request
    CS->>CS: Check model info (standard vs custom)
    alt Custom Model
        CS->>CP: get_custom_provider_info(project_id, model_name, obj_read_func)
        CP-->>CS: Return provider details or error
        CS->>LLM: Call lite_llm_completion(api_key, inputs, provider, base_url, extra_headers, return_type)
    else Standard Model
        CS->>LLM: Call lite_llm_completion(api_key, inputs, provider)
    end
    LLM-->>CS: Return API response
    CS-->>C: Send final response

Possibly related PRs

chore(weave): Add gpt-4.5-preview and deepseek to playground #3830: The changes in the main PR are related to the new schemas and types for model parameters and providers, which directly connect to the new Provider and ModelParams classes introduced in the retrieved PR.

Suggested reviewers

gtarpenning
andrewtruong

Poem

I’m a little bunny, hop, hop in the code,
New schemas and models lighten my abode.
Providers and errors handled with care,
In custom flows, we’re quite debonair.
With every hop, our code’s set to explode!
🐇 Keep on hopping through the digital mode!

✨ Finishing Touches

📝 Generate Docstrings

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

coderabbitai

Actionable comments posted: 0

🧹 Nitpick comments (9)

weave/trace_server/interface/builtin_object_classes/provider.py (2)
6-7: Consider adding more ReturnType options for future extensibility.

Currently, the ReturnType enum only includes OPENAI. As you're building custom provider functionality, you might want to consider whether other return types (like ANTHROPIC, COHERE, etc.) should be added to make the enum more future-proof.

1-14: Add docstrings to explain class purposes and usage.

The file lacks explanatory docstrings that would help developers understand the purpose of these classes and how to use them. Consider adding class-level docstrings explaining what a Provider represents in your system and how the ReturnType impacts functionality.
from enum import Enum

from weave.trace_server.interface.builtin_object_classes import base_object_def


class ReturnType(str, Enum):
+    """Defines the format in which provider responses should be returned.
+    
+    Currently only supports OpenAI-compatible response format.
+    """
    OPENAI = "openai"


class Provider(base_object_def.BaseObject):
+    """Represents an LLM API provider configuration.
+    
+    Stores connection details for an LLM provider API including the base URL,
+    the name of the API key to use, any extra headers required, and the expected
+    response format.
+    """
    base_url: str
    api_key_name: str
    extra_headers: dict[str, str] = {}
    return_type: ReturnType = ReturnType.OPENAI
weave/trace_server/interface/builtin_object_classes/llm_model.py (1)
1-33: Add docstrings to explain relationships between model classes.

The file lacks explanatory docstrings that would help developers understand the purpose of these classes and their relationships. Clear documentation would help explain the difference between LLMModel and ProviderModel and how they work together.
from enum import Enum
from typing import Optional, Union

from pydantic import BaseModel

from weave.trace_server.interface.builtin_object_classes import base_object_def


class ModelMode(str, Enum):
+    """Defines the interaction mode for language models.
+    
+    COMPLETION: For models that complete text (continuation)
+    CHAT: For models that engage in conversational exchanges
+    """
    COMPLETION = "completion"
    CHAT = "chat"


class ModelParams(BaseModel):
+    """Configuration parameters for language model inference.
+    
+    Contains common parameters used to control language model outputs
+    such as temperature, top_p sampling, token limits, and penalties.
+    """
    temperature: Optional[float] = None
    top_p: Optional[float] = None
    max_tokens: Optional[int] = None
    presence_penalty: Optional[float] = None
    frequency_penalty: Optional[float] = None
    stop: Optional[Union[str, list[str]]] = None


class ProviderModel(base_object_def.BaseObject):
+    """Represents a specific model offered by a provider.
+    
+    Links to a provider configuration and specifies model-specific
+    details such as maximum token length and interaction mode.
+    """
    provider: base_object_def.RefStr
    max_tokens: int
    mode: ModelMode = ModelMode.CHAT


class LLMModel(base_object_def.BaseObject):
+    """Represents a configured LLM for use in the application.
+    
+    Connects to a provider model and optionally includes a default prompt
+    and parameter settings to use with the model.
+    """
    provider_model: base_object_def.RefStr
    prompt: Optional[base_object_def.RefStr] = None
    default_params: ModelParams = ModelParams()
weave/trace_server/clickhouse_trace_server_batched.py (2)
1666-1679: Improve error handling granularity.

The current error handling uses a broad try-except block that catches all exceptions, which could mask important specific errors. Consider catching more specific exceptions or at least logging the exception details before returning the error response.
try:
    (
        base_url,
        api_key,
        extra_headers,
        return_type,
        actual_model_name,
    ) = get_custom_provider_info(
        project_id=req.project_id,
        model_name=model_name,
        obj_read_func=self.obj_read,
    )
-except Exception as e:
+except NotFoundError as e:
    return tsi.CompletionsCreateRes(response={"error": str(e)})
+except ValueError as e:
+    # Handle specific validation errors
+    logger.warning(f"Invalid custom provider configuration: {e}")
+    return tsi.CompletionsCreateRes(response={"error": f"Invalid provider configuration: {str(e)}"})
+except Exception as e:
+    # Log unexpected errors
+    logger.error(f"Unexpected error in custom provider flow: {e}", exc_info=True)
+    return tsi.CompletionsCreateRes(response={"error": f"Internal error: {str(e)}"})
1630-1661: Add validation for API key existence before proceeding.

For better error handling, consider validating that the API key exists before proceeding with the API call, especially for providers that aren't handled like bedrock.
# Handle standard model case
# 1. We get the model info from the map
# 2. We fetch the API key, with the secret fetcher
# 3. We set the provider, to the litellm provider
# 4. If no api key, we raise an error, except for bedrock and bedrock_converse (we fetch bedrock credentials, in lite_llm_completion)

secret_name = model_info.get("api_key_name")
if not secret_name:
    raise InvalidRequest(f"No secret name found for model {model_name}")

secret_fetcher = _secret_fetcher_context.get()
if not secret_fetcher:
    raise InvalidRequest(
        f"No secret fetcher found, cannot fetch API key for model {model_name}"
    )

api_key = (
    secret_fetcher.fetch(secret_name).get("secrets", {}).get(secret_name)
)
provider = model_info.get("litellm_provider", "openai")

# We fetch bedrock credentials, in lite_llm_completion, later
if not api_key and provider != "bedrock" and provider != "bedrock_converse":
    raise MissingLLMApiKeyError(
        f"No API key {secret_name} found for model {model_name}",
        api_key_name=secret_name,
    )
+
+# Log provider information for debugging
+logger.debug(f"Using provider {provider} for model {model_name}")
weave/trace_server/llm_completion.py (2)

42-66: Catch narrower exceptions or provide additional error handling insights
Catching a broad Exception here can obscure specific failure modes. Consider narrowing the scope or at least logging more debug information (like full tracebacks) before returning the error response. Additionally, evaluating or sanitizing the user-supplied base_url can help avoid potential security vulnerabilities.

155-259: Validate the exact composition of the model name parts
This function only checks if the model_name split length is < 2; consider verifying precisely two parts (e.g., “provider_id/model_id”) to avoid silently ignoring extra segments. This small adjustment can help maintain data integrity and reduce user confusion.

weave-js/src/components/PagePanelComponents/Home/Browse3/pages/wfReactInterface/generatedBuiltinObjectClasses.zod.ts (2)

9-10: Future-proofing return types
Currently, 'openai' is the sole enum value for ReturnTypeSchema. If future expansions are planned (e.g., 'custom', 'bedrock'), consider adding them here to avoid migrations later.

41-49: Unify the type of “stop” parameter
Allowing 'stop' to be a single string or array may complicate downstream usage. Converting single strings into a one-element array can simplify internal logic and reduce condition checks.

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between e577d26 and 9d75a0d.

⛔ Files ignored due to path filters (1)

weave/trace_server/interface/builtin_object_classes/generated/generated_builtin_object_class_schemas.json is excluded by !**/generated/**

📒 Files selected for processing (6)

weave-js/src/components/PagePanelComponents/Home/Browse3/pages/wfReactInterface/generatedBuiltinObjectClasses.zod.ts (4 hunks)
weave/trace_server/clickhouse_trace_server_batched.py (3 hunks)
weave/trace_server/interface/builtin_object_classes/builtin_object_registry.py (2 hunks)
weave/trace_server/interface/builtin_object_classes/llm_model.py (1 hunks)
weave/trace_server/interface/builtin_object_classes/provider.py (1 hunks)
weave/trace_server/llm_completion.py (4 hunks)

🧰 Additional context used

📓 Path-based instructions (2)

`**/*.py`: Focus on pythonic code patterns. Check for proper...

**/*.py: Focus on pythonic code patterns.
Check for proper exception handling.
Verify type hints usage where applicable.
Look for potential performance improvements.
Don't comment on formatting if black/isort is configured.
Check for proper dependency injection patterns.
Verify proper async handling if applicable.

weave/trace_server/interface/builtin_object_classes/provider.py
weave/trace_server/interface/builtin_object_classes/builtin_object_registry.py
weave/trace_server/clickhouse_trace_server_batched.py
weave/trace_server/llm_completion.py
weave/trace_server/interface/builtin_object_classes/llm_model.py

`**/*.{js,jsx,ts,tsx}`: Focus on architectural and logical i...

**/*.{js,jsx,ts,tsx}: Focus on architectural and logical issues rather than style (assuming ESLint is in place).
Flag potential memory leaks and performance bottlenecks.
Check for proper error handling and async/await usage.
Avoid strict enforcement of try/catch blocks - accept Promise chains, early returns, and other clear error handling patterns. These are acceptable as long as they maintain clarity and predictability.
Ensure proper type usage in TypeScript files.
Look for security vulnerabilities in data handling.
Don't comment on formatting if prettier is configured.
Verify proper React hooks usage and component lifecycle.
Check for proper state management patterns.

weave-js/src/components/PagePanelComponents/Home/Browse3/pages/wfReactInterface/generatedBuiltinObjectClasses.zod.ts

🔇 Additional comments (16)

weave/trace_server/interface/builtin_object_classes/builtin_object_registry.py (2)

9-12: LGTM! Imports are correctly organized.

The imports for the new classes are properly organized and follow the existing import pattern.

37-39: LGTM! Classes are properly registered.

The new classes are correctly registered in the registry using the established pattern.

weave/trace_server/interface/builtin_object_classes/llm_model.py (4)

9-11: LGTM! ModelMode enum is properly defined.

The enum is correctly implemented as a string-based enum with appropriate values for completion and chat modes.

14-20: LGTM! ModelParams are properly defined with appropriate types.

The parameters cover common LLM configuration options with correct optional typing.

23-26: LGTM! ProviderModel is properly implemented.

The class correctly extends BaseObject and includes the necessary fields with appropriate types.

29-32: LGTM! LLMModel is properly implemented.

The class correctly extends BaseObject and includes the necessary fields with appropriate types.

weave/trace_server/clickhouse_trace_server_batched.py (4)

89-92: LGTM! Import statements properly organized.

The imports for get_custom_provider_info and lite_llm_completion are correctly added.

1616-1685: Clear organization of standard vs. custom model handling.

The code clearly organizes the flow for handling standard models versus custom models with good inline comments explaining the logic.

1690-1697: LGTM! API call parameters properly expanded.

The lite_llm_completion call correctly uses keyword arguments with all the necessary parameters for both standard and custom model flows.

1721-1721: LGTM! Model name variable usage is consistent.

The code correctly uses the model_name variable instead of directly referencing req.inputs.model for consistency.

weave/trace_server/llm_completion.py (1)

17-19: New parameters appear consistent and well-typed
The optional parameters for base_url, extra_headers, and return_type integrate smoothly with the rest of the function and follow Pythonic type hint conventions.

weave-js/src/components/PagePanelComponents/Home/Browse3/pages/wfReactInterface/generatedBuiltinObjectClasses.zod.ts (5)

12-13: Model mode definitions look good
The ModelModeSchema with 'chat' and 'completion' is straightforward and aligns with typical usage.

51-60: Provider schema aligns with custom provider approach
No issues found. Requiring a base_url and api_key_name is consistent with the addition of custom provider logic.

61-69: Clear definition of provider models
Mandatory max_tokens and optional mode fields provide flexibility. The schema clearly enforces minimal constraints for valid ProviderModel objects.

100-108: LLM model structure is well-formed
This expanded schema correctly references default_params and retains optional fields. It looks coherent with new functionality around providers.

122-124: Successful registry updates
Adding LLMModel, Provider, and ProviderModel to the registry ensures these objects are recognized throughout the application.

coderabbitai

Actionable comments posted: 0

🧹 Nitpick comments (2)

weave/trace_server/clickhouse_trace_server_batched.py (2)

1662-1689: Use specific exception types instead of catching all exceptions

The current implementation catches all exceptions when fetching custom provider info, which could mask specific errors and make debugging more difficult.

Consider catching specific exceptions and handling them appropriately:

-            try:
-                (
-                    base_url,
-                    api_key,
-                    extra_headers,
-                    return_type,
-                    actual_model_name,
-                ) = get_custom_provider_info(
-                    project_id=req.project_id,
-                    model_name=model_name,
-                    obj_read_func=self.obj_read,
-                )
-            except Exception as e:
-                return tsi.CompletionsCreateRes(response={"error": str(e)})
+            try:
+                (
+                    base_url,
+                    api_key,
+                    extra_headers,
+                    return_type,
+                    actual_model_name,
+                ) = get_custom_provider_info(
+                    project_id=req.project_id,
+                    model_name=model_name,
+                    obj_read_func=self.obj_read,
+                )
+            except NotFoundError as e:
+                return tsi.CompletionsCreateRes(response={"error": f"Custom model not found: {str(e)}"})
+            except InvalidRequest as e:
+                return tsi.CompletionsCreateRes(response={"error": f"Invalid custom model configuration: {str(e)}"})
+            except Exception as e:
+                return tsi.CompletionsCreateRes(response={"error": f"Unexpected error with custom model: {str(e)}"})

1685-1689: Extract Ollama model handling to a helper function

The special handling for Ollama models is embedded in the main method. This logic would be clearer if extracted to a helper function.

Consider refactoring to a helper function:

+    def _format_model_name_for_provider(self, provider_name: str, actual_model_name: str) -> str:
+        """Format the model name according to provider-specific requirements."""
+        if "ollama" in provider_name:
+            return f"ollama/{actual_model_name}"
+        return actual_model_name
+

Then in the main method:

-            req.inputs.model = (
-                "ollama/" + actual_model_name
-                if "ollama" in model_name
-                else actual_model_name
-            )
+            req.inputs.model = self._format_model_name_for_provider(model_name, actual_model_name)

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 9d75a0d and 01a21f6.

📒 Files selected for processing (1)

weave/trace_server/clickhouse_trace_server_batched.py (3 hunks)

🧰 Additional context used

📓 Path-based instructions (1)

`**/*.py`: Focus on pythonic code patterns. Check for proper...

**/*.py: Focus on pythonic code patterns.
Check for proper exception handling.
Verify type hints usage where applicable.
Look for potential performance improvements.
Don't comment on formatting if black/isort is configured.
Check for proper dependency injection patterns.
Verify proper async handling if applicable.

weave/trace_server/clickhouse_trace_server_batched.py

⏰ Context from checks skipped due to timeout of 90000ms (970)

GitHub Check: Legacy (Query Service) Python unit tests (1)
GitHub Check: Legacy (Query Service) Python unit tests (0)
GitHub Check: WeaveJS Lint and Compile
GitHub Check: Trace nox tests (3, 13, pandas-test)
GitHub Check: Trace nox tests (3, 13, scorers)
GitHub Check: Trace nox tests (3, 13, vertexai)
GitHub Check: Trace nox tests (3, 13, openai)
GitHub Check: Trace nox tests (3, 13, notdiamond)
GitHub Check: Trace nox tests (3, 13, mistral1)
GitHub Check: Trace nox tests (3, 13, mistral0)
GitHub Check: Trace nox tests (3, 13, llamaindex)
GitHub Check: Trace nox tests (3, 13, litellm)
GitHub Check: Trace nox tests (3, 13, langchain)
GitHub Check: Trace nox tests (3, 13, instructor)
GitHub Check: Trace nox tests (3, 13, google_ai_studio)
GitHub Check: Trace nox tests (3, 13, huggingface)
GitHub Check: Trace nox tests (3, 13, groq)
GitHub Check: Trace nox tests (3, 13, dspy)
GitHub Check: Trace nox tests (3, 13, cohere)
GitHub Check: Trace nox tests (3, 13, cerebras)
GitHub Check: Trace nox tests (3, 13, bedrock)
GitHub Check: Trace nox tests (3, 13, anthropic)
GitHub Check: Trace nox tests (3, 13, trace_server)
GitHub Check: Trace nox tests (3, 13, trace)
GitHub Check: Trace nox tests (3, 12, pandas-test)
GitHub Check: Trace nox tests (3, 12, scorers)
GitHub Check: Trace nox tests (3, 12, vertexai)
GitHub Check: Trace nox tests (3, 12, openai)
GitHub Check: Trace nox tests (3, 12, notdiamond)
GitHub Check: Trace nox tests (3, 12, mistral1)
GitHub Check: Trace nox tests (3, 12, mistral0)
GitHub Check: Trace nox tests (3, 12, llamaindex)
GitHub Check: Trace nox tests (3, 12, litellm)
GitHub Check: Trace nox tests (3, 12, langchain)
GitHub Check: Trace nox tests (3, 12, instructor)
GitHub Check: Trace nox tests (3, 12, google_ai_studio)
GitHub Check: Trace nox tests (3, 12, huggingface)
GitHub Check: Trace nox tests (3, 12, groq)
GitHub Check: Trace nox tests (3, 12, dspy)
GitHub Check: Trace nox tests (3, 12, cohere)
GitHub Check: Trace nox tests (3, 12, cerebras)
GitHub Check: Trace nox tests (3, 12, bedrock)
GitHub Check: Trace nox tests (3, 12, anthropic)
GitHub Check: Trace nox tests (3, 12, trace_server)
GitHub Check: Trace nox tests (3, 12, trace)
GitHub Check: Trace nox tests (3, 11, pandas-test)
GitHub Check: Trace nox tests (3, 11, scorers)
GitHub Check: Trace nox tests (3, 11, vertexai)
GitHub Check: Trace nox tests (3, 11, openai)
GitHub Check: Trace nox tests (3, 11, notdiamond)
GitHub Check: Trace nox tests (3, 11, mistral1)
GitHub Check: Trace nox tests (3, 11, mistral0)
GitHub Check: Trace nox tests (3, 11, llamaindex)
GitHub Check: Trace nox tests (3, 11, litellm)
GitHub Check: Trace nox tests (3, 11, langchain)
GitHub Check: Trace nox tests (3, 11, instructor)
GitHub Check: Trace nox tests (3, 11, google_ai_studio)
GitHub Check: Trace nox tests (3, 11, huggingface)
GitHub Check: Trace nox tests (3, 11, groq)
GitHub Check: Trace nox tests (3, 11, dspy)
GitHub Check: Trace nox tests (3, 11, cohere)
GitHub Check: Trace nox tests (3, 11, cerebras)
GitHub Check: Trace nox tests (3, 11, bedrock)
GitHub Check: Trace nox tests (3, 11, anthropic)
GitHub Check: Trace nox tests (3, 11, trace_server)
GitHub Check: Trace nox tests (3, 11, trace)
GitHub Check: Trace nox tests (3, 10, pandas-test)
GitHub Check: Trace nox tests (3, 10, scorers)
GitHub Check: Trace nox tests (3, 10, vertexai)
GitHub Check: Trace nox tests (3, 10, openai)
GitHub Check: Trace nox tests (3, 10, notdiamond)
GitHub Check: Trace nox tests (3, 10, mistral1)
GitHub Check: Trace nox tests (3, 10, mistral0)
GitHub Check: Trace nox tests (3, 10, llamaindex)
GitHub Check: Trace nox tests (3, 10, litellm)
GitHub Check: Trace nox tests (3, 10, langchain)
GitHub Check: Trace nox tests (3, 10, instructor)
GitHub Check: Trace nox tests (3, 10, google_ai_studio)
GitHub Check: Trace nox tests (3, 10, huggingface)
GitHub Check: Trace nox tests (3, 10, groq)
GitHub Check: Trace nox tests (3, 10, dspy)
GitHub Check: Trace nox tests (3, 10, cohere)
GitHub Check: Trace nox tests (3, 10, cerebras)
GitHub Check: Trace nox tests (3, 10, bedrock)
GitHub Check: Trace nox tests (3, 10, anthropic)
GitHub Check: Trace nox tests (3, 10, trace_server)
GitHub Check: Trace nox tests (3, 10, trace)
GitHub Check: Trace nox tests (3, 9, pandas-test)
GitHub Check: Trace nox tests (3, 9, scorers)
GitHub Check: Trace nox tests (3, 9, vertexai)
GitHub Check: Trace nox tests (3, 9, openai)
GitHub Check: Trace nox tests (3, 9, notdiamond)
GitHub Check: Trace nox tests (3, 9, mistral1)
GitHub Check: Trace nox tests (3, 9, mistral0)
GitHub Check: Trace nox tests (3, 9, llamaindex)
GitHub Check: Trace nox tests (3, 9, litellm)
GitHub Check: Trace nox tests (3, 9, langchain)
GitHub Check: Trace nox tests (3, 9, instructor)
GitHub Check: Legacy (Query Service) Python unit tests (1)
GitHub Check: Legacy (Query Service) Python unit tests (0)
GitHub Check: WeaveJS Lint and Compile
GitHub Check: Trace nox tests (3, 13, pandas-test)
GitHub Check: Trace nox tests (3, 13, scorers)
GitHub Check: Trace nox tests (3, 13, vertexai)
GitHub Check: Trace nox tests (3, 13, openai)
GitHub Check: Trace nox tests (3, 13, notdiamond)
GitHub Check: Trace nox tests (3, 13, mistral1)
GitHub Check: Trace nox tests (3, 13, mistral0)
GitHub Check: Trace nox tests (3, 13, llamaindex)
GitHub Check: Trace nox tests (3, 13, litellm)
GitHub Check: Trace nox tests (3, 13, langchain)
GitHub Check: Trace nox tests (3, 13, instructor)
GitHub Check: Trace nox tests (3, 13, google_ai_studio)
GitHub Check: Trace nox tests (3, 13, huggingface)
GitHub Check: Trace nox tests (3, 13, groq)
GitHub Check: Trace nox tests (3, 13, dspy)
GitHub Check: Trace nox tests (3, 13, cohere)
GitHub Check: Trace nox tests (3, 13, cerebras)
GitHub Check: Trace nox tests (3, 13, bedrock)
GitHub Check: Trace nox tests (3, 13, anthropic)
GitHub Check: Trace nox tests (3, 13, trace_server)
GitHub Check: Trace nox tests (3, 13, trace)
GitHub Check: Trace nox tests (3, 12, pandas-test)
GitHub Check: Trace nox tests (3, 12, scorers)
GitHub Check: Trace nox tests (3, 12, vertexai)
GitHub Check: Trace nox tests (3, 12, openai)
GitHub Check: Trace nox tests (3, 12, notdiamond)
GitHub Check: Trace nox tests (3, 12, mistral1)
GitHub Check: Trace nox tests (3, 12, mistral0)
GitHub Check: Trace nox tests (3, 12, llamaindex)
GitHub Check: Trace nox tests (3, 12, litellm)
GitHub Check: Trace nox tests (3, 12, langchain)
GitHub Check: Trace nox tests (3, 12, instructor)
GitHub Check: Trace nox tests (3, 12, google_ai_studio)
GitHub Check: Trace nox tests (3, 12, huggingface)
GitHub Check: Trace nox tests (3, 12, groq)
GitHub Check: Trace nox tests (3, 12, dspy)
GitHub Check: Trace nox tests (3, 12, cohere)
GitHub Check: Trace nox tests (3, 12, cerebras)
GitHub Check: Trace nox tests (3, 12, bedrock)
GitHub Check: Trace nox tests (3, 12, anthropic)
GitHub Check: Trace nox tests (3, 12, trace_server)
GitHub Check: Trace nox tests (3, 12, trace)
GitHub Check: Trace nox tests (3, 11, pandas-test)
GitHub Check: Trace nox tests (3, 11, scorers)
GitHub Check: Trace nox tests (3, 11, vertexai)
GitHub Check: Trace nox tests (3, 11, openai)
GitHub Check: Trace nox tests (3, 11, notdiamond)
GitHub Check: Trace nox tests (3, 11, mistral1)
GitHub Check: Trace nox tests (3, 11, mistral0)
GitHub Check: Trace nox tests (3, 11, llamaindex)
GitHub Check: Trace nox tests (3, 11, litellm)
GitHub Check: Trace nox tests (3, 11, langchain)
GitHub Check: Trace nox tests (3, 11, instructor)
GitHub Check: Trace nox tests (3, 11, google_ai_studio)
GitHub Check: Trace nox tests (3, 11, huggingface)
GitHub Check: Trace nox tests (3, 11, groq)
GitHub Check: Trace nox tests (3, 11, dspy)
GitHub Check: Trace nox tests (3, 11, cohere)
GitHub Check: Trace nox tests (3, 11, cerebras)
GitHub Check: Trace nox tests (3, 11, bedrock)
GitHub Check: Trace nox tests (3, 11, anthropic)
GitHub Check: Trace nox tests (3, 11, trace_server)
GitHub Check: Trace nox tests (3, 11, trace)
GitHub Check: Trace nox tests (3, 10, pandas-test)
GitHub Check: Trace nox tests (3, 10, scorers)
GitHub Check: Trace nox tests (3, 10, vertexai)
GitHub Check: Trace nox tests (3, 10, openai)
GitHub Check: Trace nox tests (3, 10, notdiamond)
GitHub Check: Trace nox tests (3, 10, mistral1)
GitHub Check: Trace nox tests (3, 10, mistral0)
GitHub Check: Trace nox tests (3, 10, llamaindex)
GitHub Check: Trace nox tests (3, 10, litellm)
GitHub Check: Trace nox tests (3, 10, langchain)
GitHub Check: Trace nox tests (3, 10, instructor)
GitHub Check: Trace nox tests (3, 10, google_ai_studio)
GitHub Check: Trace nox tests (3, 10, huggingface)
GitHub Check: Trace nox tests (3, 10, groq)
GitHub Check: Trace nox tests (3, 10, dspy)
GitHub Check: Trace nox tests (3, 10, cohere)
GitHub Check: Trace nox tests (3, 10, cerebras)
GitHub Check: Trace nox tests (3, 10, bedrock)
GitHub Check: Trace nox tests (3, 10, anthropic)
GitHub Check: Trace nox tests (3, 10, trace_server)
GitHub Check: Trace nox tests (3, 10, trace)
GitHub Check: Trace nox tests (3, 9, pandas-test)
GitHub Check: Trace nox tests (3, 9, scorers)
GitHub Check: Trace nox tests (3, 9, vertexai)
GitHub Check: Trace nox tests (3, 9, openai)
GitHub Check: Trace nox tests (3, 9, notdiamond)
GitHub Check: Trace nox tests (3, 9, mistral1)
GitHub Check: Trace nox tests (3, 9, mistral0)
GitHub Check: Trace nox tests (3, 9, llamaindex)
GitHub Check: Trace nox tests (3, 9, litellm)
GitHub Check: Trace nox tests (3, 9, langchain)
GitHub Check: Trace nox tests (3, 9, instructor)
GitHub Check: Legacy (Query Service) Python unit tests (1)
GitHub Check: Legacy (Query Service) Python unit tests (0)
GitHub Check: WeaveJS Lint and Compile
GitHub Check: Trace nox tests (3, 13, pandas-test)
GitHub Check: Trace nox tests (3, 13, scorers)
GitHub Check: Trace nox tests (3, 13, vertexai)
GitHub Check: Trace nox tests (3, 13, openai)
GitHub Check: Trace nox tests (3, 13, notdiamond)
GitHub Check: Trace nox tests (3, 13, mistral1)
GitHub Check: Trace nox tests (3, 13, mistral0)
GitHub Check: Trace nox tests (3, 13, llamaindex)
GitHub Check: Trace nox tests (3, 13, litellm)
GitHub Check: Trace nox tests (3, 13, langchain)
GitHub Check: Trace nox tests (3, 13, instructor)
GitHub Check: Trace nox tests (3, 13, google_ai_studio)
GitHub Check: Trace nox tests (3, 13, huggingface)
GitHub Check: Trace nox tests (3, 13, groq)
GitHub Check: Trace nox tests (3, 13, dspy)
GitHub Check: Trace nox tests (3, 13, cohere)
GitHub Check: Trace nox tests (3, 13, cerebras)
GitHub Check: Trace nox tests (3, 13, bedrock)
GitHub Check: Trace nox tests (3, 13, anthropic)
GitHub Check: Trace nox tests (3, 13, trace_server)
GitHub Check: Trace nox tests (3, 13, trace)
GitHub Check: Trace nox tests (3, 12, pandas-test)
GitHub Check: Trace nox tests (3, 12, scorers)
GitHub Check: Trace nox tests (3, 12, vertexai)
GitHub Check: Trace nox tests (3, 12, openai)
GitHub Check: Trace nox tests (3, 12, notdiamond)
GitHub Check: Trace nox tests (3, 12, mistral1)
GitHub Check: Trace nox tests (3, 12, mistral0)
GitHub Check: Trace nox tests (3, 12, llamaindex)
GitHub Check: Trace nox tests (3, 12, litellm)
GitHub Check: Trace nox tests (3, 12, langchain)
GitHub Check: Trace nox tests (3, 12, instructor)
GitHub Check: Trace nox tests (3, 12, google_ai_studio)
GitHub Check: Trace nox tests (3, 12, huggingface)
GitHub Check: Trace nox tests (3, 12, groq)
GitHub Check: Trace nox tests (3, 12, dspy)
GitHub Check: Trace nox tests (3, 12, cohere)
GitHub Check: Trace nox tests (3, 12, cerebras)
GitHub Check: Trace nox tests (3, 12, bedrock)
GitHub Check: Trace nox tests (3, 12, anthropic)
GitHub Check: Trace nox tests (3, 12, trace_server)
GitHub Check: Trace nox tests (3, 12, trace)
GitHub Check: Trace nox tests (3, 11, pandas-test)
GitHub Check: Trace nox tests (3, 11, scorers)
GitHub Check: Trace nox tests (3, 11, vertexai)
GitHub Check: Trace nox tests (3, 11, openai)
GitHub Check: Trace nox tests (3, 11, notdiamond)
GitHub Check: Trace nox tests (3, 11, mistral1)
GitHub Check: Trace nox tests (3, 11, mistral0)
GitHub Check: Trace nox tests (3, 11, llamaindex)
GitHub Check: Trace nox tests (3, 11, litellm)
GitHub Check: Trace nox tests (3, 11, langchain)
GitHub Check: Trace nox tests (3, 11, instructor)
GitHub Check: Trace nox tests (3, 11, google_ai_studio)
GitHub Check: Trace nox tests (3, 11, huggingface)
GitHub Check: Trace nox tests (3, 11, groq)
GitHub Check: Trace nox tests (3, 11, dspy)
GitHub Check: Trace nox tests (3, 11, cohere)
GitHub Check: Trace nox tests (3, 11, cerebras)
GitHub Check: Trace nox tests (3, 11, bedrock)
GitHub Check: Trace nox tests (3, 11, anthropic)
GitHub Check: Trace nox tests (3, 11, trace_server)
GitHub Check: Trace nox tests (3, 11, trace)
GitHub Check: Trace nox tests (3, 10, pandas-test)
GitHub Check: Trace nox tests (3, 10, scorers)
GitHub Check: Trace nox tests (3, 10, vertexai)
GitHub Check: Trace nox tests (3, 10, openai)
GitHub Check: Trace nox tests (3, 10, notdiamond)
GitHub Check: Trace nox tests (3, 10, mistral1)
GitHub Check: Trace nox tests (3, 10, mistral0)
GitHub Check: Trace nox tests (3, 10, llamaindex)
GitHub Check: Trace nox tests (3, 10, litellm)
GitHub Check: Trace nox tests (3, 10, langchain)
GitHub Check: Trace nox tests (3, 10, instructor)
GitHub Check: Trace nox tests (3, 10, google_ai_studio)
GitHub Check: Trace nox tests (3, 10, huggingface)
GitHub Check: Trace nox tests (3, 10, groq)
GitHub Check: Trace nox tests (3, 10, dspy)
GitHub Check: Trace nox tests (3, 10, cohere)
GitHub Check: Trace nox tests (3, 10, cerebras)
GitHub Check: Trace nox tests (3, 10, bedrock)
GitHub Check: Trace nox tests (3, 10, anthropic)
GitHub Check: Trace nox tests (3, 10, trace_server)
GitHub Check: Trace nox tests (3, 10, trace)
GitHub Check: Trace nox tests (3, 9, pandas-test)
GitHub Check: Trace nox tests (3, 9, scorers)
GitHub Check: Trace nox tests (3, 9, vertexai)
GitHub Check: Trace nox tests (3, 9, openai)
GitHub Check: Trace nox tests (3, 9, notdiamond)
GitHub Check: Trace nox tests (3, 9, mistral1)
GitHub Check: Trace nox tests (3, 9, mistral0)
GitHub Check: Trace nox tests (3, 9, llamaindex)
GitHub Check: Trace nox tests (3, 9, litellm)
GitHub Check: Trace nox tests (3, 9, langchain)
GitHub Check: Trace nox tests (3, 9, instructor)
GitHub Check: Legacy (Query Service) Python unit tests (1)
GitHub Check: Legacy (Query Service) Python unit tests (0)
GitHub Check: WeaveJS Lint and Compile
GitHub Check: Trace nox tests (3, 13, pandas-test)
GitHub Check: Trace nox tests (3, 13, scorers)
GitHub Check: Trace nox tests (3, 13, vertexai)
GitHub Check: Trace nox tests (3, 13, openai)
GitHub Check: Trace nox tests (3, 13, notdiamond)
GitHub Check: Trace nox tests (3, 13, mistral1)
GitHub Check: Trace nox tests (3, 13, mistral0)
GitHub Check: Trace nox tests (3, 13, llamaindex)
GitHub Check: Trace nox tests (3, 13, litellm)
GitHub Check: Trace nox tests (3, 13, langchain)
GitHub Check: Trace nox tests (3, 13, instructor)
GitHub Check: Trace nox tests (3, 13, google_ai_studio)
GitHub Check: Trace nox tests (3, 13, huggingface)
GitHub Check: Trace nox tests (3, 13, groq)
GitHub Check: Trace nox tests (3, 13, dspy)
GitHub Check: Trace nox tests (3, 13, cohere)
GitHub Check: Trace nox tests (3, 13, cerebras)
GitHub Check: Trace nox tests (3, 13, bedrock)
GitHub Check: Trace nox tests (3, 13, anthropic)
GitHub Check: Trace nox tests (3, 13, trace_server)
GitHub Check: Trace nox tests (3, 13, trace)
GitHub Check: Trace nox tests (3, 12, pandas-test)
GitHub Check: Trace nox tests (3, 12, scorers)
GitHub Check: Trace nox tests (3, 12, vertexai)
GitHub Check: Trace nox tests (3, 12, openai)
GitHub Check: Trace nox tests (3, 12, notdiamond)
GitHub Check: Trace nox tests (3, 12, mistral1)
GitHub Check: Trace nox tests (3, 12, mistral0)
GitHub Check: Trace nox tests (3, 12, llamaindex)
GitHub Check: Trace nox tests (3, 12, litellm)
GitHub Check: Trace nox tests (3, 12, langchain)
GitHub Check: Trace nox tests (3, 12, instructor)
GitHub Check: Trace nox tests (3, 12, google_ai_studio)
GitHub Check: Trace nox tests (3, 12, huggingface)
GitHub Check: Trace nox tests (3, 12, groq)
GitHub Check: Trace nox tests (3, 12, dspy)
GitHub Check: Trace nox tests (3, 12, cohere)
GitHub Check: Trace nox tests (3, 12, cerebras)
GitHub Check: Trace nox tests (3, 12, bedrock)
GitHub Check: Trace nox tests (3, 12, anthropic)
GitHub Check: Trace nox tests (3, 12, trace_server)
GitHub Check: Trace nox tests (3, 12, trace)
GitHub Check: Trace nox tests (3, 11, pandas-test)
GitHub Check: Trace nox tests (3, 11, scorers)
GitHub Check: Trace nox tests (3, 11, vertexai)
GitHub Check: Trace nox tests (3, 11, openai)
GitHub Check: Trace nox tests (3, 11, notdiamond)
GitHub Check: Trace nox tests (3, 11, mistral1)
GitHub Check: Trace nox tests (3, 11, mistral0)
GitHub Check: Trace nox tests (3, 11, llamaindex)
GitHub Check: Trace nox tests (3, 11, litellm)
GitHub Check: Trace nox tests (3, 11, langchain)
GitHub Check: Trace nox tests (3, 11, instructor)
GitHub Check: Trace nox tests (3, 11, google_ai_studio)
GitHub Check: Trace nox tests (3, 11, huggingface)
GitHub Check: Trace nox tests (3, 11, groq)
GitHub Check: Trace nox tests (3, 11, dspy)
GitHub Check: Trace nox tests (3, 11, cohere)
GitHub Check: Trace nox tests (3, 11, cerebras)
GitHub Check: Trace nox tests (3, 11, bedrock)
GitHub Check: Trace nox tests (3, 11, anthropic)
GitHub Check: Trace nox tests (3, 11, trace_server)
GitHub Check: Trace nox tests (3, 11, trace)
GitHub Check: Trace nox tests (3, 10, pandas-test)
GitHub Check: Trace nox tests (3, 10, scorers)
GitHub Check: Trace nox tests (3, 10, vertexai)
GitHub Check: Trace nox tests (3, 10, openai)
GitHub Check: Trace nox tests (3, 10, notdiamond)
GitHub Check: Trace nox tests (3, 10, mistral1)
GitHub Check: Trace nox tests (3, 10, mistral0)
GitHub Check: Trace nox tests (3, 10, llamaindex)
GitHub Check: Trace nox tests (3, 10, litellm)
GitHub Check: Trace nox tests (3, 10, langchain)
GitHub Check: Trace nox tests (3, 10, instructor)
GitHub Check: Trace nox tests (3, 10, google_ai_studio)
GitHub Check: Trace nox tests (3, 10, huggingface)
GitHub Check: Trace nox tests (3, 10, groq)
GitHub Check: Trace nox tests (3, 10, dspy)
GitHub Check: Trace nox tests (3, 10, cohere)
GitHub Check: Trace nox tests (3, 10, cerebras)
GitHub Check: Trace nox tests (3, 10, bedrock)
GitHub Check: Trace nox tests (3, 10, anthropic)
GitHub Check: Trace nox tests (3, 10, trace_server)
GitHub Check: Trace nox tests (3, 10, trace)
GitHub Check: Trace nox tests (3, 9, pandas-test)
GitHub Check: Trace nox tests (3, 9, scorers)
GitHub Check: Trace nox tests (3, 9, vertexai)
GitHub Check: Trace nox tests (3, 9, openai)
GitHub Check: Trace nox tests (3, 9, notdiamond)
GitHub Check: Trace nox tests (3, 9, mistral1)
GitHub Check: Trace nox tests (3, 9, mistral0)
GitHub Check: Trace nox tests (3, 9, llamaindex)
GitHub Check: Trace nox tests (3, 9, litellm)
GitHub Check: Trace nox tests (3, 9, langchain)
GitHub Check: Trace nox tests (3, 9, instructor)
GitHub Check: Legacy (Query Service) Python unit tests (1)
GitHub Check: Legacy (Query Service) Python unit tests (0)
GitHub Check: WeaveJS Lint and Compile
GitHub Check: Trace nox tests (3, 13, pandas-test)
GitHub Check: Trace nox tests (3, 13, scorers)
GitHub Check: Trace nox tests (3, 13, vertexai)
GitHub Check: Trace nox tests (3, 13, openai)
GitHub Check: Trace nox tests (3, 13, notdiamond)
GitHub Check: Trace nox tests (3, 13, mistral1)
GitHub Check: Trace nox tests (3, 13, mistral0)
GitHub Check: Trace nox tests (3, 13, llamaindex)
GitHub Check: Trace nox tests (3, 13, litellm)
GitHub Check: Trace nox tests (3, 13, langchain)
GitHub Check: Trace nox tests (3, 13, instructor)
GitHub Check: Trace nox tests (3, 13, google_ai_studio)
GitHub Check: Trace nox tests (3, 13, huggingface)
GitHub Check: Trace nox tests (3, 13, groq)
GitHub Check: Trace nox tests (3, 13, dspy)
GitHub Check: Trace nox tests (3, 13, cohere)
GitHub Check: Trace nox tests (3, 13, cerebras)
GitHub Check: Trace nox tests (3, 13, bedrock)
GitHub Check: Trace nox tests (3, 13, anthropic)
GitHub Check: Trace nox tests (3, 13, trace_server)
GitHub Check: Trace nox tests (3, 13, trace)
GitHub Check: Trace nox tests (3, 12, pandas-test)
GitHub Check: Trace nox tests (3, 12, scorers)
GitHub Check: Trace nox tests (3, 12, vertexai)
GitHub Check: Trace nox tests (3, 12, openai)
GitHub Check: Trace nox tests (3, 12, notdiamond)
GitHub Check: Trace nox tests (3, 12, mistral1)
GitHub Check: Trace nox tests (3, 12, mistral0)
GitHub Check: Trace nox tests (3, 12, llamaindex)
GitHub Check: Trace nox tests (3, 12, litellm)
GitHub Check: Trace nox tests (3, 12, langchain)
GitHub Check: Trace nox tests (3, 12, instructor)
GitHub Check: Trace nox tests (3, 12, google_ai_studio)
GitHub Check: Trace nox tests (3, 12, huggingface)
GitHub Check: Trace nox tests (3, 12, groq)
GitHub Check: Trace nox tests (3, 12, dspy)
GitHub Check: Trace nox tests (3, 12, cohere)
GitHub Check: Trace nox tests (3, 12, cerebras)
GitHub Check: Trace nox tests (3, 12, bedrock)
GitHub Check: Trace nox tests (3, 12, anthropic)
GitHub Check: Trace nox tests (3, 12, trace_server)
GitHub Check: Trace nox tests (3, 12, trace)
GitHub Check: Trace nox tests (3, 11, pandas-test)
GitHub Check: Trace nox tests (3, 11, scorers)
GitHub Check: Trace nox tests (3, 11, vertexai)
GitHub Check: Trace nox tests (3, 11, openai)
GitHub Check: Trace nox tests (3, 11, notdiamond)
GitHub Check: Trace nox tests (3, 11, mistral1)
GitHub Check: Trace nox tests (3, 11, mistral0)
GitHub Check: Trace nox tests (3, 11, llamaindex)
GitHub Check: Trace nox tests (3, 11, litellm)
GitHub Check: Trace nox tests (3, 11, langchain)
GitHub Check: Trace nox tests (3, 11, instructor)
GitHub Check: Trace nox tests (3, 11, google_ai_studio)
GitHub Check: Trace nox tests (3, 11, huggingface)
GitHub Check: Trace nox tests (3, 11, groq)
GitHub Check: Trace nox tests (3, 11, dspy)
GitHub Check: Trace nox tests (3, 11, cohere)
GitHub Check: Trace nox tests (3, 11, cerebras)
GitHub Check: Trace nox tests (3, 11, bedrock)
GitHub Check: Trace nox tests (3, 11, anthropic)
GitHub Check: Trace nox tests (3, 11, trace_server)
GitHub Check: Trace nox tests (3, 11, trace)
GitHub Check: Trace nox tests (3, 10, pandas-test)
GitHub Check: Trace nox tests (3, 10, scorers)
GitHub Check: Trace nox tests (3, 10, vertexai)
GitHub Check: Trace nox tests (3, 10, openai)
GitHub Check: Trace nox tests (3, 10, notdiamond)
GitHub Check: Trace nox tests (3, 10, mistral1)
GitHub Check: Trace nox tests (3, 10, mistral0)
GitHub Check: Trace nox tests (3, 10, llamaindex)
GitHub Check: Trace nox tests (3, 10, litellm)
GitHub Check: Trace nox tests (3, 10, langchain)
GitHub Check: Trace nox tests (3, 10, instructor)
GitHub Check: Trace nox tests (3, 10, google_ai_studio)
GitHub Check: Trace nox tests (3, 10, huggingface)
GitHub Check: Trace nox tests (3, 10, groq)
GitHub Check: Trace nox tests (3, 10, dspy)
GitHub Check: Trace nox tests (3, 10, cohere)
GitHub Check: Trace nox tests (3, 10, cerebras)
GitHub Check: Trace nox tests (3, 10, bedrock)
GitHub Check: Trace nox tests (3, 10, anthropic)
GitHub Check: Trace nox tests (3, 10, trace_server)
GitHub Check: Trace nox tests (3, 10, trace)
GitHub Check: Trace nox tests (3, 9, pandas-test)
GitHub Check: Trace nox tests (3, 9, scorers)
GitHub Check: Trace nox tests (3, 9, vertexai)
GitHub Check: Trace nox tests (3, 9, openai)
GitHub Check: Trace nox tests (3, 9, notdiamond)
GitHub Check: Trace nox tests (3, 9, mistral1)
GitHub Check: Trace nox tests (3, 9, mistral0)
GitHub Check: Trace nox tests (3, 9, llamaindex)
GitHub Check: Trace nox tests (3, 9, litellm)
GitHub Check: Trace nox tests (3, 9, langchain)
GitHub Check: Trace nox tests (3, 9, instructor)
GitHub Check: Legacy (Query Service) Python unit tests (1)
GitHub Check: Legacy (Query Service) Python unit tests (0)
GitHub Check: WeaveJS Lint and Compile
GitHub Check: Trace nox tests (3, 13, pandas-test)
GitHub Check: Trace nox tests (3, 13, scorers)
GitHub Check: Trace nox tests (3, 13, vertexai)
GitHub Check: Trace nox tests (3, 13, openai)
GitHub Check: Trace nox tests (3, 13, notdiamond)
GitHub Check: Trace nox tests (3, 13, mistral1)
GitHub Check: Trace nox tests (3, 13, mistral0)
GitHub Check: Trace nox tests (3, 13, llamaindex)
GitHub Check: Trace nox tests (3, 13, litellm)
GitHub Check: Trace nox tests (3, 13, langchain)
GitHub Check: Trace nox tests (3, 13, instructor)
GitHub Check: Trace nox tests (3, 13, google_ai_studio)
GitHub Check: Trace nox tests (3, 13, huggingface)
GitHub Check: Trace nox tests (3, 13, groq)
GitHub Check: Trace nox tests (3, 13, dspy)
GitHub Check: Trace nox tests (3, 13, cohere)
GitHub Check: Trace nox tests (3, 13, cerebras)
GitHub Check: Trace nox tests (3, 13, bedrock)
GitHub Check: Trace nox tests (3, 13, anthropic)
GitHub Check: Trace nox tests (3, 13, trace_server)
GitHub Check: Trace nox tests (3, 13, trace)
GitHub Check: Trace nox tests (3, 12, pandas-test)
GitHub Check: Trace nox tests (3, 12, scorers)
GitHub Check: Trace nox tests (3, 12, vertexai)
GitHub Check: Trace nox tests (3, 12, openai)
GitHub Check: Trace nox tests (3, 12, notdiamond)
GitHub Check: Trace nox tests (3, 12, mistral1)
GitHub Check: Trace nox tests (3, 12, mistral0)
GitHub Check: Trace nox tests (3, 12, llamaindex)
GitHub Check: Trace nox tests (3, 12, litellm)
GitHub Check: Trace nox tests (3, 12, langchain)
GitHub Check: Trace nox tests (3, 12, instructor)
GitHub Check: Trace nox tests (3, 12, google_ai_studio)
GitHub Check: Trace nox tests (3, 12, huggingface)
GitHub Check: Trace nox tests (3, 12, groq)
GitHub Check: Trace nox tests (3, 12, dspy)
GitHub Check: Trace nox tests (3, 12, cohere)
GitHub Check: Trace nox tests (3, 12, cerebras)
GitHub Check: Trace nox tests (3, 12, bedrock)
GitHub Check: Trace nox tests (3, 12, anthropic)
GitHub Check: Trace nox tests (3, 12, trace_server)
GitHub Check: Trace nox tests (3, 12, trace)
GitHub Check: Trace nox tests (3, 11, pandas-test)
GitHub Check: Trace nox tests (3, 11, scorers)
GitHub Check: Trace nox tests (3, 11, vertexai)
GitHub Check: Trace nox tests (3, 11, openai)
GitHub Check: Trace nox tests (3, 11, notdiamond)
GitHub Check: Trace nox tests (3, 11, mistral1)
GitHub Check: Trace nox tests (3, 11, mistral0)
GitHub Check: Trace nox tests (3, 11, llamaindex)
GitHub Check: Trace nox tests (3, 11, litellm)
GitHub Check: Trace nox tests (3, 11, langchain)
GitHub Check: Trace nox tests (3, 11, instructor)
GitHub Check: Trace nox tests (3, 11, google_ai_studio)
GitHub Check: Trace nox tests (3, 11, huggingface)
GitHub Check: Trace nox tests (3, 11, groq)
GitHub Check: Trace nox tests (3, 11, dspy)
GitHub Check: Trace nox tests (3, 11, cohere)
GitHub Check: Trace nox tests (3, 11, cerebras)
GitHub Check: Trace nox tests (3, 11, bedrock)
GitHub Check: Trace nox tests (3, 11, anthropic)
GitHub Check: Trace nox tests (3, 11, trace_server)
GitHub Check: Trace nox tests (3, 11, trace)
GitHub Check: Trace nox tests (3, 10, pandas-test)
GitHub Check: Trace nox tests (3, 10, scorers)
GitHub Check: Trace nox tests (3, 10, vertexai)
GitHub Check: Trace nox tests (3, 10, openai)
GitHub Check: Trace nox tests (3, 10, notdiamond)
GitHub Check: Trace nox tests (3, 10, mistral1)
GitHub Check: Trace nox tests (3, 10, mistral0)
GitHub Check: Trace nox tests (3, 10, llamaindex)
GitHub Check: Trace nox tests (3, 10, litellm)
GitHub Check: Trace nox tests (3, 10, langchain)
GitHub Check: Trace nox tests (3, 10, instructor)
GitHub Check: Trace nox tests (3, 10, google_ai_studio)
GitHub Check: Trace nox tests (3, 10, huggingface)
GitHub Check: Trace nox tests (3, 10, groq)
GitHub Check: Trace nox tests (3, 10, dspy)
GitHub Check: Trace nox tests (3, 10, cohere)
GitHub Check: Trace nox tests (3, 10, cerebras)
GitHub Check: Trace nox tests (3, 10, bedrock)
GitHub Check: Trace nox tests (3, 10, anthropic)
GitHub Check: Trace nox tests (3, 10, trace_server)
GitHub Check: Trace nox tests (3, 10, trace)
GitHub Check: Trace nox tests (3, 9, pandas-test)
GitHub Check: Trace nox tests (3, 9, scorers)
GitHub Check: Trace nox tests (3, 9, vertexai)
GitHub Check: Trace nox tests (3, 9, openai)
GitHub Check: Trace nox tests (3, 9, notdiamond)
GitHub Check: Trace nox tests (3, 9, mistral1)
GitHub Check: Trace nox tests (3, 9, mistral0)
GitHub Check: Trace nox tests (3, 9, llamaindex)
GitHub Check: Trace nox tests (3, 9, litellm)
GitHub Check: Trace nox tests (3, 9, langchain)
GitHub Check: Legacy (Query Service) Python unit tests (1)
GitHub Check: Legacy (Query Service) Python unit tests (0)
GitHub Check: WeaveJS Lint and Compile
GitHub Check: Trace nox tests (3, 13, pandas-test)
GitHub Check: Trace nox tests (3, 13, scorers)
GitHub Check: Trace nox tests (3, 13, vertexai)
GitHub Check: Trace nox tests (3, 13, openai)
GitHub Check: Trace nox tests (3, 13, notdiamond)
GitHub Check: Trace nox tests (3, 13, mistral1)
GitHub Check: Trace nox tests (3, 13, mistral0)
GitHub Check: Trace nox tests (3, 13, llamaindex)
GitHub Check: Trace nox tests (3, 13, litellm)
GitHub Check: Trace nox tests (3, 13, langchain)
GitHub Check: Trace nox tests (3, 13, instructor)
GitHub Check: Trace nox tests (3, 13, google_ai_studio)
GitHub Check: Trace nox tests (3, 13, huggingface)
GitHub Check: Trace nox tests (3, 13, groq)
GitHub Check: Trace nox tests (3, 13, dspy)
GitHub Check: Trace nox tests (3, 13, cohere)
GitHub Check: Trace nox tests (3, 13, cerebras)
GitHub Check: Trace nox tests (3, 13, bedrock)
GitHub Check: Trace nox tests (3, 13, anthropic)
GitHub Check: Trace nox tests (3, 13, trace_server)
GitHub Check: Trace nox tests (3, 13, trace)
GitHub Check: Trace nox tests (3, 12, pandas-test)
GitHub Check: Trace nox tests (3, 12, scorers)
GitHub Check: Trace nox tests (3, 12, vertexai)
GitHub Check: Trace nox tests (3, 12, openai)
GitHub Check: Trace nox tests (3, 12, notdiamond)
GitHub Check: Trace nox tests (3, 12, mistral1)
GitHub Check: Trace nox tests (3, 12, mistral0)
GitHub Check: Trace nox tests (3, 12, llamaindex)
GitHub Check: Trace nox tests (3, 12, litellm)
GitHub Check: Trace nox tests (3, 12, langchain)
GitHub Check: Trace nox tests (3, 12, instructor)
GitHub Check: Trace nox tests (3, 12, google_ai_studio)
GitHub Check: Trace nox tests (3, 12, huggingface)
GitHub Check: Trace nox tests (3, 12, groq)
GitHub Check: Trace nox tests (3, 12, dspy)
GitHub Check: Trace nox tests (3, 12, cohere)
GitHub Check: Trace nox tests (3, 12, cerebras)
GitHub Check: Trace nox tests (3, 12, bedrock)
GitHub Check: Trace nox tests (3, 12, anthropic)
GitHub Check: Trace nox tests (3, 12, trace_server)
GitHub Check: Trace nox tests (3, 12, trace)
GitHub Check: Trace nox tests (3, 11, pandas-test)
GitHub Check: Trace nox tests (3, 11, scorers)
GitHub Check: Trace nox tests (3, 11, vertexai)
GitHub Check: Trace nox tests (3, 11, openai)
GitHub Check: Trace nox tests (3, 11, notdiamond)
GitHub Check: Trace nox tests (3, 11, mistral1)
GitHub Check: Trace nox tests (3, 11, mistral0)
GitHub Check: Trace nox tests (3, 11, llamaindex)
GitHub Check: Trace nox tests (3, 11, litellm)
GitHub Check: Trace nox tests (3, 11, langchain)
GitHub Check: Trace nox tests (3, 11, instructor)
GitHub Check: Trace nox tests (3, 11, google_ai_studio)
GitHub Check: Trace nox tests (3, 11, huggingface)
GitHub Check: Trace nox tests (3, 11, groq)
GitHub Check: Trace nox tests (3, 11, dspy)
GitHub Check: Trace nox tests (3, 11, cohere)
GitHub Check: Trace nox tests (3, 11, cerebras)
GitHub Check: Trace nox tests (3, 11, bedrock)
GitHub Check: Trace nox tests (3, 11, anthropic)
GitHub Check: Trace nox tests (3, 11, trace_server)
GitHub Check: Trace nox tests (3, 11, trace)
GitHub Check: Trace nox tests (3, 10, pandas-test)
GitHub Check: Trace nox tests (3, 10, scorers)
GitHub Check: Trace nox tests (3, 10, vertexai)
GitHub Check: Trace nox tests (3, 10, openai)
GitHub Check: Trace nox tests (3, 10, notdiamond)
GitHub Check: Trace nox tests (3, 10, mistral1)
GitHub Check: Trace nox tests (3, 10, mistral0)
GitHub Check: Trace nox tests (3, 10, llamaindex)
GitHub Check: Trace nox tests (3, 10, litellm)
GitHub Check: Trace nox tests (3, 10, langchain)
GitHub Check: Trace nox tests (3, 10, instructor)
GitHub Check: Trace nox tests (3, 10, google_ai_studio)
GitHub Check: Trace nox tests (3, 10, huggingface)
GitHub Check: Trace nox tests (3, 10, groq)
GitHub Check: Trace nox tests (3, 10, dspy)
GitHub Check: Trace nox tests (3, 10, cohere)
GitHub Check: Trace nox tests (3, 10, cerebras)
GitHub Check: Trace nox tests (3, 10, bedrock)
GitHub Check: Trace nox tests (3, 10, anthropic)
GitHub Check: Trace nox tests (3, 10, trace_server)
GitHub Check: Trace nox tests (3, 10, trace)
GitHub Check: Trace nox tests (3, 9, pandas-test)
GitHub Check: Trace nox tests (3, 9, scorers)
GitHub Check: Trace nox tests (3, 9, vertexai)
GitHub Check: Trace nox tests (3, 9, openai)
GitHub Check: Trace nox tests (3, 9, notdiamond)
GitHub Check: Trace nox tests (3, 9, mistral1)
GitHub Check: Trace nox tests (3, 9, mistral0)
GitHub Check: Trace nox tests (3, 9, llamaindex)
GitHub Check: Trace nox tests (3, 9, litellm)
GitHub Check: Trace nox tests (3, 9, langchain)
GitHub Check: Legacy (Query Service) Python unit tests (1)
GitHub Check: Legacy (Query Service) Python unit tests (0)
GitHub Check: WeaveJS Lint and Compile
GitHub Check: Trace nox tests (3, 13, pandas-test)
GitHub Check: Trace nox tests (3, 13, scorers)
GitHub Check: Trace nox tests (3, 13, vertexai)
GitHub Check: Trace nox tests (3, 13, openai)
GitHub Check: Trace nox tests (3, 13, notdiamond)
GitHub Check: Trace nox tests (3, 13, mistral1)
GitHub Check: Trace nox tests (3, 13, mistral0)
GitHub Check: Trace nox tests (3, 13, llamaindex)
GitHub Check: Trace nox tests (3, 13, litellm)
GitHub Check: Trace nox tests (3, 13, langchain)
GitHub Check: Trace nox tests (3, 13, instructor)
GitHub Check: Trace nox tests (3, 13, huggingface)
GitHub Check: Trace nox tests (3, 13, groq)
GitHub Check: Trace nox tests (3, 13, dspy)
GitHub Check: Trace nox tests (3, 13, cohere)
GitHub Check: Trace nox tests (3, 13, cerebras)
GitHub Check: Trace nox tests (3, 13, bedrock)
GitHub Check: Trace nox tests (3, 13, anthropic)
GitHub Check: Trace nox tests (3, 13, trace_server)
GitHub Check: Trace nox tests (3, 13, trace)
GitHub Check: Trace nox tests (3, 12, pandas-test)
GitHub Check: Trace nox tests (3, 12, scorers)
GitHub Check: Trace nox tests (3, 12, vertexai)
GitHub Check: Trace nox tests (3, 12, openai)
GitHub Check: Trace nox tests (3, 12, notdiamond)
GitHub Check: Trace nox tests (3, 12, mistral1)
GitHub Check: Trace nox tests (3, 12, mistral0)
GitHub Check: Trace nox tests (3, 12, llamaindex)
GitHub Check: Trace nox tests (3, 12, litellm)
GitHub Check: Trace nox tests (3, 12, langchain)
GitHub Check: Trace nox tests (3, 12, instructor)
GitHub Check: Trace nox tests (3, 12, google_ai_studio)
GitHub Check: Trace nox tests (3, 12, huggingface)
GitHub Check: Trace nox tests (3, 12, groq)
GitHub Check: Trace nox tests (3, 12, dspy)
GitHub Check: Trace nox tests (3, 12, cohere)
GitHub Check: Trace nox tests (3, 12, cerebras)
GitHub Check: Trace nox tests (3, 12, bedrock)
GitHub Check: Trace nox tests (3, 12, anthropic)
GitHub Check: Trace nox tests (3, 12, trace_server)
GitHub Check: Trace nox tests (3, 12, trace)
GitHub Check: Trace nox tests (3, 11, pandas-test)
GitHub Check: Trace nox tests (3, 11, scorers)
GitHub Check: Trace nox tests (3, 11, vertexai)
GitHub Check: Trace nox tests (3, 11, openai)
GitHub Check: Trace nox tests (3, 11, notdiamond)
GitHub Check: Trace nox tests (3, 11, mistral1)
GitHub Check: Trace nox tests (3, 11, mistral0)
GitHub Check: Trace nox tests (3, 11, llamaindex)
GitHub Check: Trace nox tests (3, 11, litellm)
GitHub Check: Trace nox tests (3, 11, langchain)
GitHub Check: Trace nox tests (3, 11, instructor)
GitHub Check: Trace nox tests (3, 11, google_ai_studio)
GitHub Check: Trace nox tests (3, 11, huggingface)
GitHub Check: Trace nox tests (3, 11, groq)
GitHub Check: Trace nox tests (3, 11, dspy)
GitHub Check: Trace nox tests (3, 11, cohere)
GitHub Check: Trace nox tests (3, 11, cerebras)
GitHub Check: Trace nox tests (3, 11, bedrock)
GitHub Check: Trace nox tests (3, 11, anthropic)
GitHub Check: Trace nox tests (3, 11, trace_server)
GitHub Check: Trace nox tests (3, 11, trace)
GitHub Check: Trace nox tests (3, 10, pandas-test)
GitHub Check: Trace nox tests (3, 10, scorers)
GitHub Check: Trace nox tests (3, 10, vertexai)
GitHub Check: Trace nox tests (3, 10, openai)
GitHub Check: Trace nox tests (3, 10, notdiamond)
GitHub Check: Trace nox tests (3, 10, mistral1)
GitHub Check: Trace nox tests (3, 10, mistral0)
GitHub Check: Trace nox tests (3, 10, llamaindex)
GitHub Check: Trace nox tests (3, 10, litellm)
GitHub Check: Trace nox tests (3, 10, langchain)
GitHub Check: Trace nox tests (3, 10, instructor)
GitHub Check: Trace nox tests (3, 10, google_ai_studio)
GitHub Check: Trace nox tests (3, 10, huggingface)
GitHub Check: Trace nox tests (3, 10, groq)
GitHub Check: Trace nox tests (3, 10, dspy)
GitHub Check: Trace nox tests (3, 10, cohere)
GitHub Check: Trace nox tests (3, 10, cerebras)
GitHub Check: Trace nox tests (3, 10, bedrock)
GitHub Check: Trace nox tests (3, 10, anthropic)
GitHub Check: Trace nox tests (3, 10, trace_server)
GitHub Check: Trace nox tests (3, 10, trace)
GitHub Check: Trace nox tests (3, 9, pandas-test)
GitHub Check: Trace nox tests (3, 9, scorers)
GitHub Check: Trace nox tests (3, 9, vertexai)
GitHub Check: Trace nox tests (3, 9, openai)
GitHub Check: Trace nox tests (3, 9, notdiamond)
GitHub Check: Trace nox tests (3, 9, mistral1)
GitHub Check: Trace nox tests (3, 9, mistral0)
GitHub Check: Trace nox tests (3, 9, llamaindex)
GitHub Check: Trace nox tests (3, 9, litellm)
GitHub Check: Trace nox tests (3, 9, langchain)
GitHub Check: Legacy (Query Service) Python unit tests (1)
GitHub Check: Legacy (Query Service) Python unit tests (0)
GitHub Check: WeaveJS Lint and Compile
GitHub Check: Trace nox tests (3, 13, pandas-test)
GitHub Check: Trace nox tests (3, 13, vertexai)
GitHub Check: Trace nox tests (3, 13, openai)
GitHub Check: Trace nox tests (3, 13, notdiamond)
GitHub Check: Trace nox tests (3, 13, mistral1)
GitHub Check: Trace nox tests (3, 13, mistral0)
GitHub Check: Trace nox tests (3, 13, llamaindex)
GitHub Check: Trace nox tests (3, 13, litellm)
GitHub Check: Trace nox tests (3, 13, langchain)
GitHub Check: Trace nox tests (3, 13, instructor)
GitHub Check: Trace nox tests (3, 13, huggingface)
GitHub Check: Trace nox tests (3, 13, groq)
GitHub Check: Trace nox tests (3, 13, dspy)
GitHub Check: Trace nox tests (3, 13, cohere)
GitHub Check: Trace nox tests (3, 13, cerebras)
GitHub Check: Trace nox tests (3, 13, bedrock)
GitHub Check: Trace nox tests (3, 13, anthropic)
GitHub Check: Trace nox tests (3, 13, trace_server)
GitHub Check: Trace nox tests (3, 13, trace)
GitHub Check: Trace nox tests (3, 12, pandas-test)
GitHub Check: Trace nox tests (3, 12, scorers)
GitHub Check: Trace nox tests (3, 12, vertexai)
GitHub Check: Trace nox tests (3, 12, openai)
GitHub Check: Trace nox tests (3, 12, notdiamond)
GitHub Check: Trace nox tests (3, 12, mistral1)
GitHub Check: Trace nox tests (3, 12, mistral0)
GitHub Check: Trace nox tests (3, 12, llamaindex)
GitHub Check: Trace nox tests (3, 12, litellm)
GitHub Check: Trace nox tests (3, 12, langchain)
GitHub Check: Trace nox tests (3, 12, instructor)
GitHub Check: Trace nox tests (3, 12, google_ai_studio)
GitHub Check: Trace nox tests (3, 12, huggingface)
GitHub Check: Trace nox tests (3, 12, groq)
GitHub Check: Trace nox tests (3, 12, dspy)
GitHub Check: Trace nox tests (3, 12, cohere)
GitHub Check: Trace nox tests (3, 12, cerebras)
GitHub Check: Trace nox tests (3, 12, bedrock)
GitHub Check: Trace nox tests (3, 12, anthropic)
GitHub Check: Trace nox tests (3, 12, trace_server)
GitHub Check: Trace nox tests (3, 12, trace)
GitHub Check: Trace nox tests (3, 11, pandas-test)
GitHub Check: Trace nox tests (3, 11, scorers)
GitHub Check: Trace nox tests (3, 11, vertexai)
GitHub Check: Trace nox tests (3, 11, openai)
GitHub Check: Trace nox tests (3, 11, notdiamond)
GitHub Check: Trace nox tests (3, 11, mistral1)
GitHub Check: Trace nox tests (3, 11, mistral0)
GitHub Check: Trace nox tests (3, 11, llamaindex)
GitHub Check: Trace nox tests (3, 11, litellm)
GitHub Check: Trace nox tests (3, 11, langchain)
GitHub Check: Trace nox tests (3, 11, instructor)
GitHub Check: Trace nox tests (3, 11, google_ai_studio)
GitHub Check: Trace nox tests (3, 11, huggingface)
GitHub Check: Trace nox tests (3, 11, groq)
GitHub Check: Trace nox tests (3, 11, dspy)
GitHub Check: Trace nox tests (3, 11, cohere)
GitHub Check: Trace nox tests (3, 11, cerebras)
GitHub Check: Trace nox tests (3, 11, bedrock)
GitHub Check: Trace nox tests (3, 11, anthropic)
GitHub Check: Trace nox tests (3, 11, trace_server)
GitHub Check: Trace nox tests (3, 11, trace)
GitHub Check: Trace nox tests (3, 10, pandas-test)
GitHub Check: Trace nox tests (3, 10, scorers)
GitHub Check: Trace nox tests (3, 10, vertexai)
GitHub Check: Trace nox tests (3, 10, openai)
GitHub Check: Trace nox tests (3, 10, notdiamond)
GitHub Check: Trace nox tests (3, 10, mistral1)
GitHub Check: Trace nox tests (3, 10, mistral0)
GitHub Check: Trace nox tests (3, 10, llamaindex)
GitHub Check: Trace nox tests (3, 10, litellm)
GitHub Check: Trace nox tests (3, 10, langchain)
GitHub Check: Trace nox tests (3, 10, instructor)
GitHub Check: Trace nox tests (3, 10, google_ai_studio)
GitHub Check: Trace nox tests (3, 10, huggingface)
GitHub Check: Trace nox tests (3, 10, groq)
GitHub Check: Trace nox tests (3, 10, dspy)
GitHub Check: Trace nox tests (3, 10, cohere)
GitHub Check: Trace nox tests (3, 10, cerebras)
GitHub Check: Trace nox tests (3, 10, bedrock)
GitHub Check: Trace nox tests (3, 10, anthropic)
GitHub Check: Trace nox tests (3, 10, trace_server)
GitHub Check: Trace nox tests (3, 10, trace)
GitHub Check: Trace nox tests (3, 9, pandas-test)
GitHub Check: Trace nox tests (3, 9, scorers)
GitHub Check: Trace nox tests (3, 9, vertexai)
GitHub Check: Trace nox tests (3, 9, openai)
GitHub Check: Trace nox tests (3, 9, notdiamond)
GitHub Check: Trace nox tests (3, 9, mistral1)
GitHub Check: Trace nox tests (3, 9, mistral0)
GitHub Check: Trace nox tests (3, 9, llamaindex)
GitHub Check: Trace nox tests (3, 9, litellm)
GitHub Check: Trace nox tests (3, 9, langchain)
GitHub Check: Legacy (Query Service) Python unit tests (1)
GitHub Check: Legacy (Query Service) Python unit tests (0)
GitHub Check: WeaveJS Lint and Compile
GitHub Check: Trace nox tests (3, 13, pandas-test)
GitHub Check: Trace nox tests (3, 13, vertexai)
GitHub Check: Trace nox tests (3, 13, openai)
GitHub Check: Trace nox tests (3, 13, notdiamond)
GitHub Check: Trace nox tests (3, 13, mistral1)
GitHub Check: Trace nox tests (3, 13, mistral0)
GitHub Check: Trace nox tests (3, 13, llamaindex)
GitHub Check: Trace nox tests (3, 13, litellm)
GitHub Check: Trace nox tests (3, 13, langchain)
GitHub Check: Trace nox tests (3, 13, instructor)
GitHub Check: Trace nox tests (3, 13, huggingface)
GitHub Check: Trace nox tests (3, 13, groq)
GitHub Check: Trace nox tests (3, 13, dspy)
GitHub Check: Trace nox tests (3, 13, cohere)
GitHub Check: Trace nox tests (3, 13, cerebras)
GitHub Check: Trace nox tests (3, 13, bedrock)
GitHub Check: Trace nox tests (3, 13, anthropic)
GitHub Check: Trace nox tests (3, 13, trace_server)
GitHub Check: Trace nox tests (3, 13, trace)
GitHub Check: Trace nox tests (3, 12, pandas-test)
GitHub Check: Trace nox tests (3, 12, scorers)
GitHub Check: Trace nox tests (3, 12, vertexai)
GitHub Check: Trace nox tests (3, 12, openai)
GitHub Check: Trace nox tests (3, 12, notdiamond)
GitHub Check: Trace nox tests (3, 12, mistral1)
GitHub Check: Trace nox tests (3, 12, mistral0)
GitHub Check: Trace nox tests (3, 12, llamaindex)
GitHub Check: Trace nox tests (3, 12, litellm)
GitHub Check: Trace nox tests (3, 12, langchain)
GitHub Check: Trace nox tests (3, 12, instructor)
GitHub Check: Trace nox tests (3, 12, google_ai_studio)
GitHub Check: Trace nox tests (3, 12, huggingface)
GitHub Check: Trace nox tests (3, 12, groq)
GitHub Check: Trace nox tests (3, 12, dspy)
GitHub Check: Trace nox tests (3, 12, cohere)
GitHub Check: Trace nox tests (3, 12, cerebras)
GitHub Check: Trace nox tests (3, 12, bedrock)
GitHub Check: Trace nox tests (3, 12, anthropic)
GitHub Check: Trace nox tests (3, 12, trace_server)
GitHub Check: Trace nox tests (3, 12, trace)
GitHub Check: Trace nox tests (3, 11, pandas-test)
GitHub Check: Trace nox tests (3, 11, scorers)
GitHub Check: Trace nox tests (3, 11, vertexai)
GitHub Check: Trace nox tests (3, 11, openai)
GitHub Check: Trace nox tests (3, 11, notdiamond)
GitHub Check: Trace nox tests (3, 11, mistral1)
GitHub Check: Trace nox tests (3, 11, mistral0)
GitHub Check: Trace nox tests (3, 11, llamaindex)
GitHub Check: Trace nox tests (3, 11, litellm)
GitHub Check: Trace nox tests (3, 11, langchain)
GitHub Check: Trace nox tests (3, 11, instructor)
GitHub Check: Trace nox tests (3, 11, google_ai_studio)
GitHub Check: Trace nox tests (3, 11, huggingface)
GitHub Check: Trace nox tests (3, 11, groq)
GitHub Check: Trace nox tests (3, 11, dspy)
GitHub Check: Trace nox tests (3, 11, cohere)
GitHub Check: Trace nox tests (3, 11, cerebras)
GitHub Check: Trace nox tests (3, 11, bedrock)
GitHub Check: Trace nox tests (3, 11, anthropic)
GitHub Check: Trace nox tests (3, 11, trace_server)
GitHub Check: Trace nox tests (3, 11, trace)
GitHub Check: Trace nox tests (3, 10, pandas-test)
GitHub Check: Trace nox tests (3, 10, scorers)
GitHub Check: Trace nox tests (3, 10, vertexai)
GitHub Check: Trace nox tests (3, 10, openai)
GitHub Check: Trace nox tests (3, 10, notdiamond)
GitHub Check: Trace nox tests (3, 10, mistral1)
GitHub Check: Trace nox tests (3, 10, mistral0)
GitHub Check: Trace nox tests (3, 10, llamaindex)
GitHub Check: Trace nox tests (3, 10, litellm)
GitHub Check: Trace nox tests (3, 10, langchain)
GitHub Check: Trace nox tests (3, 10, instructor)
GitHub Check: Trace nox tests (3, 10, google_ai_studio)
GitHub Check: Trace nox tests (3, 10, huggingface)
GitHub Check: Trace nox tests (3, 10, groq)
GitHub Check: Trace nox tests (3, 10, dspy)
GitHub Check: Trace nox tests (3, 10, cohere)
GitHub Check: Trace nox tests (3, 10, cerebras)
GitHub Check: Trace nox tests (3, 10, bedrock)
GitHub Check: Trace nox tests (3, 10, anthropic)
GitHub Check: Trace nox tests (3, 10, trace_server)
GitHub Check: Trace nox tests (3, 10, trace)
GitHub Check: Trace nox tests (3, 9, pandas-test)
GitHub Check: Trace nox tests (3, 9, scorers)
GitHub Check: Trace nox tests (3, 9, vertexai)
GitHub Check: Trace nox tests (3, 9, openai)
GitHub Check: Trace nox tests (3, 9, notdiamond)
GitHub Check: Trace nox tests (3, 9, mistral1)
GitHub Check: Trace nox tests (3, 9, mistral0)
GitHub Check: Trace nox tests (3, 9, llamaindex)
GitHub Check: Trace nox tests (3, 9, litellm)
GitHub Check: Trace nox tests (3, 9, langchain)

🔇 Additional comments (6)

weave/trace_server/clickhouse_trace_server_batched.py (6)

89-92: Appropriate import changes to support custom provider models

The imports now include get_custom_provider_info from the llm_completion module, which aligns with the PR objective of supporting custom provider models.

1616-1625: Good initialization of variables for better code structure

Initializing dedicated variables for API key, provider, and custom model fields at the beginning of the method improves readability and makes the code flow more understandable.

1626-1630: Clear documentation of model handling logic

The comments clearly explain the two different paths for handling standard models vs custom models, which makes the code more maintainable.

1631-1661: Standard model handling looks correct

The standard model path appropriately fetches API keys and provider information from the model map, with proper error handling when keys or fetchers are missing.

1691-1702: Good structured API call with keyword arguments

The API call to lite_llm_completion now uses keyword arguments, which improves readability and makes parameter passing more explicit.

1726-1726: Consistent usage of model_name variable

Using the model_name variable consistently instead of directly referencing req.inputs.model ensures consistency in the code, particularly important when the model name might be modified for custom providers.

tssweeney · 2025-03-20T19:52:09Z

weave/trace_server/interface/builtin_object_classes/provider.py

+    base_url: str
+    api_key_name: str
+    extra_headers: dict[str, str] = {}
+    return_type: ReturnType = ReturnType.OPENAI


does this change between completions, response, and chat? What does "OpenAI" return type mean?

I was thinking change between opeanai, anthropic, and other types of provider apis

I guess I was wondering how it would change your code - but it looks like this is not used in the current implementation.

tssweeney · 2025-03-20T19:54:11Z

weave/trace_server/interface/builtin_object_classes/llm_model.py

+class LLMModel(base_object_def.BaseObject):
+    provider_model: base_object_def.RefStr
+    prompt: Optional[base_object_def.RefStr] = None
+    default_params: ModelParams = ModelParams()


use Field for default factory

tssweeney · 2025-03-20T19:57:01Z

weave/trace_server/interface/builtin_object_classes/llm_model.py

+    stop: Optional[Union[str, list[str]]] = None
+
+
+class ProviderModel(base_object_def.BaseObject):


can you help me understand the purpose of a "providerModel"? Why not have these properties inside of LLMModel? Not saying to change, just understanding

Theres no specific reason as to why not have these properties inside of LLMModel, maybe jsut separation of concerns

ProviderModel: Represents the base configuration of an LLM on top of a provider

LLMModel: Represents a specific usage configuration of a provider model

This separation provides several advantages:

Reusability: The same ProviderModel can be referenced by multiple LLMModel instances with different prompts and parameters

Separation of concerns:
ProviderModel handles "which model am I using?"
LLMModel handles "how am I using this model?"

Flexibility: You could swap out provider models while keeping the same prompt and parameters

Configuration management: Provider-specific details are isolated from usage-specific details

the current implementation kind of prioritizes flexibility between the two objects

Oh i see - so maybe this class belongs in provider.py? Since it is more about the configuration of the provider as opposed to config of the user's model

Ah that does make more sense, thats why for the initial implementation we only use providerModel since its necessary for the configuration of the provider and not LLMModel since thats a separate milestone with prompts and the like 👍

tssweeney · 2025-03-20T19:57:39Z

weave/trace_server/interface/builtin_object_classes/llm_model.py

+from weave.trace_server.interface.builtin_object_classes import base_object_def
+
+
+class ModelMode(str, Enum):


For here and everywhere else, I would recommend being a bit more verbose in naming: LLMProviderModelMode - this will avoid name conflicts in the future

tssweeney · 2025-03-20T19:59:25Z

weave/trace_server/interface/builtin_object_classes/llm_model.py

+
+class ProviderModel(base_object_def.BaseObject):
+    provider: base_object_def.RefStr
+    max_tokens: int


was max tokens supposed to be here and in ModelParams?

max_tokens here means, max tokens allowed by the model (ie max max_tokens value in params)
in model params its the max tokens to use

tssweeney · 2025-03-20T20:02:05Z

weave/trace_server/llm_completion.py

@@ -36,6 +39,31 @@ def lite_llm_completion(
    # This allows us to drop params that are not supported by the LLM provider
    litellm.drop_params = True

+    # Handle custom provider
+    if provider == "custom" and base_url:


error when provider = "custom" and no base url? What about base url and provider != custom?

tssweeney

Logically, this looks pretty solid. My main concern is that I am not seeing any tests! (:

tssweeney · 2025-03-21T21:25:42Z

weave/trace_server/llm_completion.py

+    project_id: str,
+    model_name: str,
+    obj_read_func: Callable,
+) -> tuple[str, str, dict[str, str], str, str]:


let's formally type this return payload (: tuple[str, str, dict[str, str], str, str] is nasty

tssweeney · 2025-03-21T21:28:51Z

weave/trace_server/llm_completion.py

+                f"Object {provider_id} is not a Provider, it is a {provider_obj.base_object_class}"
+            )
+
+        # Extract provider information


you could convert the payload into the Provider class (and same for the model below) which would reduce the use of raw dictionary access

tssweeney · 2025-03-21T21:32:00Z

weave/trace_server/interface/builtin_object_classes/provider.py

+    return_type: ProviderReturnType = Field(default=ProviderReturnType.OPENAI)
+
+
+class ProviderModelMode(str, Enum):


this distinction does not seem to be respected anywhere in code - what is the use?

None currently, but if we ever want to add completion models it will be useful, I can remove and add when useful

tssweeney · 2025-03-21T21:37:28Z

weave/trace_server/interface/builtin_object_classes/llm_model.py

+    stop: Optional[Union[str, list[str]]] = None
+
+
+class LLMModel(base_object_def.BaseObject):


I really agree with what you are doing here. We def need LLModel. However, I don't think it is strictly needed for this PR (model provider setup). Unless you need it here, i think it might be good to pull this one out of this PR. There is nearly an identical need in the on demand evals AND overlap with scorers. I think we might want to take a bit of time to make sure there is alignment there

Specifically, I think we will want to implement this as a weave.Object actually (and have server-side support for that - a little tech debt). That way it comes with it's own pre-packaged "predict" function and can run inside of evaluations.

that makes sense to me, no reason to keep this in until its actually implemented

custom provider model built in object backend

9d75a0d

jwlee64 requested review from a team as code owners March 4, 2025 20:30

coderabbitai bot reviewed Mar 4, 2025

View reviewed changes

jwlee64 and others added 2 commits March 6, 2025 08:32

Merge branch 'master' into josiah/custom-provider-backend

967d309

update for ollama

01a21f6

coderabbitai bot reviewed Mar 6, 2025

View reviewed changes

Merge branch 'master' into josiah/custom-provider-backend

7618271

tssweeney reviewed Mar 20, 2025

View reviewed changes

jwlee64 force-pushed the josiah/custom-provider-backend branch from 99cf88b to 32bb2da Compare March 21, 2025 16:51

pr comments

8734ad8

jwlee64 force-pushed the josiah/custom-provider-backend branch from 32bb2da to 8734ad8 Compare March 21, 2025 17:16

fix tests

7d44837

jwlee64 requested a review from tssweeney March 21, 2025 17:58

tssweeney reviewed Mar 21, 2025

View reviewed changes

pr comments

2d6840a

jwlee64 requested a review from tssweeney March 24, 2025 21:15

tssweeney approved these changes Mar 24, 2025

View reviewed changes

jwlee64 merged commit ff984de into master Mar 24, 2025
129 of 130 checks passed

jwlee64 deleted the josiah/custom-provider-backend branch March 24, 2025 21:37

github-actions bot locked and limited conversation to collaborators Mar 24, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(weave): custom provider model built in object backend #3841

chore(weave): custom provider model built in object backend #3841

jwlee64 commented Mar 4, 2025 •

edited

Loading

circle-job-mirror bot commented Mar 4, 2025 •

edited

Loading

coderabbitai bot commented Mar 4, 2025 •

edited

Loading

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

Documentation and Community

coderabbitai bot left a comment

coderabbitai bot left a comment

tssweeney Mar 20, 2025

jwlee64 Mar 20, 2025

tssweeney Mar 20, 2025

tssweeney Mar 20, 2025

tssweeney Mar 20, 2025

jwlee64 Mar 20, 2025

tssweeney Mar 20, 2025

jwlee64 Mar 20, 2025

tssweeney Mar 20, 2025

tssweeney Mar 20, 2025

jwlee64 Mar 20, 2025

tssweeney Mar 20, 2025

tssweeney left a comment

tssweeney Mar 21, 2025

tssweeney Mar 21, 2025

tssweeney Mar 21, 2025

jwlee64 Mar 21, 2025

tssweeney Mar 21, 2025

tssweeney Mar 21, 2025

jwlee64 Mar 21, 2025

		stop: Optional[Union[str, list[str]]] = None


		class ProviderModel(base_object_def.BaseObject):

		from weave.trace_server.interface.builtin_object_classes import base_object_def


		class ModelMode(str, Enum):

		return_type: ProviderReturnType = Field(default=ProviderReturnType.OPENAI)


		class ProviderModelMode(str, Enum):

		stop: Optional[Union[str, list[str]]] = None


		class LLMModel(base_object_def.BaseObject):

chore(weave): custom provider model built in object backend #3841

chore(weave): custom provider model built in object backend #3841

Conversation

jwlee64 commented Mar 4, 2025 • edited Loading

Description

Testing

Summary by CodeRabbit

circle-job-mirror bot commented Mar 4, 2025 • edited Loading

coderabbitai bot commented Mar 4, 2025 • edited Loading

Walkthrough

Changes

Sequence Diagram(s)

Possibly related PRs

Suggested reviewers

Poem

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

Documentation and Community

coderabbitai bot left a comment

Choose a reason for hiding this comment

coderabbitai bot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tssweeney left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jwlee64 commented Mar 4, 2025 •

edited

Loading

circle-job-mirror bot commented Mar 4, 2025 •

edited

Loading

coderabbitai bot commented Mar 4, 2025 •

edited

Loading