client.evals.create throws NameError

[jreeter]

Confirm this is an issue with the Python library and not an underlying OpenAI API

• This is an issue with the Python library

Describe the bug

When creating the eval below the following error is surfaced: NameError: name 'Input' is not defined

Code

eval_cfg = client.evals.create(
  name="visio_json_exact_match",
  data_source_config={
    "type": "custom",
    "item_schema": {
      "type": "object",
      "properties": {
        "prompt": {"type": "string"},
        "expected": {"type": "string"}     # canonical JSON string
      },
      "required": ["prompt","expected"]
    },
    "include_sample_schema": True
  },
  testing_criteria=[
    {
      "type": "string_check",
      "name": "Exact JSON match",
      "operation": "eq",
      # Compare the model's (string) output to your canonical expected string
      "input": "{{sample.output_text}}",
      "reference": "{{item.expected}}"
    }
  ]
)

Stacktrace

Traceback (most recent call last):
  File "api_test.py", line 7, in <module>
    result = client.evals.create(name='Hello World Eval', data_source_config={"type": "custom",
  File "/Users/joshreeter/PycharmProjects/ZendeskAgentClassification/venv/lib/python3.8/site-packages/openai/resources/evals/evals.py", line 113, in create
    body=maybe_transform(
  File "/Users/joshreeter/PycharmProjects/ZendeskAgentClassification/venv/lib/python3.8/site-packages/openai/_utils/_transform.py", line 86, in maybe_transform
    return transform(data, expected_type)
  File "/Users/joshreeter/PycharmProjects/ZendeskAgentClassification/venv/lib/python3.8/site-packages/openai/_utils/_transform.py", line 109, in transform
    transformed = _transform_recursive(data, annotation=cast(type, expected_type))
  File "/Users/joshreeter/PycharmProjects/ZendeskAgentClassification/venv/lib/python3.8/site-packages/openai/_utils/_transform.py", line 176, in _transform_recursive
    return _transform_typeddict(data, stripped_type)
  File "/Users/joshreeter/PycharmProjects/ZendeskAgentClassification/venv/lib/python3.8/site-packages/openai/_utils/_transform.py", line 274, in _transform_typeddict
    result[_maybe_transform_key(key, type_)] = _transform_recursive(value, annotation=type_)
  File "/Users/joshreeter/PycharmProjects/ZendeskAgentClassification/venv/lib/python3.8/site-packages/openai/_utils/_transform.py", line 203, in _transform_recursive
    return [_transform_recursive(d, annotation=annotation, inner_type=inner_type) for d in data]
  File "/Users/joshreeter/PycharmProjects/ZendeskAgentClassification/venv/lib/python3.8/site-packages/openai/_utils/_transform.py", line 203, in <listcomp>
    return [_transform_recursive(d, annotation=annotation, inner_type=inner_type) for d in data]
  File "/Users/joshreeter/PycharmProjects/ZendeskAgentClassification/venv/lib/python3.8/site-packages/openai/_utils/_transform.py", line 211, in _transform_recursive
    data = _transform_recursive(data, annotation=annotation, inner_type=subtype)
  File "/Users/joshreeter/PycharmProjects/ZendeskAgentClassification/venv/lib/python3.8/site-packages/openai/_utils/_transform.py", line 176, in _transform_recursive
    return _transform_typeddict(data, stripped_type)
  File "/Users/joshreeter/PycharmProjects/ZendeskAgentClassification/venv/lib/python3.8/site-packages/openai/_utils/_transform.py", line 262, in _transform_typeddict
    annotations = get_type_hints(expected_type, include_extras=True)
  File "/Users/joshreeter/PycharmProjects/ZendeskAgentClassification/venv/lib/python3.8/site-packages/openai/_utils/_transform.py", line 447, in get_type_hints
    return _get_type_hints(obj, globalns=globalns, localns=localns, include_extras=include_extras)
  File "/Users/joshreeter/PycharmProjects/ZendeskAgentClassification/venv/lib/python3.8/site-packages/typing_extensions.py", line 1319, in get_type_hints
    hint = typing.get_type_hints(obj, globalns=globalns, localns=localns)
  File "/Users/joshreeter/.pyenv/versions/3.8.13/lib/python3.8/typing.py", line 1232, in get_type_hints
    value = _eval_type(value, base_globals, localns)
  File "/Users/joshreeter/.pyenv/versions/3.8.13/lib/python3.8/typing.py", line 270, in _eval_type
    return t._evaluate(globalns, localns)
  File "/Users/joshreeter/.pyenv/versions/3.8.13/lib/python3.8/typing.py", line 518, in _evaluate
    eval(self.__forward_code__, globalns, localns),

To Reproduce

Create an eval like so:

eval_cfg = client.evals.create(
  name="visio_json_exact_match",
  data_source_config={
    "type": "custom",
    "item_schema": {
      "type": "object",
      "properties": {
        "prompt": {"type": "string"},
        "expected": {"type": "string"}     # canonical JSON string
      },
      "required": ["prompt","expected"]
    },
    "include_sample_schema": True
  },
  testing_criteria=[
    {
      "type": "string_check",
      "name": "Exact JSON match",
      "operation": "eq",
      # Compare the model's (string) output to your canonical expected string
      "input": "{{sample.output_text}}",
      "reference": "{{item.expected}}"
    }
  ]
)

Code snippets

OS

macOS

Python version

3.8.13

Library version

1.100.2

[RobertCraigie]

Thanks for the report, it looks like this only happens on python 3.8, I would recommend upgrading as 3.8 is EOL.

[LuminaX-alt]

This error (NameError: name 'Input' is not defined) is not caused by the API itself, but by a missing forward reference inside the Python client library’s type hints for evals.

The library’s internal schema refers to Input, but the symbol is never defined in that module. As a result, typing.get_type_hints throws a NameError when client.evals.create() is called.

✅ Fix: Explicitly provide a safe fallback for Input before calling client.evals.create. I chose typing.Any because it is semantically correct (any type is acceptable at runtime) and prevents breaking downstream logic.
from typing import Any
from openai import OpenAI
import openai.resources.evals.evals as evals_mod

--- Fix missing forward reference ---

The library internally expects "Input" to exist for type hint resolution.

Define it as Any to prevent NameError during eval creation.

evals_mod.Input = Any

--- Normal usage ---

client = OpenAI()

eval_cfg = client.evals.create(
name="visio_json_exact_match",
data_source_config={
"type": "custom",
"item_schema": {
"type": "object",
"properties": {
"prompt": {"type": "string"},
"expected": {"type": "string"} # canonical JSON string
},
"required": ["prompt", "expected"]
},
"include_sample_schema": True
},
testing_criteria=[
{
"type": "string_check",
"name": "Exact JSON match",
"operation": "eq",
# Compare model output to expected JSON string
"input": "{{sample.output_text}}",
"reference": "{{item.expected}}"
}
]
)

print("Eval created successfully:", eval_cfg)
Adds the missing Input symbol only in the evals module namespace.
Does not alter library logic.
Keeps compatibility with future library updates

[RobertCraigie]

@LuminaX-alt please stop spamming AI generated replies, especially when they're not necessary. If you continue to do so, you will be banned.

[jreeter]

@RobertCraigie thanks for the heads up.

[RobertCraigie]

@jreeter are you able to upgrade to a newer python version? I looked briefly into fixing this and it'll be quite difficult, so I'm going to close this as wontfix due to it only being reproducible on 3.8