-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Add basic support for UploadedFile UserContent #2611
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
raise UserError('UploadedFile.file must be a genai.types.File object') | ||
# genai.types.File is its own ContentUnionDict and not a | ||
# PartDict, so append to the contents directly. | ||
contents.append(item.file) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As shown in the google file upload example:
result = client.models.generate_content(
model="gemini-2.0-flash",
contents=[
myfile,
"\n\n",
"Can you tell me about the instruments in this photo?",
],
)
The file is actually the whole content, instead of being a "message part". That's why I'm appending to the contents array directly here. Reference: https://ai.google.dev/api/files
d6f2bb3
to
4fd4a88
Compare
This wraps an opaque reference to a provider-specific representation of an uploaded file.
2917ac8
to
8a837b5
Compare
8a837b5
to
8140f52
Compare
8140f52
to
0d6e486
Compare
@DouweM CI still failing on code coverage. I will fix it, but first I'd love some feedback on the API . LMK if you agree with the choices or if I should make some adjustments! |
@DouweM here's some initial support for #2574
The
UploadedFile
user content is simply wrapping an opaque reference to some return value of provider-specific file upload API, which is then validated in the corresponding model_map_user_prompt
.I've only added support for Google and OpenAI, but I believe the API should be flexible enough to add support for other providers. I started working on Anthropic, but decided to leave it out for now as the official SDK doesn't support this feature yet, and I was having trouble referencing it using the SDK data objects.
I've opted to not implement a
Provider.upload_file
abstraction, as the options can be different across providers and I would need to get more familiar with pydantic-ai before feeling confident enough to design a proper abstraction (Can follow up with another PR later!)One caveat with the tests: The VCR framework apparently doesn't support requests containing binary content, so I had to turn off for uploading files. This is how I proceeded to add the tests:
Since this is just a recording and we are only verifying that we can reference an uploaded file, it probably doesn't matter much that we are not actually running the upload request for now. This can be changed later when VCR is fixed to support this type of request.
Close #2574