Project Limit Request: SQLAlchemy - 20 GB

[zzzeek]

Project URL

https://pypi.org/project/SQLAlchemy/

Does this project already exist?

• Yes

New limit

20G

Update issue title

• I have updated the title.

Which indexes

PyPI

About the project

this is SQLAlchemy's wheel builds and it looks like for 2.0.43 we hit the limit:

25hUploading sqlalchemy-2.0.43-cp38-cp38-macosx_10_9_x86_64.whl
25l
  0% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/2.2 MB • --:-- • ?
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB • 00:00 • 77.8 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB • 00:00 • 77.8 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB • 00:00 • 77.8 MB/s
25hWARNING  Error during upload. Retry with the --verbose option for more details. 
ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/        
         Project size too large. Limit for project 'SQLAlchemy' total size is 10
         GB. See                                                                
         https://docs.pypi.org/project-management/storage-limits#requesting-a-pr
         oject-size-limit-increase                                              

We use cython so people like our wheels a lot. It looks like it took us 20 years to hit 10G, however, for the majority of years we didn't provide wheels or any kind of binaries. Based on our current release size of about 132M and we release around once a month, to keep us going for five more years assuming no growth in the release size that would add up to an additional 8G, but we should assume release size will keep growing - looks like it has grown from 95M to 130M over the 2.0.x series likely due to new architectures being added. We do remove EOL python versions from wheels but we obviously add new python versions on the other end so that washes out.

How large is each release?

Noting that release 2.0.43 failed to upload completely, here's 2.0.42 summed up by a robot I found wandering outside:

SQLAlchemy 2.0.42 Release - Detailed Size Summary

Overall Statistics:

• Total files: 47 files
• Total size: 138,305,305 bytes (131.96 MB)

File Types Breakdown:

• 1 Source Distribution:
  • sqlalchemy-2.0.42.tar.gz - 9,749,972 bytes (9.30 MB)
• 46 Wheel Distributions (.whl files):
  • Universal wheel: 1 file - sqlalchemy-2.0.42-py3-none-any.whl (1.83 MB)
  • Platform-specific wheels: 45 files covering:

Platform Coverage:

• macOS: x86_64 and ARM64 architectures
• Linux: x86_64 and aarch64 architectures (both glibc and musl)
• Windows: 32-bit and 64-bit

Python Version Support:

• Python 3.7 through Python 3.13
• Each platform typically has wheels for all supported Python versions

Typical Wheel Sizes:

• macOS wheels: ~2.1-2.13 MB each
• Linux wheels: ~3.1-3.2 MB each
• Windows wheels: ~2.1-2.12 MB each
• musllinux wheels: ~3.1-3.17 MB each

Size Distribution:

• Source distribution: 7.0% of total
• Universal wheel: 1.4% of total
• Platform-specific wheels: 91.6% of total

This release demonstrates SQLAlchemy's comprehensive platform support, with pre-compiled wheels available for all major operating systems and Python versions, making installation faster for end users.

How frequently do you make a release?

typically every 4 weeks, sometimes more sometimes less

Code of Conduct

• I agree to follow the PSF Code of Conduct

[zzzeek]

it looks like we have 3.7 and 3.8 in our wheel builds for 2.0.x since 2.0.x first came out in 2023. We should be trimming those soon and 2.1 will start support at 3.9 or even 3.10

[CaselIT]

As an usability improvement, it would have been great to have received warning emails once 90-95% of the size limit were reached.
Warning email are sent already on upload for other cases, such as if the name of the binary wheel is the old format, so a warning for this is definitely expected.

The maintainers of sqlalchemy had no idea there was a limit and that we were close to it. We discovered it because the wheel uploads started to fail, fortunately the latest release seem to miss only wheels for 3.8 that's no longer widely used.

[miketheman]

As an usability improvement, it would have been great to have received warning emails once 90-95% of the size limit were reached. Warning email are sent already on upload for other cases, such as if the name of the binary wheel is the old format, so a warning for this is definitely expected.

The maintainers of sqlalchemy had no idea there was a limit and that we were close to it. We discovered it because the wheel uploads started to fail, fortunately the latest release seem to miss only wheels for 3.8 that's no longer widely used.

This is captured here: pypi/warehouse#17650 If you have the time and desire, send a PR!

[miketheman]

Thanks for the in-depth details of space usage and analysis.

Limit raised to 20GB.

[CaselIT]

Thanks!

Does the single file limit apply only to the binary files, or it's the same also for the source tarball?

If it's the same, then it could be useful to raise that also, since the source distribution is currently 9.3mb, so that's close to the default limit of 10mb

[miketheman]

The current default limits are 100.0 MB for individual files and 10.0 GB for the entire project.

Refs https://docs.pypi.org/project-management/storage-limits/

[CaselIT]

Not sure why I had 10mb in mind. Sorry for the noise