Why does the google cloud storage backend use a temporary file

[aucampia]

It is not quite clear why a temporary file is needed, could it not just use the google cloud storage API directly?

I'm trying to process a rather large file as a stream but this behavior is not very helpful in this, as I now need to have enough temporary file storage to accommodate the file.

[funkyshu]

Wow, I totally missed this question. The short answer is a not a good one. Expediency. Basically it was easier for a us to rely on the standard os lib to keep track for positional information. This is really not a great reason to be honest. But part of the thinking was to match os's filehandle behavior as closely as possible. So, in OS, one might be able to write 1k bytes, then seek back to position 500 and overwrite the original bytes. With S3 and GCS, writing is atomic(at least, it doesn't exist until you close) action so OS give us some flexibility but for huge tradeoff in resources and time. It's a tradeoff that I'm not sure is warranted. since seeking mid-write seems some contrived and useless.

[funkyshu]

It also seems we only open a temp file when reading from and or seeking. Otherwise we seem to writing to a buffer for no good reason. So not only might you need extra disk space, but you may need a lot of memory. I'm just not sure when I might have time to address those issues but I'm open to patches.

[funkyshu]

This was addressed in #172.