Can't send large folders due to memory based tmpfs

[raingloom]

Checklist

• I have searched the issue tracker for open issues that relate to the same problem, before opening a new one.
• This issue only relates to a single bug. I will open new issues for any other problems.

Description

Here is the error:

2023/07/27 14:49:43 Fyne error:  Error on sending directory
2023/07/27 14:49:43   Cause: write /tmp/wormhole-william-dir1332995391: no space left on device
2023/07/27 14:49:43   At: github.com/Jacalz/rymdport/v3/internal/transport/bridge/send.go:190

I suspect that this is because similar to the Python version you are constructing the zip file in /tmp instead of doing the proper thing and streaming it. Otherwise you are doubling disk usage for sending a file, which is pretty bad when the file or directory is several gigabytes in size.

It's likely running out of memory because Flatpak mounts /tmp as a tmpfs instead of backing it by disk.

Steps to Reproduce

1. Find a large folder or create one, it should be bigger than the amount of free RAM or storage you have.
2. Try sending it.

Screenshots

No response

Rymdport version

3.4.0 (Flatpak)

Operating System

Linux

Operating System Version

guix 182be30

Go Compiler Version

No response

Additional Information

No response

[Jacalz]

I appreciate the report but I don't think there is anything that I can do about this at this point in time.

The zip-creation happens inside the workhole-william dependency and is not a bug in this project. However, I also believe that it is an inherent weakness of the wormhole protocol that you need to know the size of the file (the zip-file in the case of directory transfers) before you send it.

I 100% agree with you that it would be a lot better to stream the zip-file but we need to know the size before starting the transfer (I think) and I doubt that it will be possible to use streaming in that case. However, I will look into this to see if there is anything that can be done about it.

FYI: It is also worth noting that the Python version of wormhole is the reference implementation of the protocol. If it doesn't work there then it probably wouldn't work in the Go implementation either.

[Jacalz]

It definitely is problematic with the Flatpak. Will have to try and see if there is any way to work around the issue

[raingloom]

Thanks, in the meantime I found that this error was coming from core. I started working on converting the Python reference implementation to simply use a streaming zip library. Especially with my use case of sending a bunch of JPEGs there is zero point in compressing the folder, since each file is already compressed, so I can just create the stream twice. Once to find the length and the second time to actually send it. An alternative I'm considering is to just simply give an upper bound based on the combined size of the files, but I'll have to check the specs to make sure that sending a smaller file is allowed.

[Jacalz]

That sounds like a good idea. I had some similar thoughts about how this might be solved. It sounds like we are on the same page

[Jacalz]

Sorry for the long delay. I believe that this should be fixed with flathub/io.github.jacalz.rymdport#25. I'll close the issue for now but feel free to reopen if it doesn't work (note that it will take a few hours for the new build to land in the stable channel of Flathub).

[qlyoung]

I am still encountering this issue on 3.6.0. I am not using the flatpak, I am using the AUR, which builds from source: https://aur.archlinux.org/packages/rymdport

The "fix" causes the flatpak to use the host /tmp. At least on my system that is still a memory backed filesystem so nothing changes.

Attempting a 20gb transfer on my system I observe rymdport write 20gb to /tmp. If I remount /tmp with a 60gb filesystem then that works (I have 32gb of memory so the 20gb happens to fit).

Aside from that, during the transfer rymdport's own memory usage steadily grows:

I guess that's a separate issue, though. Filed at #165

[Jacalz]

Thinks for bringing this to my attention. I'll reopen this issue and look into it again. The intermediary zip file write is unfortunately part of how https://github.com/psanford/wormhole-william handles transfers so I might have to come up with a better solution there.

[Jacalz]

This is due to how the directory sends in the wormhole package are creating a temporary zip file when sending directories, see https://github.com/rymdport/wormhole/blob/4e36e05aff6c3cce139db9be3d973181e2b5a753/wormhole/send.go#L508. This file is then used when sending the transfer. This might be a bit tricky to fix in a good way given that the wormhole protocol requires the size to be known before initiating the send. I'm honestly not entirely sure how we can fix this without breaking that protocol :(

[qlyoung]

Would it be viable to allow configuring where the zip is created, so that it could be set to use a path not located on a ramdisk?

[Jacalz]

Would it be viable to allow configuring where the zip is created, so that it could be set to use a path not located on a ramdisk?

Maybe. It solves the problem kind of but it also introduces other things do consider:

• Likely will be problematic for Flatpak apps when running inside a sandbox.
• Users would have to manually change this and know that changing temporary directly allows them to send larger files.

The best option would be to have some kind of streaming protocol for doing the transfers. That would mean finding something else than magic wormhole (limits compatibility with other clients).