Unable to create a .tar archive bigger than 256MB

[fungiboletus]

Hei,

I have a folder with 300 000 files and Hubic recommends to have 50 000 files maximum. So I want to create a .tar archive directly in the mount point (the files and the archive don't fit on my system).

It doesn't seem to work 😢

Context

• svfs version : 0.9.1
• storage provider : OVH
• product : Hubic

Steps to reproduce this issue :

1. Have a folder bigger than 256MB
2. tar --verbose --create --file /svfs_mount/archive.tar /folder
3. Go to the beach and come back when tar has finished. 🏖

Results you expected :

The .tar archive is created and contains all the files.

Results you observed :

Tar exits with return code 2 :

tar: /svfs_mount/archive.tar: Cannot write: Input/output error
tar: Error is not recoverable: exiting now

The archive.tar file exists and is 256MB (268431360).

[Confushion]

I've got similar issues (on OVH/hubic).

It seems that the maximum segment size (default mount option segment_size=256) doesn't work as expected. Any file larger than the maximum segment size is not properly split into multiple segments.
If you mount with option segment_size=10 you will observe the error a lot earlier at an upload of 10Mb.

This currently limits the maximum file size to be about 5Gb (the maximum size of a segment), when mounted with option segment_size=5120.

As I have a lot of files > 5Gb that I'd like to store on Hubic, this is a blocker for me...

[pisto]

Getting the same issue, also with the suggested rsync method. Basically all files greater that the segmententation limit are broken. Endpoint is hubic.

[xlucas]

Can you provide a debug trace ? I suspect this is a duplicate of #100.

[pisto]

How do I generate it?

[xlucas]

Use the debug mount option documented here.

[pisto]

DEBU[2017-09-22T13:58:29+02:00] <- Lookup [ID=0xa2 Node=0x2 Uid=1000 Gid=1000 Pid=22793] "rick.mkv"  source=fuse
DEBU[2017-09-22T13:58:29+02:00] -> [ID=0xa2] Lookup error=ENOENT     source=fuse
DEBU[2017-09-22T13:58:29+02:00] <- Create [ID=0xa3 Node=0x2 Uid=1000 Gid=1000 Pid=22793] "rick.mkv" fl=OpenWriteOnly+OpenCreate mode=-rw------- umask=--------w-  source=fuse
DEBU[2017-09-22T13:58:30+02:00] -> [ID=0xa3] Create {0x3 gen=0 valid=1m0s attr={valid=1m0s ino=12768038556851320042 size=0 mode=-rwx------}} {0x1 fl=OpenDirectIO+OpenNonSeekable}  source=fuse
DEBU[2017-09-22T13:58:30+02:00] <- Write [ID=0xa4 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131056 @0 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:58:30+02:00] -> [ID=0xa4] Write 131056            source=fuse
DEBU[2017-09-22T13:58:30+02:00] <- Write [ID=0xa5 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131072 @131056 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:58:30+02:00] -> [ID=0xa5] Write 131072            source=fuse
DEBU[2017-09-22T13:58:30+02:00] <- Write [ID=0xa6 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 16 @262128 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:58:30+02:00] -> [ID=0xa6] Write 16                source=fuse
DEBU[2017-09-22T13:58:30+02:00] <- Write [ID=0xa7 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131056 @262144 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:58:30+02:00] -> [ID=0xa7] Write 131056            source=fuse
[...]
DEBU[2017-09-22T13:59:20+02:00] <- Write [ID=0xc9c Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131072 @267780080 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:59:20+02:00] -> [ID=0xc9c] Write 131072           source=fuse
DEBU[2017-09-22T13:59:20+02:00] <- Write [ID=0xc9d Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 16 @267911152 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:59:20+02:00] -> [ID=0xc9d] Write 16               source=fuse
DEBU[2017-09-22T13:59:20+02:00] <- Write [ID=0xc9e Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131056 @267911168 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:59:20+02:00] -> [ID=0xc9e] Write 131056           source=fuse
DEBU[2017-09-22T13:59:20+02:00] <- Write [ID=0xc9f Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131072 @268042224 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:59:21+02:00] -> [ID=0xc9f] Write 131072           source=fuse
DEBU[2017-09-22T13:59:21+02:00] <- Write [ID=0xca0 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 16 @268173296 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:59:21+02:00] -> [ID=0xca0] Write 16               source=fuse
DEBU[2017-09-22T13:59:21+02:00] <- Write [ID=0xca1 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131056 @268173312 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:59:21+02:00] -> [ID=0xca1] Write 131056           source=fuse
DEBU[2017-09-22T13:59:21+02:00] <- Write [ID=0xca2 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131072 @268304368 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:59:21+02:00] -> [ID=0xca2] Write 131072           source=fuse
DEBU[2017-09-22T13:59:21+02:00] <- Write [ID=0xca3 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 16 @268435440 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:59:21+02:00] -> [ID=0xca3] Write 16               source=fuse
DEBU[2017-09-22T13:59:21+02:00] <- Write [ID=0xca4 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131056 @268435456 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:59:39+02:00] -> [ID=0xca4] Write error=EIO: Timeout when reading or writing data  source=fuse
DEBU[2017-09-22T13:59:39+02:00] <- Write [ID=0xca5 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131056 @268435456 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:59:54+02:00] -> [ID=0xca5] Write error=EIO: Timeout when reading or writing data  source=fuse
DEBU[2017-09-22T13:59:54+02:00] <- Flush [ID=0xca6 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 fl=0x0 lk=0x5fc476094e3c05ce  source=fuse
DEBU[2017-09-22T13:59:54+02:00] -> [ID=0xca6] Flush                  source=fuse
DEBU[2017-09-22T13:59:54+02:00] <- Release [ID=0xca7 Node=0x3 Uid=0 Gid=0 Pid=0] 0x1 fl=OpenWriteOnly rfl=0 owner=0x0  source=fuse
DEBU[2017-09-22T13:59:54+02:00] -> [ID=0xca7] Release                source=fuse

[xlucas]

Try setting connect_timeout=5m like suggested in #100

[fungiboletus]

Hei. I will try that, but doesn't that mean the underlying system has a slowness issue ?