Feature: support operate the underlying incoming data for uploading large file

[uddmorningsun]

For file uploading, default behaviour will ReadForm and write data to new temporary file if oversize maxMemory.

If temporary directory size is too small, it will fail for uploading large file and even consume OS memory since copy to buffer.

Inspired by Flask, Flask request.stream.read will read underlying socket data directly if payload found. Refer to: https://werkzeug.palletsprojects.com/en/2.0.x/wsgi/#werkzeug.wsgi.LimitedStream

This method makes assumptions because the request payload exists for operating POST/PATCH/PUT.

A simple demo for uploading 5G large file in low memory based on docker:

[root@control-master tmp]# docker run -ti --rm --memory 1G --tmpfs /tmp:rw,size=1G,mode=1777 -v $(pwd)/go:/go golang:1.16.5 /bin/bash

root@b672a98e5314:/go/testmodules# cat main.go
package main

import (
	"fmt"
	"github.com/gin-gonic/gin"
	"io"
	"os"
	"path/filepath"
)

func main() {
	engine := gin.Default()
	engine.POST("/streamupload", func(c *gin.Context) {
		if c.GetHeader("Content-Type") != "application/octet-stream" {
			err := fmt.Errorf("required octet-stream")
			c.AbortWithStatusJSON(400, map[string]string{"message": err.Error()})
			return
		}
		info, err := os.Create(filepath.Join("/home", "foo"))
		if err != nil {
			c.AbortWithStatusJSON(400, gin.H{"message": "Create: "+err.Error()})
			return
		}
		defer info.Close()
		_, err = io.Copy(info, c.Request.Body)
		if err != nil {
			c.AbortWithStatusJSON(400, gin.H{"message": "Copy: "+err.Error()})
			return
		}
		c.JSON(200, map[string]string{"message": "ok stream"})
	})

	if err := engine.Run("0.0.0.0:19090"); err != nil {
		panic(err)
	}
}

root@b53810b3e294:/go/testmodules# GOTMPDIR=/opt/ go run main.go

[root@control-master ~]# dd if=/dev/zero of=5G bs=1M count=5170 status=progress
[root@control-master ~]# curl -vvv -H "Content-Type:application/octet-stream" -T 5G -X POST 172.17.0.2:19090/streamupload

Signed-off-by: Chenyang Yan [email protected]

P.S.: if it's useful, I'll do the rest of the work, e.g.: update README.md, etc.