Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use *multipart.Reader to stream file parts instead of storing temporary FileHeaders in memory #7

Open
ciehanski opened this issue Feb 4, 2019 · 0 comments
Assignees
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@ciehanski
Copy link
Owner

A possible implementation is below:

func uploadMR(mr *multipart.Reader, writer io.Writer) error {
	for {
		part, err := mr.NextPart()
		if err == io.EOF {
			break
		}
		if part.FileName() == "" {
			continue
		}
		var count int
		chunk, err := Allocate(256)
		if err != nil {
			return err
		}
		for {
			count, err = part.Read(chunk)
			if err == io.EOF {
				break
			}
			if _, err := writer.Write(chunk[:count]); err != nil {
				return err
			}
		}
	}
	return nil
}

However, I may not need to read chunks of the part considering the part will already be a chunk as its being received by the HTML form. Will need to create benchmarks, this could drastically improve upload performance which is the largest bottleneck currently due to the HTML form contents being stored in memory, sanitized and then re-written back to memory. Would like to see the use of *multipart.Reader before any major release.

@ciehanski ciehanski self-assigned this Feb 4, 2019
@ciehanski ciehanski added enhancement New feature or request help wanted Extra attention is needed labels Feb 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

1 participant