Ability to use a TempFile instead of an in-memory buffer for tarball snapshots to build large Docker containers?

[nelsonjchen]

kaniko/pkg/snapshot/snapshot.go

Lines 57 to 70 in 13accba

	// TakeSnapshot takes a snapshot of the specified files, avoiding directories in the whitelist, and creates
	// a tarball of the changed files. Return contents of the tarball, and whether or not any files were changed
	func (s *Snapshotter) TakeSnapshot(files []string) ([]byte, error) {
	buf := bytes.NewBuffer([]byte{})
	filesAdded, err := s.snapshotFiles(buf, files)
	if err != nil {
	return nil, err
	}
	contents := buf.Bytes()
	if !filesAdded {
	return nil, nil
	}
	return contents, err
	}

I'm interested in building multi-GB Docker images with Kaniko. Unfortunately, doing this immediately exhausts the RAM. I think this is because the snapshotting steps take a tarball snapshot of the whole filesystem in-memory.

It would be nice to have an option to tell kaniko to use a Tempfile for the snapshot backing.

[priyawadhwa]

@nelsonjchen this should be fixed with #428. I'll go ahead and close this issue, please comment or open another if you run into any more problems!

[nelsonjchen]

Woah! I didn't expect this to be resolved. "Have a bit of faith." - Some videogame I'm currently playing.

I look forward to trying this.

[lovetheguitar]

Hey,

sorry for the stupid question, but how do I enable this?
I'd like to build a 10+GB image from within Gitlab CI.
Thanks for any hints.

[nelsonjchen]

@lovetheguitar I would look at the code. Always look at the code. It's authoritative. Check out the PR that was merged and mentioned above. Apparently, you don't have to do anything except provide disk space.

[lovetheguitar]

@lovetheguitar I would look at the code. Always look at the code. It's authoritative. Check out the PR that was merged and mentioned above. Apparently, you don't have to do anything except provide disk space.

Thanks for your answer. Yeah, I tried, but I don't know go at all.
Either way it appears to be a different bug #909.