Package bytearray provides a fixed size chunk []byte array with slab allocation. ByteArrays utilize next-chunk-linking which provides full reader/writer compatibility with any size you need. It uses an automatic chunk allocation with manual deallocation, which means that all ByteArrays must be manually released when not used anymore. The slab and chunk sizes are globally configurable and must be done before any chunks are allocated.
import "github.com/fredli74/bytearray"
// Setup bytearray sizes to 2048 bytes (2KiB) chunks, 4MiB memory slabs and limit to 4096 slabs (16GiB memory)
// These are global values and can only be setup once before anything is allocated
bytearray.Setup(2048, 2048 * 2048, 4096)
// Enable automatic garbage collect to run every 60 seconds and remove anything slab that has had no allocations for at least 120 seconds
bytearray.EnableAutoGC(60, 120)
// Open a file
f, err := os.Open("input.txt")
// Declare a byte array variable
var ba ByteArray
// Copy file content to the byte array
io.Copy(ba, f)
// Print size of array
fmt.Printf("Size is %v", ba.Len())
// Print debug memory stats
fmt.Printf("Stats: %v", bytearray.Stats())
// Release the array (important, this is never done automatically, if a variable falls out of scope before it has been Released, the data will still remain in the slab)
ba.Release()const (
SEEK_SET int = os.SEEK_SET // seek relative to the origin of the array
SEEK_CUR int = os.SEEK_CUR // seek relative to the current offset
SEEK_END int = os.SEEK_END // seek relative to the end of the array
)var ChunkSize int = 2048ChunkSize defaults to 2048 bytes (2KiB) chunks. Use Setup function to change.
var MaxSlabs int = 4097 // +1 because slab 0 is never used (or allocated), it is reserved for emptyLocation pointersMaxSlabs defaults to 4096 slabs (16GiB memory with default SlabSize and ChunkSize). Use Setup function to change.
var SlabSize int = ChunkSize * 2048SlabSize defaults to 2048 slabs (4MiB memory with default ChunkSize). Use Setup function to change.
func ChunkQuantize(size int64) int64ChunkQuantize takes a size and round it up to an even chunk size (this can be used to calculate used memory)
func DisableAutoGC()DisableAutoGC disables the automatic GC goroutine
func EnableAutoGC(releaseInterval int, maxAge int)EnableAutoGC enables the automatic GC goroutine releaseInterval is how often the GC is run. maxAge is the number of seconds since the slab was touched before it can be freed
func GC(maxAge int)Not really a GC, more of a slab releaser in case it has not been used for a while. maxAge is the number of seconds since the slab was touched before it can be freed
func Setup(chunkSize int, slabSize int, maxSlabs int)Setup the ByteArray global sizes. IMPORTANT, Setup can only be called before any chunks have been allocated or it will throw a panic.
func Stats() (AllocatedSlabs int64, GrabbedChunks int64, ReleasedChunks int64, MemoryAllocated int64, MemoryInUse int64)Stats returns the current allocation statistics
type ByteArray struct {
}Byte array read and write is not concurrency safe however the underlying slab structures are so you can use multiple ByteArrays at the same time
func (b ByteArray) Len() intLen returns the current length of the ByteArray
func (b *ByteArray) Read(p []byte) (n int, err error)Read from the byte array into a buffer and advance the current read position
func (b *ByteArray) ReadFrom(r io.Reader) (n int64, err error)ReadFrom reads from the Reader until EOF and fills up the ByteArray (at the current write position)
func (b ByteArray) ReadPosition() intReadPosition returns the current write position
func (b *ByteArray) ReadSeek(offset int, whence int) (absolute int, err error)ReadSeek will check bounds and return EOF error if seeking outside
func (b *ByteArray) ReadSlice() ([]byte, error)ReadSlice returns a byte slice chunk for the current read position (it does not advance read position)
func (b *ByteArray) Release()Release will release all chunks associated with the ByteArray
func (b *ByteArray) Split(offset int) (newArray ByteArray)Split a byte array into a new ByteArray at the specified offset, the old byte array will be truncated at the split offset
func (b *ByteArray) Truncate(offset int) intTruncate sets the length, it also expands the length in case offset > usedBytes
func (b *ByteArray) Write(p []byte) (n int, err error)Write to the byte array from a buffer and advance the current write position
func (b ByteArray) WritePosition() intWritePosition returns the current write position
func (b *ByteArray) WriteSeek(offset int, whence int) intWriteSeek will allocate and expand bounds if needed
func (b *ByteArray) WriteSlice() []byteWriteSlice returns a byte slice chunk for the current write position (it does not advance write position)
func (b *ByteArray) WriteTo(w io.Writer) (n int64, err error)WriteTo writes all ByteArray data (from the current read position) to a writer interface