Memory leak while reading from sparse arrays

[danielgafni]

Bug

A snippet to reproduce (you can monitor the consumed memory in any convenient way, like htop):

import tiledb
import numpy as np

users = np.arange(1000000)
items = np.arange(1000000)
labels = np.random.randint(0, 2, 1000000)

dom = tiledb.Domain(
    tiledb.Dim(name="user", domain=(0, users.max()), tile=4, dtype=np.int32),
    tiledb.Dim(name="item", domain=(0, items.max()), tile=4, dtype=np.int32),
)

schema = tiledb.ArraySchema(domain=dom, sparse=True, attrs=(tiledb.Attr(name="label", dtype=np.int8),))

tiledb.SparseArray.create("array.tiledb", schema)

with tiledb.open("array.tiledb", mode="w") as arr:
    arr[users, items] = labels
    
arr = tiledb.open("array.tiledb")

while True:
    arr[0]

Info

OS
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"

Python 3.7.12
tiledb 0.11.5
numpy 1.21.4

[ihnorton]

Hi @danielgafni, thanks for trying TileDB and for opening the issue. We do have several tests for memory usage, although this may be too slow to show up in that case. We'll take a closer look and get back to you soon. Thanks!

[danielgafni]

Hey guys! Any progress with this?

[ihnorton]

Hi @danielgafni, I've put in two fixes which will be in the next libtiledb/tiledb-py release:

• [pending 2.6.1] Fix leak of C++ Array object in PyQuery #882
  • tiledb::Array destructor no longer calls ::close for non-owned C ptr TileDB#2823
• (also Free array_schema in array preload routine #881 which was not per-query)

Thanks again for the report!

[danielgafni]

Thank you @ihnorton !

Can't wait to try the fixed version