Possible memory leak beginning with 4.1.5

[azukov]

The client seems to have memory leak when running with Python3.9 on Linux.
The problem started with 4.1.5 and still present in 4.1.12, version 4.1.4 is OK.
Everything is installed with pip install -r requirements.txt from PyPi.

Tested with mprof run client.py while python server.py is running as a separate process.

Tested configurations

Config	p4p	pvxslibs
39_4	4.1.4	1.0.1
39_5	4.1.5	1.1.4
39_12	4.1.12	1.3.1

requirements.txt with X being specific version

numpy<2.0.0
p4p==4.1.X
memory_profiler
matplotlib

server.py

import time
from p4p.nt import NTNDArray
from p4p.server import Server
from p4p.server.thread import SharedPV
import numpy as np

SIZE = 1_000_000
NAME = 'test:image'

pv = SharedPV(nt=NTNDArray(), initial=np.random.rand(SIZE))

print(f'serving pv: {NAME}')
with Server(providers=[{NAME: pv, }]):
    while True:
        pv.post(np.random.rand(SIZE), timestamp=time.time())
        time.sleep(1)

client.py

from p4p.client.thread import Context
from time import sleep
NAME = 'test:image'


def cb(value):
    print(f"update at: {value}")


with Context('pva') as ctxt:
    print(f"Subscribe to {NAME}")
    S = ctxt.monitor(NAME, cb)
    try:
        sleep(100)
    except KeyboardInterrupt:
        pass

    print("Close subscription")
    S.close()
print("Done")
print(ctxt.get(NAME))

The results of memory usage are:

[jeinstei]

I can replicate this in a conda environment using different p4p versions through pip with numpy 1.26.42. With 4.1.4, I see initial high cpu usage that then drops, with low memory usage. 4.1.12 sees similar CPU performance, with memory slowly increasing similar to the above diagram.

The increasing line is python 3.9, p4p==4.1.12; the flat line is p4p==4.1.4

[mdavidsaver]

Can memory_profiler provide any more granular information than when is (I guess?) resident memory size?

Also, could you try my p4p.disect utility?

https://github.com/mdavidsaver/p4p/blob/70b030de2f30cf690bb317429f59bd092e65420e/src/p4p/disect.py#L19-L23

from p4p.disect import periodic

periodic(1.0, sys.stderr)

Each second periodic() will print a list of python type, and the difference in the number of instances from the previous second. As the GC runs, some up and down is expected. A PyObject reference leaks would show up as persistently increasing instance count. If not, then this would point towards the C++ code.

eg.

# Types 677 -> 678
New Types
  <class 'prompt_toolkit.auto_suggest.Suggestion'> 1
Known Types
  <class 'prompt_toolkit.layout.screen.WritePosition'> 11 delta 4
  <class 'prompt_toolkit.document._DocumentCache'> 15 delta -1
  <class 'prompt_toolkit.lexers.pygments.PygmentsLexer'> 10 delta 1
  <class 'pygments.lexers.python.PythonLexer'> 2 delta 1
  <class 'weakref.ReferenceType'> 3191 delta 3
  <class 'prompt_toolkit.document._ImmutableLineList'> 15 delta -1
  <class 'tuple'> 17053 delta 148
  <class 'dict'> 8984 delta 20
  <class 'prompt_toolkit.filters.base.Condition'> 112 delta 1
  <class 'prompt_toolkit.layout.screen.Screen'> 9 delta 4
  <class 'cell'> 4031 delta 84
  <class 'function'> 16461 delta 31
  <class 'prompt_toolkit.data_structures.Point'> 13 delta 4
  <class 'prompt_toolkit.filters.base._AndList'> 227 delta 1
  <class 'collections.defaultdict'> 51 delta 16
  <class 'prompt_toolkit.lexers.pygments.RegexSync'> 10 delta 1
  <class 'list'> 19166 delta -1
  <class 'weakref.KeyedRef'> 16 delta -1

[jeinstei]

$ mprof run --include-children client.py > disectlog 2>&1

Here is the output from a run that uses the 4.1.12 configuration. I can add PVXS debug logging, but I'm not sure if it would add much.

All Types
  <class 'numpy.ma.core.MaskedConstant'> 1
  <class 'dict'> 5111
  <class 'tuple'> 5240
  <class 'function'> 10698
  <class 'cell'> 1189
  <class 'type'> 1239
  <class 'weakref'> 2047
  <class 'getset_descriptor'> 1493
  <class 'numpy.ma.core._frommethod'> 26
  <class 'numpy.ma.core._extrema_operation'> 2
  <class 'numpy.ma.core._convert2ma'> 13
  <class '_frozen_importlib_external.SourceFileLoader'> 438
  <class '_frozen_importlib.ModuleSpec'> 513
  <class 'module'> 516
  <class 'list'> 871
  <class 'numpy.ma.extras._fromnxfunction_allargs'> 3
  <class 'numpy.ma.extras._fromnxfunction_seq'> 5
  <class 'numpy.ma.extras._fromnxfunction_single'> 2
  <class 'staticmethod'> 274
  <class 'classmethod'> 270
  <class 'numpy.ma.extras.mr_class'> 1
  <class 'numpy._pytesttester.PytestTester'> 9
  <class 'set'> 264
  <class 'numpy.core.getlimits.MachArLike'> 6
  <class 'builtin_function_or_method'> 1410
  <class 'cython_function_or_method'> 3
  <class '_weakrefset.WeakSet'> 6
  <class 'property'> 455
  <class 'p4p.version.Version'> 1
  <class 'logging.Logger'> 10
  <class 'p4p.disect._StatsThread'> 1
  <class 'threading.Thread'> 1
  <class 'threading.Event'> 2
  <class 'threading.Condition'> 2
  <class 'collections.deque'> 2
  <class 'method'> 106
  <class 'frozenset'> 192
  <class '_frozen_importlib_external.FileFinder'> 59
  <class 'functools.partial'> 38
  <class '_frozen_importlib_external.ExtensionFileLoader'> 44
  <class 'abc.ABCMeta'> 124
  <class '_abc._abc_data'> 140
  <class 'slice'> 4
  <class 'wrapper_descriptor'> 2222
  <class 'method_descriptor'> 1403
  <class 'member_descriptor'> 533
  <class '_collections._tuplegetter'> 341
  <class 'random.SystemRandom'> 1
  <class 'numpy.random.mtrand.RandomState'> 1
  <class 'numpy.random._mt19937.MT19937'> 1
  <class 'numpy.random.bit_generator.SeedSequence'> 1
  <class 'numpy.random._generator.Enum'> 5
  <class '_ctypes.PyCSimpleType'> 28
  <class 'StgDict'> 39
  <class '_cython_3_0_8.cython_function_or_method'> 2
  <class 'numpy.errstate'> 2
  <class 'numpy.ma.core._MaskedUnaryOperation'> 26
  <class 'numpy.ma.core._DomainGreaterEqual'> 2
  <class 'numpy.ma.core._DomainGreater'> 3
  <class 'numpy.ma.core._DomainTan'> 1
  <class 'numpy.ma.core._DomainCheckInterval'> 3
  <class 'numpy.ma.core._MaskedBinaryOperation'> 18
  <class 'numpy.ma.core._DomainedBinaryOperation'> 6
  <class 'numpy.ma.core._DomainSafeDivide'> 6
  <class 'numpy.ma.core._MaskedPrintOption'> 1
  <class 'numpy.ma.core.MaskedArray'> 1
  <class 'classmethod_descriptor'> 72
  <class 'types.SimpleNamespace'> 1
  <class 'codecs.CodecInfo'> 1
  <class '_io.TextIOWrapper'> 4
  <class 'encodings.utf_8.IncrementalDecoder'> 2
  <class 'encodings.utf_8.IncrementalEncoder'> 2
  <class '_io.BufferedReader'> 2
  <class '_io.FileIO'> 4
  <class '_io.BufferedWriter'> 2
  <class 'os._Environ'> 2
  <class '_distutils_hack.DistutilsMetaFinder'> 1
  <class '_sitebuiltins.Quitter'> 2
  <class '_sitebuiltins._Printer'> 3
  <class '_sitebuiltins._Helper'> 1
  <class 'reprlib.Repr'> 1
  <class 'enum.EnumMeta'> 43
  <class 'types.DynamicClassAttribute'> 2
  <class 'sre_constants._NamedIntConstant'> 74
  <enum 'RegexFlag'> 14
  <class 'functools._lru_cache_wrapper'> 40
  <class 'typing._SpecialForm'> 7
  <class 'typing._ProtocolMeta'> 16
  <class 'typing.TypeVar'> 27
  <class 'typing._SpecialGenericAlias'> 38
  <class 'typing._CallableType'> 1
  <class 'typing._TupleType'> 1
  <class 'typing.NamedTupleMeta'> 1
  <class 'typing._TypedDictMeta'> 4
  <class 'typing._GenericAlias'> 175
  <class 'typing._UnionGenericAlias'> 104
  <class 'abc.abstractproperty'> 1
  <class 'mappingproxy'> 10
  <class 'weakref.WeakKeyDictionary'> 8
  <class 'itertools.count'> 5
  <class 'method-wrapper'> 3
  <class 'threading._MainThread'> 1
  <class 'string.Formatter'> 1
  <class 'logging.Formatter'> 1
  <class 'weakref.WeakValueDictionary'> 1
  <class 'logging._StderrHandler'> 1
  <class 'logging.RootLogger'> 1
  <class 'logging.PercentStyle'> 1
  <class 'logging.Manager'> 1
  <class 'logging.PlaceHolder'> 10
  <enum 'AddressFamily'> 29
  <enum 'SocketKind'> 7
  <enum 'MsgFlag'> 13
  <enum 'AddressInfo'> 7
  <enum 'Signals'> 32
  <enum 'Handlers'> 2
  <enum 'Sigmasks'> 3
  <enum '_SSLMethod'> 6
  <enum 'Options'> 12
  <enum 'AlertDescription'> 27
  <enum 'SSLErrorNumber'> 9
  <enum 'VerifyFlags'> 5
  <enum 'VerifyMode'> 3
  <enum 'TLSVersion'> 7
  <enum '_TLSContentType'> 6
  <enum '_TLSAlertType'> 34
  <enum '_TLSMessageType'> 22
  <enum 'Purpose'> 2
  <class 'ssl._ASN1Object'> 2
  <enum '_SendfileMode'> 3
  <class 'ast.Load'> 1
  <class 'ast.Store'> 1
  <class 'ast.Del'> 1
  <class 'ast.And'> 1
  <class 'ast.Or'> 1
  <class 'ast.Add'> 1
  <class 'ast.Sub'> 1
  <class 'ast.Mult'> 1
  <class 'ast.MatMult'> 1
  <class 'ast.Div'> 1
  <class 'ast.Mod'> 1
  <class 'ast.Pow'> 1
  <class 'ast.LShift'> 1
  <class 'ast.RShift'> 1
  <class 'ast.BitOr'> 1
  <class 'ast.BitXor'> 1
  <class 'ast.BitAnd'> 1
  <class 'ast.FloorDiv'> 1
  <class 'ast.Invert'> 1
  <class 'ast.Not'> 1
  <class 'ast.UAdd'> 1
  <class 'ast.USub'> 1
  <class 'ast.Eq'> 1
  <class 'ast.NotEq'> 1
  <class 'ast.Lt'> 1
  <class 'ast.LtE'> 1
  <class 'ast.Gt'> 1
  <class 'ast.GtE'> 1
  <class 'ast.Is'> 1
  <class 'ast.IsNot'> 1
  <class 'ast.In'> 1
  <class 'ast.NotIn'> 1
  <class 'ast._ABC'> 5
  <enum '_Precedence'> 17
  <enum '_ParameterKind'> 5
  <class 'asyncio.events._RunningLoop'> 1
  <class 'typing._CallableGenericAlias'> 10
  <class '__future__._Feature'> 10
  <enum 'NicDuplex'> 3
  <enum 'BatteryTime'> 2
  <class 'psutil._common._WrapNumbers'> 1
  <enum 'IOPriority'> 4
  <enum 'Negsignal'> 32
  <class 'ctypes.LibraryLoader'> 2
  <class 'ctypes.PyDLL'> 1
  <class 'ctypes.CFUNCTYPE.<locals>.CFunctionType'> 2
  <class 'ctypes.PYFUNCTYPE.<locals>.CFunctionType'> 3
  <class 'ctypes._endian._swapped_meta'> 1
  <class '_ctypes.PyCPointerType'> 2
  <class 'ctypes.CDLL'> 1
  <class 'ctypes.CDLL.__init__.<locals>._FuncPtr'> 1
  <class 'psutil._pslinux.NetConnections'> 1
  <class '_ctypes.PyCFuncPtrType'> 7
  <class 'psutil._pslinux.scputimes'> 17
  <class 'multiprocessing.process._MainProcess'> 1
  <class 'multiprocessing.process.AuthenticationString'> 1
  <class 'multiprocessing.context.DefaultContext'> 1
  <class '_thread._local'> 1
  <class 'multiprocessing.context.ForkContext'> 1
  <class 'multiprocessing.context.SpawnContext'> 1
  <class 'multiprocessing.context.ForkServerContext'> 1
  <class 'argparse.ArgumentParser'> 1
  <class 'argparse._ArgumentGroup'> 2
  <class 'argparse._HelpAction'> 1
  <class 'argparse._VersionAction'> 1
  <class 'argparse._StoreAction'> 5
  <class 'argparse._StoreTrueAction'> 2
  <class 'argparse.Namespace'> 1
  <class '__main__.TimeStamper'> 1
  <class 'platform.uname_result'> 1
  <class 'collections.OrderedDict'> 1
  <class 'functools.cached_property'> 2
  <class '_distutils_hack.DistutilsMetaFinder.spec_for_distutils.<locals>.DistutilsLoader'> 1
  <class 'random.Random'> 1
  <class 'calendar._localized_day'> 2
  <class 'calendar._localized_month'> 2
  <class 'calendar.TextCalendar'> 1
  <class 'email.charset.Charset'> 2
  <class 'email._policybase.Compat32'> 1
  <class 'email._encoded_words._QByteMap'> 1
  <class 'configparser.Interpolation'> 1
  <class 'configparser.BasicInterpolation'> 1
  <class 'pathlib._WindowsFlavour'> 1
  <class 'pathlib._PosixFlavour'> 1
  <class 'pathlib._NormalAccessor'> 1
  <class 'collections.defaultdict'> 3
  <enum 'EIClass'> 6
  <enum 'EIData'> 6
  <enum 'EMachine'> 15
  <class 'distutils._vendor.packaging._structures.InfinityType'> 1
  <class 'distutils._vendor.packaging._structures.NegativeInfinityType'> 1
  <class 'operator.attrgetter'> 2
  <class 'json.encoder.JSONEncoder'> 1
  <class 'json.decoder.JSONDecoder'> 1
  <class '_json.Scanner'> 1
  <class 'importlib_metadata.MetadataPathFinder'> 1
  <class 'importlib_metadata.FastPath'> 6
  <class 'importlib_metadata.DistributionFinder.Context'> 2
  <class 'importlib_metadata.Lookup'> 6
  <class 'importlib_metadata._collections.FreezableDefaultDict'> 12
  <class 'importlib_metadata._text.FoldedCase'> 10
  <class 'pathlib.PosixPath'> 32
  <class 'packaging._structures.InfinityType'> 1
  <class 'packaging._structures.NegativeInfinityType'> 1
  <class 'types.GenericAlias'> 120
  <class 'typing.ForwardRef'> 37
  <class 'dataclasses._HAS_DEFAULT_FACTORY_CLASS'> 1
  <class 'dataclasses._MISSING_TYPE'> 1
  <class 'dataclasses._FIELD_BASE'> 3
  <class '_frozen_importlib_external._NamespacePath'> 1
  <class '_frozen_importlib_external._NamespaceLoader'> 1
  <class 'dataclasses._DataclassParams'> 2
  <class 'dataclasses.Field'> 6
  <enum 'HTTPStatus'> 62
  <class 'jaraco.context.ExceptionTrap'> 1
  <class 'distutils.command.bdist.ListCompat'> 1
  <class 'email._header_value_parser.ValueTerminal'> 3
  <class 'email.contentmanager.ContentManager'> 1
  <class 'email.policy.EmailPolicy'> 5
  <class 'weakcallableproxy'> 1
  <class 'email.headerregistry.HeaderRegistry'> 1
  <class '_ctypes.PyCArrayType'> 1
  <class 'wheel.vendored.packaging._structures.InfinityType'> 1
  <class 'wheel.vendored.packaging._structures.NegativeInfinityType'> 1
  <class 'distutils._collections.RangeValueUndefined'> 1
  <class 'distutils._collections.RangeMap.Item'> 2
  <class 'numpy._globals._NoValueType'> 1
  <enum '_CopyMode'> 3
  <class 'numpy.core.numerictypes._typedict'> 5
  <class 'numpy.core._ufunc_config._unspecified'> 1
  <enum 'DisplayModes'> 2
  <class 'numpy.iinfo'> 3
  <class 'numpy.lib.index_tricks.MGridClass'> 1
  <class 'numpy.lib.index_tricks.OGridClass'> 1
  <class 'numpy.lib.index_tricks.RClass'> 1
  <class 'numpy.lib.index_tricks.CClass'> 1
  <class 'numpy.lib.index_tricks.IndexExpression'> 2
  <class 'numpy.lib._datasource._FileOpeners'> 1
# Types 268 -> 279
New Types
  <class 'uuid.UUID'> 4
  <class 'p4p.util._DefaultWorkQueue'> 1
  <class 'p4p.nt.ndarray.NTNDArray'> 1
  <class 'p4p.nt.ClientUnwrapper'> 1
  <class 'p4p.util.ThreadedWorkQueue'> 1
  <class 'queue.Queue'> 1
  <class 'p4p.client.thread.Subscription'> 1
  <enum 'SafeUUID'> 3
  <class 'p4p.client.raw._ClientProvider'> 1
  <class 'p4p.client.raw.Subscription'> 1
  <class 'p4p.client.thread.Context'> 1
Known Types
  <class '_frozen_importlib_external.SourceFileLoader'> 451 delta 13
  <class 'set'> 271 delta 7
  <class 'module'> 529 delta 13
  <class 'type'> 1280 delta 41
  <class 'threading.Condition'> 6 delta 4
  <class '_frozen_importlib.ModuleSpec'> 526 delta 13
  <class '_frozen_importlib_external.FileFinder'> 62 delta 3
  <class 'collections.deque'> 7 delta 5
  <class 'property'> 476 delta 21
  <class 'classmethod'> 275 delta 5
  <class 'cell'> 1199 delta 10
  <class 'weakref'> 2092 delta 45
  <class 'list'> 912 delta 41
  <class 'enum.EnumMeta'> 44 delta 1
  <class 'logging.PlaceHolder'> 11 delta 1
  <class 'getset_descriptor'> 1544 delta 51
  <class 'tuple'> 5133 delta -107
  <class 'threading.Thread'> 2 delta 1
  <class 'member_descriptor'> 535 delta 2
  <class 'method'> 114 delta 8
  <class 'logging.Logger'> 18 delta 8
  <class 'dict'> 5228 delta 117
  <class 'staticmethod'> 285 delta 11
  <class 'builtin_function_or_method'> 1420 delta 10
  <class 'threading.Event'> 3 delta 1
  <class 'function'> 10916 delta 218
Subscribe to test:image
update at: Mon Jul 29 15:19:22 2024 ntndarray([0.88948077, 0.80808534, 0.15774478, ..., 0.2339425 ,
           0.50943913, 0.80818115])
update at: Mon Jul 29 15:19:23 2024 ntndarray([0.00915326, 0.3034378 , 0.57675786, ..., 0.38086599,
           0.74565214, 0.73563673])
update at: Mon Jul 29 15:19:24 2024 ntndarray([0.53531862, 0.95518647, 0.72261808, ..., 0.88671387,
           0.42174363, 0.97457293])
update at: Mon Jul 29 15:19:25 2024 ntndarray([0.16529446, 0.07240393, 0.67914699, ..., 0.57518091,
           0.17959206, 0.53960211])
update at: Mon Jul 29 15:19:26 2024 ntndarray([0.39322927, 0.27907059, 0.81532388, ..., 0.99358384,
           0.71334073, 0.81009463])
update at: Mon Jul 29 15:19:27 2024 ntndarray([0.68660242, 0.04225231, 0.29041005, ..., 0.10759069,
           0.2270905 , 0.08700961])
update at: Mon Jul 29 15:19:28 2024 ntndarray([0.56796174, 0.89054764, 0.83958117, ..., 0.50008557,
           0.4631533 , 0.92509806])
update at: Mon Jul 29 15:19:29 2024 ntndarray([0.49487581, 0.12622231, 0.92087848, ..., 0.09538005,
           0.94597368, 0.67363178])
update at: Mon Jul 29 15:19:30 2024 ntndarray([0.99065152, 0.25975972, 0.33997346, ..., 0.75340647,
           0.3691044 , 0.56180981])
update at: Mon Jul 29 15:19:31 2024 ntndarray([0.78958105, 0.7836688 , 0.80766434, ..., 0.31993987,
           0.20575739, 0.87545632])
update at: Mon Jul 29 15:19:32 2024 ntndarray([0.60691574, 0.69881929, 0.9765346 , ..., 0.75573684,
           0.08274068, 0.99311975])
update at: Mon Jul 29 15:19:33 2024 ntndarray([0.6725386 , 0.08743383, 0.82371958, ..., 0.95213933,
           0.53894364, 0.4869051 ])
update at: Mon Jul 29 15:19:34 2024 ntndarray([0.79289153, 0.67281777, 0.21488787, ..., 0.94937952,
           0.6726821 , 0.21943264])
update at: Mon Jul 29 15:19:35 2024 ntndarray([0.52097549, 0.55685007, 0.61895185, ..., 0.26170889,
           0.66955926, 0.60894508])
update at: Mon Jul 29 15:19:36 2024 ntndarray([0.23320284, 0.26308733, 0.07732113, ..., 0.45742058,
           0.82897402, 0.81002462])
update at: Mon Jul 29 15:19:37 2024 ntndarray([0.6509252 , 0.23394302, 0.59224077, ..., 0.5291412 ,
           0.05816402, 0.71078043])
update at: Mon Jul 29 15:19:38 2024 ntndarray([0.33627183, 0.55598246, 0.61134521, ..., 0.82797374,
           0.15480704, 0.6034754 ])
update at: Mon Jul 29 15:19:39 2024 ntndarray([0.9837606 , 0.58969441, 0.60435316, ..., 0.45738261,
           0.53349773, 0.44791723])
update at: Mon Jul 29 15:19:40 2024 ntndarray([0.82536841, 0.75276263, 0.50036576, ..., 0.41733934,
           0.0258995 , 0.69630477])
update at: Mon Jul 29 15:19:41 2024 ntndarray([0.0014078 , 0.22379201, 0.12031005, ..., 0.70326287,
           0.30977296, 0.04322276])
update at: Mon Jul 29 15:19:42 2024 ntndarray([0.26297044, 0.49422244, 0.27311176, ..., 0.95266452,
           0.71225162, 0.21939404])
update at: Mon Jul 29 15:19:43 2024 ntndarray([0.46008888, 0.26198423, 0.42318045, ..., 0.58451527,
           0.2458005 , 0.89106681])
update at: Mon Jul 29 15:19:44 2024 ntndarray([0.84412881, 0.5763333 , 0.4945668 , ..., 0.71023147,
           0.4983693 , 0.75782506])
update at: Mon Jul 29 15:19:45 2024 ntndarray([0.98390841, 0.40113701, 0.17593701, ..., 0.93008275,
           0.82467484, 0.0691953 ])
update at: Mon Jul 29 15:19:46 2024 ntndarray([0.60189594, 0.27496132, 0.02948808, ..., 0.21107828,
           0.6068704 , 0.26987279])
update at: Mon Jul 29 15:19:47 2024 ntndarray([0.24304713, 0.7548808 , 0.36031065, ..., 0.46673133,
           0.55319368, 0.92161975])
update at: Mon Jul 29 15:19:48 2024 ntndarray([0.11361457, 0.33819789, 0.88916427, ..., 0.42672115,
           0.69692996, 0.15852173])
update at: Mon Jul 29 15:19:49 2024 ntndarray([0.05088877, 0.33029505, 0.51874383, ..., 0.68942426,
           0.01990551, 0.15299849])
update at: Mon Jul 29 15:19:50 2024 ntndarray([0.0233713 , 0.37392876, 0.14618515, ..., 0.96926514,
           0.14657285, 0.0036325 ])
update at: Mon Jul 29 15:19:51 2024 ntndarray([0.34490529, 0.79080721, 0.26472225, ..., 0.52572048,
           0.15589003, 0.69471129])
update at: Mon Jul 29 15:19:52 2024 ntndarray([0.32441889, 0.49165095, 0.89323763, ..., 0.77954772,
           0.73916887, 0.75405599])
update at: Mon Jul 29 15:19:53 2024 ntndarray([0.31302421, 0.62102897, 0.45177831, ..., 0.59670304,
           0.63373283, 0.02035265])
update at: Mon Jul 29 15:19:54 2024 ntndarray([0.89413249, 0.8478253 , 0.19436802, ..., 0.68812069,
           0.37946531, 0.72458167])
update at: Mon Jul 29 15:19:55 2024 ntndarray([0.54652486, 0.66531539, 0.02247208, ..., 0.35115278,
           0.32438975, 0.69007471])
update at: Mon Jul 29 15:19:56 2024 ntndarray([0.20738018, 0.87049638, 0.8466778 , ..., 0.00712368,
           0.62461175, 0.27796851])
update at: Mon Jul 29 15:19:57 2024 ntndarray([0.82695497, 0.29528117, 0.75151522, ..., 0.30431358,
           0.38964821, 0.63189963])
update at: Mon Jul 29 15:19:58 2024 ntndarray([0.48933846, 0.8951128 , 0.43665282, ..., 0.3069141 ,
           0.34329735, 0.92762246])
update at: Mon Jul 29 15:19:59 2024 ntndarray([0.43965088, 0.81692041, 0.61158864, ..., 0.31844092,
           0.4303928 , 0.55845074])
update at: Mon Jul 29 15:20:00 2024 ntndarray([0.83440943, 0.10748356, 0.66581659, ..., 0.53325634,
           0.55893557, 0.30374983])
update at: Mon Jul 29 15:20:01 2024 ntndarray([0.35665538, 0.03309808, 0.65885275, ..., 0.95045246,
           0.12398847, 0.7621034 ])
update at: Mon Jul 29 15:20:02 2024 ntndarray([0.27612244, 0.83109052, 0.25846442, ..., 0.13629387,
           0.67698527, 0.3924973 ])
update at: Mon Jul 29 15:20:03 2024 ntndarray([0.90378791, 0.99931894, 0.67629058, ..., 0.44113512,
           0.47664187, 0.50823046])
update at: Mon Jul 29 15:20:04 2024 ntndarray([0.49637257, 0.9592305 , 0.83140579, ..., 0.4286001 ,
           0.15992849, 0.27433051])
update at: Mon Jul 29 15:20:05 2024 ntndarray([0.96395544, 0.70859017, 0.49625357, ..., 0.78240334,
           0.35189901, 0.96031691])
update at: Mon Jul 29 15:20:06 2024 ntndarray([0.37534977, 0.47765067, 0.41509627, ..., 0.29631395,
           0.19125364, 0.55715458])
update at: Mon Jul 29 15:20:07 2024 ntndarray([0.97627195, 0.17674062, 0.09773373, ..., 0.13908582,
           0.0454327 , 0.21122954])
update at: Mon Jul 29 15:20:08 2024 ntndarray([0.73776735, 0.96362895, 0.97246904, ..., 0.65315925,
           0.24831554, 0.48967361])
update at: Mon Jul 29 15:20:09 2024 ntndarray([0.09442379, 0.57585859, 0.45841942, ..., 0.14843339,
           0.59126965, 0.8771941 ])
update at: Mon Jul 29 15:20:10 2024 ntndarray([0.9543876 , 0.95470854, 0.27451696, ..., 0.69482771,
           0.27957594, 0.0083334 ])
update at: Mon Jul 29 15:20:11 2024 ntndarray([0.55811088, 0.42836096, 0.35588612, ..., 0.60312459,
           0.36352646, 0.1049829 ])
update at: Mon Jul 29 15:20:12 2024 ntndarray([0.4728351 , 0.15944432, 0.73973863, ..., 0.5171731 ,
           0.7964223 , 0.18699251])
update at: Mon Jul 29 15:20:13 2024 ntndarray([0.10195623, 0.80238395, 0.49453008, ..., 0.44152081,
           0.13527225, 0.59480336])
update at: Mon Jul 29 15:20:14 2024 ntndarray([0.4538856 , 0.51781237, 0.91219826, ..., 0.7048768 ,
           0.60310758, 0.68092779])
update at: Mon Jul 29 15:20:15 2024 ntndarray([0.72563999, 0.72592219, 0.69824143, ..., 0.64873761,
           0.73315359, 0.59597588])
update at: Mon Jul 29 15:20:16 2024 ntndarray([0.57323982, 0.32418256, 0.65688366, ..., 0.03778747,
           0.65665265, 0.7424717 ])
update at: Mon Jul 29 15:20:17 2024 ntndarray([0.30541154, 0.45380412, 0.8672212 , ..., 0.73908269,
           0.61975355, 0.48355305])
update at: Mon Jul 29 15:20:18 2024 ntndarray([0.09057828, 0.45610635, 0.58901528, ..., 0.85434556,
           0.43300542, 0.1753396 ])
update at: Mon Jul 29 15:20:19 2024 ntndarray([0.79669298, 0.33795375, 0.05385382, ..., 0.60182664,
           0.01273596, 0.37662605])
update at: Mon Jul 29 15:20:20 2024 ntndarray([0.60836125, 0.34940863, 0.93338438, ..., 0.73783395,
           0.57972581, 0.73929917])
update at: Mon Jul 29 15:20:21 2024 ntndarray([0.84082739, 0.05879541, 0.05573507, ..., 0.99664217,
           0.12024415, 0.77010999])
update at: Mon Jul 29 15:20:22 2024 ntndarray([0.25531444, 0.0471842 , 0.37717968, ..., 0.35695181,
           0.14693953, 0.85046624])Known Types
  <class 'list'> 911 delta -1
Known Types
  <class 'list'> 912 delta 1

update at: Mon Jul 29 15:20:23 2024 ntndarray([0.85222406, 0.40707288, 0.29024359, ..., 0.92628315,
           0.94249881, 0.63788947])
update at: Mon Jul 29 15:20:24 2024 ntndarray([0.69813662, 0.55594554, 0.03298022, ..., 0.39698805,
           0.62821712, 0.50095346])
update at: Mon Jul 29 15:20:25 2024 ntndarray([0.74457529, 0.43988259, 0.82698175, ..., 0.67826952,
           0.06866892, 0.6977172 ])
update at: Mon Jul 29 15:20:26 2024 ntndarray([0.29682963, 0.58157287, 0.10257006, ..., 0.82432709,
           0.19723163, 0.43194245])
update at: Mon Jul 29 15:20:27 2024 ntndarray([0.02080136, 0.6026978 , 0.52165979, ..., 0.16999228,
           0.53064357, 0.02183319])
update at: Mon Jul 29 15:20:28 2024 ntndarray([0.24768224, 0.00593085, 0.02890672, ..., 0.68385423,
           0.80425682, 0.75015797])
update at: Mon Jul 29 15:20:29 2024 ntndarray([0.44399558, 0.49068455, 0.11606789, ..., 0.28550532,
           0.79392765, 0.80399569])
update at: Mon Jul 29 15:20:30 2024 ntndarray([0.46347518, 0.76725723, 0.73738944, ..., 0.99810749,
           0.29128765, 0.2487097 ])
update at: Mon Jul 29 15:20:31 2024 ntndarray([0.12390415, 0.12262416, 0.14776824, ..., 0.20265055,
           0.51178425, 0.87619671])
update at: Mon Jul 29 15:20:32 2024 ntndarray([0.00408314, 0.8674957 , 0.31849364, ..., 0.55260886,
           0.06582749, 0.54212758])
update at: Mon Jul 29 15:20:33 2024 ntndarray([0.33403712, 0.87474671, 0.78033959, ..., 0.95850651,
           0.82947237, 0.20702904])
update at: Mon Jul 29 15:20:34 2024 ntndarray([0.35316909, 0.00232194, 0.76573105, ..., 0.74322818,
           0.38026412, 0.39581964])
update at: Mon Jul 29 15:20:36 2024 ntndarray([0.33060674, 0.3171133 , 0.59033976, ..., 0.16192177,
           0.43092758, 0.07618644])
update at: Mon Jul 29 15:20:37 2024 ntndarray([0.64679848, 0.63042987, 0.36655063, ..., 0.44232135,
           0.41181902, 0.16089122])
update at: Mon Jul 29 15:20:38 2024 ntndarray([0.99450647, 0.28730654, 0.48956947, ..., 0.52405487,
           0.45957377, 0.63296123])
update at: Mon Jul 29 15:20:39 2024 ntndarray([0.55326713, 0.59293533, 0.43338545, ..., 0.51499578,
           0.03563482, 0.91891789])
update at: Mon Jul 29 15:20:40 2024 ntndarray([0.88507298, 0.98865585, 0.97640628, ..., 0.14797122,
           0.20285978, 0.2512072 ])
update at: Mon Jul 29 15:20:41 2024 ntndarray([0.71327667, 0.50961723, 0.95736912, ..., 0.19066693,
           0.42865549, 0.05109269])
update at: Mon Jul 29 15:20:42 2024 ntndarray([0.2653611 , 0.12481371, 0.25485442, ..., 0.29591461,
           0.5658925 , 0.063862  ])
update at: Mon Jul 29 15:20:43 2024 ntndarray([0.982261  , 0.96520168, 0.01716731, ..., 0.66193709,
           0.64924992, 0.00848089])
update at: Mon Jul 29 15:20:44 2024 ntndarray([0.44660625, 0.83096795, 0.84435299, ..., 0.72689301,
           0.4803344 , 0.57891324])
update at: Mon Jul 29 15:20:45 2024 ntndarray([0.15257487, 0.30478363, 0.11038114, ..., 0.06746222,
           0.47956012, 0.42415688])
update at: Mon Jul 29 15:20:46 2024 ntndarray([0.15644172, 0.74333779, 0.14904707, ..., 0.67049765,
           0.0898854 , 0.23109068])
update at: Mon Jul 29 15:20:47 2024 ntndarray([0.44659205, 0.82604333, 0.07046828, ..., 0.75878683,
           0.08473492, 0.13439464])
update at: Mon Jul 29 15:20:48 2024 ntndarray([0.05173595, 0.26615633, 0.52563158, ..., 0.52414293,
           0.14561885, 0.2339958 ])
update at: Mon Jul 29 15:20:49 2024 ntndarray([0.31158463, 0.04524821, 0.94739916, ..., 0.79531008,
           0.44874875, 0.62438314])
update at: Mon Jul 29 15:20:50 2024 ntndarray([0.10814975, 0.1652295 , 0.70202478, ..., 0.15201437,
           0.81572758, 0.89032739])
update at: Mon Jul 29 15:20:51 2024 ntndarray([0.3394914 , 0.72895521, 0.40157159, ..., 0.78442808,
           0.38509837, 0.89096201])
update at: Mon Jul 29 15:20:52 2024 ntndarray([0.45127544, 0.82184511, 0.54052672, ..., 0.83550642,
           0.47670169, 0.04743766])
update at: Mon Jul 29 15:20:53 2024 ntndarray([0.58427894, 0.21679251, 0.86364407, ..., 0.36699396,
           0.44253922, 0.62374388])
update at: Mon Jul 29 15:20:54 2024 ntndarray([0.7514706 , 0.06648131, 0.53148841, ..., 0.27489916,
           0.70102665, 0.8956108 ])
update at: Mon Jul 29 15:20:55 2024 ntndarray([0.48723517, 0.2348459 , 0.26680263, ..., 0.40000431,
           0.20767691, 0.58016166])
update at: Mon Jul 29 15:20:56 2024 ntndarray([0.97083879, 0.5641177 , 0.91847162, ..., 0.46441448,
           0.28847415, 0.86735251])
update at: Mon Jul 29 15:20:57 2024 ntndarray([0.98064348, 0.29516665, 0.9334868 , ..., 0.07939419,
           0.29793   , 0.53374308])
update at: Mon Jul 29 15:20:58 2024 ntndarray([0.63080942, 0.83907059, 0.66959862, ..., 0.39714106,
           0.86511722, 0.21947576])
update at: Mon Jul 29 15:20:59 2024 ntndarray([0.21397611, 0.47928562, 0.74032965, ..., 0.87059588,
           0.12103931, 0.17056349])
update at: Mon Jul 29 15:21:00 2024 ntndarray([0.95409719, 0.06211293, 0.81171761, ..., 0.69012497,
           0.67350074, 0.8247829 ])
update at: Mon Jul 29 15:21:01 2024 ntndarray([0.13407584, 0.68440224, 0.73413749, ..., 0.38026757,
           0.17339079, 0.56069999])
update at: Mon Jul 29 15:21:02 2024 ntndarray([0.44154235, 0.13787747, 0.74178009, ..., 0.08717143,
           0.48017373, 0.81984309])
Close subscription
Done
mprof: Sampling memory every 0.1s
running new process
running as a Python program...

[azukov]

I think one needs to decorate methods inside your code base to make mprof show detailed information.

Meanwhile I ran memray run --native client.py and it possibly suggests (I, by no means, am a serious C++ person) that memory allocation happens in from_wire_field() in pvxs code.

And, if so, the affecting change happened somewhere between version 1.0.1 and 1.1.4 of pvxs.
Although I might be misinterpreting the result of memray profiler.

[mdavidsaver]

suggests ... that memory allocation happens in from_wire_field()

This is unsurprising. from_wire_field() is responsible for deserializing data updates. For array updates, this involves allocating memory here: https://github.com/mdavidsaver/pvxs/blob/25780f950619ff30dd25b544b3896afa34889614/src/pvaproto.h#L466

On the C++ side, these allocations are tracked by the shared_array<T> reference counting container. So a simple memory leak, in the sense of omitting free() is unlikely. Rather, it is more likely that there is some place where map or list which is growing without bound. These are harder to find.

Another builtin diagnostic is p4p.listRefs(), which returns instance counts for some C++ classes. In particular, StructTop which is the top level container referenced by pvxs::Value and p4p.Value.

[mdavidsaver]

Also, could you run the P4P unittests for each configuration?

[mdavidsaver]

I am able to replicate this leak. I think I see what is going on. cf. epics-base/pvxs#71

[azukov]

@mdavidsaver Thanks for looking into that!

[mdavidsaver]

I think that epics-base/pvxs#77 plugs this leak. @azukov @jeinstei Can you confirm?

[azukov]

It's a bit unclear how to do it.
I tried:

pip install git+https://github.com/mdavidsaver/pvxs.git@330097b7d455fbae689c890e5b5fef1a59d885e7
pip install p4p==4.1.12

First command ran fine but the second gave conflict with EPICS base:

ERROR: Cannot install p4p==4.1.12 and pvxslibs==1.3.1 because these package versions have conflicting dependencies.

The conflict is caused by:
    p4p 4.1.12 depends on epicscorelibs<7.0.7.99.1 and >=7.0.7.99.0.2
    pvxslibs 1.3.1 depends on epicscorelibs<7.0.7.99.2 and >=7.0.7.99.1.1a2

[mdavidsaver]

I think you want --no-binary p4p to force a local build:

pip install --no-binary p4p p4p==4.1.12

However, you might also need to use a --constraint file to require PIP to pull in pvxslibs from git when building p4p.

If you find that PIP is ruining your friday, you could instead try: https://epics-base.github.io/p4p/building.html#build-as-epics-module. Then end with make nose and look to see how PYTHONPATH is set. (should be something like PYTHONPATH=$(TOP)/python3.11/linux-x86_64)

[azukov]

Looks like the problem is in pvxs pyproject.toml that insists on having specific version of epicscorelibs

https://github.com/mdavidsaver/pvxs-dev/blob/330097b7d455fbae689c890e5b5fef1a59d885e7/pyproject.toml#L1-L2

if I remove specific version it works. Here is how I tested it.
Prepare environment and clone pvxs (I haven't figured out how to run it with numpy>=2.0.0)

pip install --no-cache-dir "numpy<2" memory_profiler
pip install --no-cache-dir epicscorelibs
git clone -b fix_cache_sync --recursive https://github.com/mdavidsaver/pvxs-dev.git

Remove specific version and build/install pvxs and p4p

sed -i 's/\bepicscorelibs[^"]*/epicscorelibs/' pvxs-dev/pyproject.toml
pip install --no-cache-dir pvxs-dev/
pip install --no-cache-dir p4p==4.1.12

Run test

mprof run -o patch.dat client.py

Switch to previous commit 1ca7600a2b5322a and run test again

cd pvxs-dev
git checkout 1ca7600a2b5322a
pip install --no-cache-dir pvxs-dev/
mprof run -o no_patch.dat client.py

So the patch you provided seems to fix the bug, thanks a lot!