Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge branch 'master' into enums #1

Merged
merged 40 commits into from
Feb 7, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
74de347
Support PyCapsule (#477)
anivegesana Jun 7, 2022
260b372
tests: fix a test for session saving (#492)
leogama Jun 7, 2022
4462409
Fix `dill._dill` submodule being saved as `GLOBAL "dill._shims" "_dil…
leogama Jun 8, 2022
a6bf9a2
Create a soft_def shim to combine move_to and Getattr (#473)
anivegesana Jun 9, 2022
0ce3baf
Incidental implementation specific types (#472)
anivegesana Jun 10, 2022
dc9a1b9
Fix bug in pickling MappingProxyType in PyPy 3.7+ (#506)
anivegesana Jun 10, 2022
408b8de
Bring back old method pickling function (#511)
anivegesana Jun 16, 2022
13e3f80
protect against sys.modules contents change
mmckerns Jun 17, 2022
60911fc
correct failures and succeeds dicts
mmckerns Jun 25, 2022
178e7c0
correct failures for registered
mmckerns Jun 26, 2022
73d9f85
unregister SymtableEntryType
mmckerns Jun 26, 2022
2cbc2a9
Merge pull request #519 from uqfoundation/objects
mmckerns Jun 26, 2022
ecf60d0
help silence some test noise
mmckerns Jun 26, 2022
38d2a95
better python handling in test main
mmckerns Jun 26, 2022
9000ab9
Add detailed trace mode showing saved object size and visual depth le…
leogama Jul 1, 2022
4220929
Kickstart support drop for Python < 3.7 (#499)
leogama Jul 1, 2022
3881a2b
fix #521: support develop mode
mmckerns Jul 2, 2022
dc35b66
Rewrite _create_code() with Structural Pattern Matching (limited to t…
leogama Jul 3, 2022
7b63977
fix #522: unify README and module doc
mmckerns Jul 6, 2022
2382432
add some no-indent bullets
mmckerns Jul 6, 2022
80ad351
Fix tox tests so they will actually fail (#524)
AdamWill Jul 8, 2022
0802fdd
minor cleanup
mmckerns Jul 9, 2022
0fa524a
Fix load_session() and restrict loading a session in a different modu…
leogama Jul 10, 2022
c23e049
update load_module dump_module docs
mmckerns Jul 10, 2022
6b90f52
fix dump_module() bugs and rename parameter 'main' to 'module' (#526)
leogama Jul 15, 2022
166a024
tweak load_module/dump_module docs
mmckerns Jul 15, 2022
74e0fd4
A temporary quick fix for dataclass serialization (#500) (#503)
anivegesana Jul 21, 2022
990b5a1
fix: #528, test_registered syncs objects w typemap
mmckerns Jul 21, 2022
c21d67a
print on fail for test_registered
mmckerns Jul 21, 2022
863d73c
noisy testing
mmckerns Jul 21, 2022
7489dd3
CLibraryLoader on unix in 3.11
mmckerns Jul 21, 2022
59b7b8f
quiet testing
mmckerns Jul 21, 2022
72a43b9
add logger module to sphinx
mmckerns Jul 22, 2022
5a66152
Move session-related code to new session module (#530)
leogama Jul 23, 2022
0ca195f
fix: #517 special handling of typing module
mmckerns Jul 23, 2022
ee740ab
test special case typing.Any
mmckerns Jul 23, 2022
a9ec103
more clearly note cpython bug
mmckerns Jul 26, 2022
b2fa04d
Temporary quick fix for getsource() on IPython interpreter (fixes #34…
leogama Jul 29, 2022
87b8541
Fix pickling errors thrown when saving some Stdlib modules (#529)
leogama Aug 1, 2022
bfba6d1
Merge branch 'master' into enums
leogama Aug 6, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@
/docs/build
/build
/README
/dill/info.py
/dill/__info__.py
4 changes: 2 additions & 2 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,8 @@ install:
- python -m pip install .

script:
- for test in tests/__init__.py; do echo $test ; if [[ $COVERAGE == "true" ]]; then coverage run -a $test > /dev/null; else python $test > /dev/null; fi ; done
- for test in tests/test_*.py; do echo $test ; if [[ $COVERAGE == "true" ]]; then coverage run -a $test > /dev/null; else python $test > /dev/null; fi ; done
- for test in dill/tests/__init__.py; do echo $test ; if [[ $COVERAGE == "true" ]]; then coverage run -a $test > /dev/null; else python $test > /dev/null; fi ; done
- for test in dill/tests/test_*.py; do echo $test ; if [[ $COVERAGE == "true" ]]; then coverage run -a $test > /dev/null; else python $test > /dev/null; fi ; done

after_success:
- if [[ $COVERAGE == "true" ]]; then bash <(curl -s https://codecov.io/bash); else echo ''; fi
Expand Down
2 changes: 1 addition & 1 deletion MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ include README*
include MANIFEST.in
include pyproject.toml
include tox.ini
include version.py
include scripts/*
include tests/*py
recursive-include docs *
include .*
prune .git
Expand Down
55 changes: 30 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,26 +28,27 @@ a trustworthy source.

``dill`` is part of ``pathos``, a python framework for heterogeneous computing.
``dill`` is in active development, so any user feedback, bug reports, comments,
or suggestions are highly appreciated. A list of issues is located at https://github.com/uqfoundation/dill/issues, with a legacy list maintained at https://uqfoundation.github.io/project/pathos/query.
or suggestions are highly appreciated. A list of issues is located at
https://github.com/uqfoundation/dill/issues, with a legacy list maintained at
https://uqfoundation.github.io/project/pathos/query.


Major Features
--------------
``dill`` can pickle the following standard types:

* none, type, bool, int, long, float, complex, str, unicode,
* none, type, bool, int, float, complex, bytes, str,
* tuple, list, dict, file, buffer, builtin,
* both old and new style classes,
* instances of old and new style classes,
* python classes, namedtuples, dataclasses, metaclasses,
* instances of classes,
* set, frozenset, array, functions, exceptions

``dill`` can also pickle more 'exotic' standard types:

* functions with yields, nested functions, lambdas
* functions with yields, nested functions, lambdas,
* cell, method, unboundmethod, module, code, methodwrapper,
* dictproxy, methoddescriptor, getsetdescriptor, memberdescriptor,
* wrapperdescriptor, xrange, slice,
* notimplemented, ellipsis, quit
* methoddescriptor, getsetdescriptor, memberdescriptor, wrapperdescriptor,
* dictproxy, slice, notimplemented, ellipsis, quit

``dill`` cannot yet pickle these standard types:

Expand Down Expand Up @@ -133,7 +134,7 @@ There are a number of options to control serialization which are provided
as keyword arguments to several ``dill`` functions:

* with *protocol*, the pickle protocol level can be set. This uses the
same value as the ``pickle`` module, *HIGHEST_PROTOCOL* or *DEFAULT_PROTOCOL*.
same value as the ``pickle`` module, *DEFAULT_PROTOCOL*.
* with *byref=True*, ``dill`` to behave a lot more like pickle with
certain objects (like modules) pickled by reference as opposed to
attempting to pickle the object itself.
Expand Down Expand Up @@ -170,26 +171,30 @@ To aid in debugging pickling issues, use *dill.detect* which provides
tools like pickle tracing::

>>> import dill.detect
>>> dill.detect.trace(True)
>>> f = dumps(squared)
F1: <function <lambda> at 0x108899e18>
F2: <function _create_function at 0x108db7488>
# F2
Co: <code object <lambda> at 0x10866a270, file "<stdin>", line 1>
F2: <function _create_code at 0x108db7510>
# F2
# Co
D1: <dict object at 0x10862b3f0>
# D1
D2: <dict object at 0x108e42ee8>
# D2
# F1
>>> dill.detect.trace(False)
>>> with dill.detect.trace():
>>> dumps(squared)
┬ F1: <function <lambda> at 0x7fe074f8c280>
├┬ F2: <function _create_function at 0x7fe074c49c10>
│└ # F2 [34 B]
├┬ Co: <code object <lambda> at 0x7fe07501eb30, file "<stdin>", line 1>
│├┬ F2: <function _create_code at 0x7fe074c49ca0>
││└ # F2 [19 B]
│└ # Co [87 B]
├┬ D1: <dict object at 0x7fe0750d4680>
│└ # D1 [22 B]
├┬ D2: <dict object at 0x7fe074c5a1c0>
│└ # D2 [2 B]
├┬ D2: <dict object at 0x7fe074f903c0>
│├┬ D2: <dict object at 0x7fe074f8ebc0>
││└ # D2 [2 B]
│└ # D2 [23 B]
└ # F1 [180 B]

With trace, we see how ``dill`` stored the lambda (``F1``) by first storing
``_create_function``, the underlying code object (``Co``) and ``_create_code``
(which is used to handle code objects), then we handle the reference to
the global dict (``D2``). A ``#`` marks when the object is actually stored.
the global dict (``D2``) plus other dictionaries (``D1`` and ``D2``) that
save the lambda object's state. A ``#`` marks when the object is actually stored.


More Information
Expand Down
14 changes: 4 additions & 10 deletions dill/__diff.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,18 +10,15 @@
Module to show if an object has changed since it was memorised
"""

import builtins
import os
import sys
import types
try:
import numpy
HAS_NUMPY = True
except:
HAS_NUMPY = False
try:
import builtins
except ImportError:
import __builtin__ as builtins
HAS_NUMPY = False

# pypy doesn't use reference counting
getrefcount = getattr(sys, 'getrefcount', lambda x:0)
Expand All @@ -44,10 +41,7 @@ def get_attrs(obj):
if type(obj) in builtins_types \
or type(obj) is type and obj in builtins_types:
return
try:
return obj.__dict__
except:
return
return getattr(obj, '__dict__', None)


def get_seq(obj, cache={str: False, frozenset: False, list: True, set: True,
Expand Down Expand Up @@ -235,6 +229,6 @@ def _imp(*args, **kwds):

# memorise all already imported modules. This implies that this must be
# imported first for any changes to be recorded
for mod in sys.modules.values():
for mod in list(sys.modules.values()):
memorise(mod)
release_gone()
Loading