Pickling & decorators

[emptierset]

We are using this library for an implementation of a board game.

We need the ability to pickle and unpickle games. We are using dill for this because my initial attempts to pickle with basic pickle saw failures[0] when it tried to pickle certain things inside StateMachine. We noticed an interesting bug when unpicking saved games and I was wondering if you might have any input.

Essentially, we have a transition list like this:

end_game = s.TURN_END.to(s.GAME_END)

@end_game.cond
def game_is_over(self) -> bool:
   ...

advance_round =  end_game | s.TURN_END.to(s.GAME_PLAY)

When playing a game after it was pickled and unpickled, sometimes it goes to GAME_END at surprising times, even if manually calling game_is_over via REPL shows that it is false. After some digging, I identified that the cond isn't actually running, so the "game over" transition is always getting picked.

Do you have any idea why pickling and/or unpickling causes cond to "fall off"?

Is your expectation that state machines are serializable?

Do you think switching our codebase to a non-decorator approach would help?

---

[0]: Specifically, it failed with AttributeError: Can't pickle local object 'SignatureAdapter.wrap.<locals>.signature_adapter'.

[fgmacedo]

Hi @emptierset, how's it going? Thanks for reporting this!

I noticed there aren't automated tests specifically for pickle. I did some manual tests in the past, and it worked fine back then. However, a lot has changed internally, so this might be a regression.

State machines are meant to be serializable, but I'd love to understand your use case better. Could you share what you're trying to achieve by serializing the state machine? I'm asking because maybe an approach like this could be good enough.

I'll set up an automated test to reproduce the issue. Do you happen to have a complete example that demonstrates the problem?

Looking forward to your reply!

[fgmacedo]

Hi @emptierset, how are you? I"ve updated the code. Can you please check again?

[emptierset]

Out of curiosity, why do you need to mess with things like rtc and engine? Won't those be unpickled properly by the default pickling logic?

[emptierset]

Out of curiosity, why do you need to mess with things like rtc and engine? Won't those be unpickled properly by the default pickling logic?

[fgmacedo]

The engine struggled to be pickled because it contains a non-serializable Lock object (used to avoid race condition problems when utilizing multithreading). I also considered the option of removing the Lock object before copying, but I ultimately decided it was simpler to recreate the object as a "closed box." Since the StateMachine init parameter rtc is only stored in the engine, I needed to store its value in the state dictionary to facilitate recreation.

[fgmacedo]

Also, non rtc is deprecated and will be removed on the next major version so that this code can be simplified in the future.

Did you get the chance to validate your use case with this version?

[emptierset]

Thanks for your patience.

I noticed that #500 got merged -- can I use the develop branch instead? I tried pip install git+https://github.com/fgmacedo/python-statemachine.git@macedo/develop, but it didn't seem to work. Would you expect me to be able to clone that branch?

[emptierset]

Oh, it's just develop. I didn't grok that that was part of the branch name.

[emptierset]

We are running into other issues (like the one described in the OP) that make me feel like callbacks are not happening as expected, but I can't seem to produce an object with missing callbacks, like I was able to before. So it may not be a pickling issue (but I am still suspicious 🤷 ).

Do you anticipate a new version sometime soon with the pickling logic? Overriding it myself in my own object is nontrivial because it seems you also did some refactoring work that renames things. Or, is develop sufficiently stable that I can just depend on that?

[fgmacedo]

Hi @emptierset , glad to know it worked (ish :)

Since I only merge green PRs, develop is relatively stable but still a development playground.

I can release a new version if it helps. I'm working on experimental features that can wait for another release.

[emptierset]

Oops, sorry for not continuing the thread. UX was not obvious. Thank you for your help here.

[fgmacedo]

Hi @emptierset , just published 2.5.0! 🎉

[emptierset]

Amazing! Thank you. We've been using it for a few days and I think our app no longer has the bug I described. Don't want to jinx it, though, lol.

🙏