Revisit changes to the `Signal` lifetime semantics to improve tooling interoperability. #355

andersio · 2017-04-30T11:14:17Z

Rationale

Cease the self reference in AliveState which causes the Signal and all its retained objects being reported as leaks, so that ReactiveSwift would work better with the memory debuggers.

Details

This PR adapts the original design in ReactiveCocoa/ReactiveCocoa#2959.

To recap, the rule of automatic disposal is having no observer and having no external retain. This PR does not change this principle, but makes a few changes in the implementation:

A signal is now broken into a public shell Signal and an internal core Signal.Core.
The shell is what the public users retain and interact with. The shell is neither strongly or weakly referenced by the core, the generator disposable or any disposable created.
The generator observer now holds a strong reference to Signal.Core, and pipes events to Signal.Core.send(_:). Previously, it holds a weak reference to Signal.

Here is the ownership graph from the perspective of an operator. Note that there is no circular strong reference in the graph as we had before.

------------               --------------                --------
|          |               | endObserve |                |      |
|          | <~~ weak ~~~  | disposable | <== strong === |      |
|          |               --------------                |      | ... downstream(s)
|          |                ------------                 |      |
| upstream | === strong ==> | Observer |  === strong ==> | Core |
------------                ------------                 --------
                            ------------------              ^^
                            | Signal (shell) | === strong ==//
                            ------------------

These changes consequently lead to a few boundary cases that is not source compatible with ReactiveSwift 1.0:

A nil weak reference to a signal no longer means that the signal has been disposed of. on(disposed:) would become the only mean to assert this.
If the generator observer deinitialises, it now triggers the automatic disposal since no event can be sent to the Signal from this point of time. Previously, it has no side-effect.

In other words, the rule of automatic disposal is now expanded to be (1) having no observer and having no external retain; OR (2) having the generator observer deinitialized".

These are not expected to be of huge impact.

Implementation

The threading design and the logic of event delivery is generally untouched.

The implementation in Signal is moved to Signal.Core. Signal now only forwards calls to Signal.Core.
Signal.Core.init now disposed of the generator disposable immediately, if the signal is terminated within the generator. This was the correct and the original behavior, but had not been covered by the test cases and had been removed by previous changes. (Bugfix)
tryTerminate is now promoted to be Signal.Core.tryCommitTermination.
Signal.Core.tryTerminateSilently is introduced for automatic disposal. Calling it guarantees the Signal to be eventually terminated safely, but the termination may not happen immediately if there is an on-going event delivery.
Signal.Core.shellDidDeinitialize is introduced for the Signal shell to inform the Signal core of its deinitialization.

Alternatives considered

Replace the explicit self reference with manual retains and releases via Unmanaged. This did not work as expected — the memory debuggers still report leaks.
Status quo, which was the alternative of this more complicated design in [5.0] Changes to the lifetime semantic of Signal. ReactiveCocoa#2959.

Testing

This has been tested with a very barebone single UITableView project that had a tremendous amount of false positives from the memory debugger when compiled with master. With the PR, the memory debugger reports no leak.

Edit: This has also been tested with Carthage 0.22, and all test specs have passed.

Potential benefits

The design might enable an optimisation in property composition to dynamically collapse intermediate signals. Specifically in #340, it might allow the observers of the composed property's relay signal to be collapsed into the upstream signal when it is not externally retained. This is not possible at this moment due to the self reference and the semantic of isKnownUniquelyReferenced.

NachoSoto · 2017-04-30T14:55:04Z

Tests/ReactiveSwiftTests/SignalSpec.swift

+				expect(disposable.isDisposed) == true
+			}
+
+			it("should dispose of the returned disposable upon interrupted") {


mdiep · 2017-05-01T19:58:03Z

A nil weak reference to a signal no longer means that the signal has been disposed of. on(disposed:) would become the only mean to assert this.

I'll have to think about the impact of this.

If the generator observer deinitialises, it now triggers the automatic disposal since no event can be sent to the Signal from this point of time. Previously, it has no side-effect.

In other words, the rule of automatic disposal is now expanded to be (1) having no observer and having no external retain; OR (2) having the generator observer deinitialized".

Can you explain this a bit more? I haven't looked over the code, but I'm not sure when this would happen.

andersio · 2017-05-01T22:23:10Z

The rule of active observers keeping the Signal alive remains unchanged, but the means to achieve it has changed:

In 1.x, it is done via the self reference in AliveState. The self reference is cleared when the observer bag is emptied, which triggers the deinitialization that leads to the disposal of the generator disposable.
- Bag emptied => Clear last strong +1 => Signal deinit => Disposal
In this PR, the self reference is replaced by the strong +1 of the generator observer. It is assumed to be retained by the owner or the upstream. For derived signals, it is also assumed that the generator disposable would be responsible of removing the generator observer from the upstreams (as it is used to be).

The Signal.Core disposes of its generator disposable when shellDeinitialized is asserted and the observer bag is emptied. This would indirectly leads the upstream, if any, to remove the generator observer from its bag. As the generator observer eventually deinitializes, the last strong +1 on the Signal.Core ceases.
- Bag emptied & Shell deinitialized => Disposal => Last strong +1 gone => Signal.Core deinit

In other words, the rule of automatic disposal is now expanded to be (1) having no observer and having no external retain; OR (2) having the generator observer deinitialized".

Can you explain this a bit more? I haven't looked over the code, but I'm not sure when this would happen.

This change does not affect operators, since they have their generator observer retained by the upstream signals. Only the explicit, non-composition uses of Signal.init and Signal.pipe would be affected.

Say we have a pipe and we have observed it:

func scope() {
    let (signal, observer) = Signal.pipe()
    signal.observe { _ in }
}

In 1.x, the self reference asserted by the sole active observer would keep the Signal alive.

With this PR, the Signal.Core would lose all both of its retains as scope() returns, so the signal would dispose of itself. This is a new but IMO logical behaviour, given that the missing generator observer means no event can be further sent statically.

We may consider emitting interrupted since there are still active observers in this case. But for now it is silent.

mdiep · 2017-05-16T13:56:26Z

Can you add the external observers to your diagram above? I think that's the tricky piece to think about.

Would it work to replace the self reference with an actual strong reference from the observer to the signal? That seems like it would maintain the existing semantics (which are conceptually simpler IMO) but also fix the tooling issues.

A nil weak reference to a signal no longer means that the signal has been disposed of. on(disposed:) would become the only mean to assert this.

I think it's preferable to not have this be the case, which is why I ask. But if this the only way to improve tooling interop, then I think it's worth it.

andersio · 2017-05-17T04:41:38Z

Would it work to replace the self reference with an actual strong reference from the observer to the signal? That seems like it would maintain the existing semantics (which are conceptually simpler IMO) but also fix the tooling issues.

It does not solve the issue of deliberate retain cycle being caught with the current implementation.

Specifically, if an observer now holds a strong reference to the Signal, it forms a legit retain cycle. We cannot break the cycle with a weak reference from the Signal to the observers, as the Signal retaining observers is pragmatic.

 ------------               --------------                --------
 |          |               | endObserve |                |      |
 |          | <~~ weak ~~~  | disposable | <== strong === |      |
 |          |               --------------                |      | ... downstream(s)
 | Upstream |                ------------                 |      |
 | Core     | === strong ==> | Observer |  === strong ==> | Core |
 ------------ ===\\          ------------                 -------- ===\\
                  \\         ------------------              ^^        \\
                   \\        | Signal (shell) | === strong ==//         \\
                    \\       ------------------                          \\
                    || strong                                            || strong
                    vv                                                   vv
            -------------------                                 -------------------
            | Other observers |                                 | Other observers |
            -------------------                                 -------------------

andersio · 2017-05-17T05:02:48Z

A side note: I have tried many tricks, hoping to trick the memory debugger not to recognise the retain cycle. But none of these works.

A Unmanaged pointer.
A UnsafeMutablePointer pointer.
An OpaquePointer.
Storing a UnsafeMutablePointer as a raw Int.
Storing a UnsafeMutablePointer as a raw Int that is right shifted by 1 bit.
This does "break" the retain cycle in the debugger, but the debugger still considers it a leak.

mdiep

Finally wrapped my head around this.

A few requests that I think will improve clarity, but overall the approach seems 👍.

mdiep · 2017-05-22T12:41:46Z

Sources/Signal.swift

-					signal.state = .terminated
-					signal.updateLock.unlock()
+		/// Used to indicate if the `Signal` shell has deinitialized.
+		private var shellDeinitialized: Bool


I think we should use signal instead of shell here and in the related methods. I think that's clearer.

mdiep · 2017-05-22T12:48:16Z

Sources/Signal.swift

+				withExtendedLifetime(observer) {
+					// Dispose of the `Signal` core if the shell has deinitialized
+					// and it has no active observer.
+					result = tryTerminateSilently(if: state.observers.isEmpty && shellDeinitialized)


The condition (state.observer.isEmpty && shellDeinitialized) should move into tryTerminateSilently. It's the always the same logic. (In the other case, we know that shellDeinitialized is true, but it doesn't hurt to explicitly check it.)

mdiep · 2017-05-22T12:50:19Z

Sources/Signal.swift

+				// released to avoid deadlocks.
+				withExtendedLifetime(observer) {
+					// Dispose of the `Signal` core if the shell has deinitialized
+					// and it has no active observer.


Can you clarify that this comment only starts the disposal?

mdiep · 2017-05-22T12:50:47Z

Sources/Signal.swift

-			observer.sendCompleted()
-			return nil
+				if result == .shouldDispose {
+					disposable.dispose()


Can you add a comment here clarifying that this completes the disposal by removing the last strong reference to the core?

mdiep · 2017-05-22T12:58:51Z

Sources/Signal.swift

+				return .none
+			}
+
+			defer { _sendLock?.unlock() }


I think this a bit confusing as written. I think we should go with a more straightforward version that's self explanatory:

if sendLock == nil { if !self.sendLock.try() { return .none } defer { self.sendLock.unlock() } }

mdiep · 2017-05-22T12:59:37Z

Sources/Signal.swift

+		///               `tryTerminate` would attempt to acquire the `sendLock`.
+		///
+		/// - returns: `.shouldDispose` if the attempt succeeds. `.none` otherwise.
+		private func tryCommitTermination(acquired sendLock: Lock? = nil) -> OperationResult {


Can you add to after try in this method and the one below? tryToCommitTermination

mdiep · 2017-05-22T13:10:18Z

Sources/Signal.swift

-			}
+			shellDeinitialized = false
+			disposable = SerialDisposable()
+			disposable.inner = generator(Observer(self.send))


Maybe add a comment here to highlight that it's the other +1?

mdiep · 2017-05-22T16:24:09Z

That test failure looks like it might be legit.

andersio · 2017-05-22T16:27:02Z

MutableProperty should not deadlock occasionally fails on single-core CI. 🙈

andersio added the api breaking label Apr 30, 2017

andersio force-pushed the signal-self-retain branch 2 times, most recently from 5beb43f to df7f83c Compare April 30, 2017 12:02

NachoSoto reviewed Apr 30, 2017

View reviewed changes

andersio force-pushed the signal-self-retain branch from e825a09 to e47b6d6 Compare May 1, 2017 17:24

andersio modified the milestone: 2.0 May 5, 2017

andersio mentioned this pull request May 15, 2017

Avoid the active signal from being retained in the race strategy. #386

Merged

andersio added 3 commits May 15, 2017 21:40

Drop the self reference in Signal with modified lifetime semantics.

26c1e19

Update test cases for the modified signal lifetime semantics.

c79ad21

Add new test cases for Signal.init.

8d9e128

andersio force-pushed the signal-self-retain branch from e47b6d6 to 8d9e128 Compare May 15, 2017 13:43

andersio mentioned this pull request May 15, 2017

Generic n-ary producer operators for n > 1. #351

Merged

mdiep requested changes May 22, 2017

View reviewed changes

andersio added 2 commits May 22, 2017 23:41

Addressed feedbacks.

07e46ad

Merge branch 'master' into signal-self-retain

49f9a03

andersio requested a review from mdiep May 22, 2017 15:58

mdiep approved these changes May 23, 2017

View reviewed changes

mdiep merged commit 1fa0a89 into master May 23, 2017

mdiep deleted the signal-self-retain branch May 23, 2017 02:04

andersio mentioned this pull request Jun 13, 2017

Fixed an impedance mismatch between Signal.Core RCU and Bag. #449

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revisit changes to the `Signal` lifetime semantics to improve tooling interoperability. #355

Revisit changes to the `Signal` lifetime semantics to improve tooling interoperability. #355

andersio commented Apr 30, 2017 •

edited

Loading

NachoSoto Apr 30, 2017

mdiep commented May 1, 2017

andersio commented May 1, 2017 •

edited

Loading

mdiep commented May 16, 2017

andersio commented May 17, 2017 •

edited

Loading

andersio commented May 17, 2017 •

edited

Loading

mdiep left a comment

mdiep May 22, 2017

mdiep May 22, 2017

mdiep May 22, 2017

mdiep May 22, 2017

mdiep May 22, 2017

mdiep May 22, 2017

mdiep May 22, 2017

mdiep commented May 22, 2017

andersio commented May 22, 2017

Revisit changes to the Signal lifetime semantics to improve tooling interoperability. #355

Revisit changes to the Signal lifetime semantics to improve tooling interoperability. #355

Conversation

andersio commented Apr 30, 2017 • edited Loading

Rationale

Details

Implementation

Alternatives considered

Testing

Potential benefits

Choose a reason for hiding this comment

mdiep commented May 1, 2017

andersio commented May 1, 2017 • edited Loading

mdiep commented May 16, 2017

andersio commented May 17, 2017 • edited Loading

andersio commented May 17, 2017 • edited Loading

mdiep left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdiep commented May 22, 2017

andersio commented May 22, 2017

Revisit changes to the `Signal` lifetime semantics to improve tooling interoperability. #355

Revisit changes to the `Signal` lifetime semantics to improve tooling interoperability. #355

andersio commented Apr 30, 2017 •

edited

Loading

andersio commented May 1, 2017 •

edited

Loading

andersio commented May 17, 2017 •

edited

Loading

andersio commented May 17, 2017 •

edited

Loading