Lazy Susan: Implement lazy-load and future CanBind support in AJAX responses

[Shadowfiend]

This PR really does three things:

• Reworks the LazyLoad snippet and separates out the AsyncRenderComet to its own
  file. This refactoring allows the LazyLoad snippet to work even if you're rendering an AJAX
  response, thanks to the fiery comets work that went in a few weeks ago.
• Adds a CanResolveAsync type class that can be used to handle any object that can be
  resolved to a value asynchronously in the same way. Right now, two CanResolveAsync
  implicits are provided: one for Scala Future and one for LAFuture.
• Adds a CanBind for any type that has a CanResolveAsync implicit defined for it. This
  uses some of the helper functions exposed by AsyncRenderComet to handle asynchronously
  sending down the results of resolving the object in question.

Concretely, this means four big things:

• LazyLoad now works correctly even if it's rendered during an AJAX response. This wasn't
  the case in Lift 2.x.
• You can use Scala Future and LAFuture on the right-hand-side of a CSS selector transform
  out-of-the-box. Like @fmpwizard's original implementation and unlike my dirt-simple one, this
  does not waste a thread trying to resolve the futures.
• Those futures can be used even if rendering during an AJAX response. I know my implementation
  couldn't do this, I didn't check if Diego's could.
• Support for additional future-like types (for example, Twitter Futures as used by Finagle) is
  just a matter of providing a CanResolveAsync implicit. Their implementations are incredibly
  simple (they are simply abstractions over onSuccess or whatever equivalent is provided).

If you want to see this in action, check out the lift_30 branch of my lift-future-canbind-example repository. It mostly keeps the same code as the original example I posted a few days ago on the list, but uses Lift 3's built-in CanBind support for futures. It also demonstrates futures and lazy loads continuing to work even when re-rendered via an idMemoize.

I've also created a failing-lift-2 branch in that repo that has basically the same code as the lift_30 branch, but uses Lift 2.6 and the original FutureBinds CanBinds. This illustrates the non-functioning AJAX lazy loading/future handling.

[andreak]

This is way cool!

I have some suggestions to make it even more useful.

I often have the need for some custom execution-context and have gotten very used to LoadWrappers for providing this. I have my custom version of LazyLoadSnippet with support for LoanWrappers, with the following signature:
def render(xhtml: => NodeSeq, placeholderNS: NodeSeq, lw: List[LoanWrapper]): NodeSeq

It builds the deferredFunction like this:
val func: () => JsCmd = session.buildDeferredFunction(() => around(lw)(Replace(id, xhtml))

Where around is like in SpecializedLiftActor:

private def around[R](lw: List[LoanWrapper])(f: => R): R = lw match {
    case Nil => f
    case xs => CommonLoanWrapper(xs)(f)
}

The cool thing about it is that I can use it for programatically installing lazy-load zones where evaluation of the argument is lazy, meaning all computation is done async:

".myZone *" #> ((ns: NodeSeq) => 
  OrigoLazyLoad.render(someCssSelFunc(ns), <div>Loading content...</div>, LoanWrapperHelper.getDefaultClientLoanWrapper)
)

It would be cool if something like this went into Lift-3, shall I add something similar to the new LazyLoadSnippet, or will you?

[Shadowfiend]

Hmm… I think I'm getting it, but I'd like a more concrete example to grasp the use case more clearly. What kinds of things do you put in your loan wrappers in these cases?

It seems like that functionality would best be pushed all the way into buildDeferredFunction rather than LazyLoad or AsyncRenderComet (though having helpers there that pass the appropriate stuff on into buildDeferredFunction is an option, of course). But first I want a better feel for what's going on—I haven't really used LoanWrapper much, so I'm less familiar with how folks are using it.

On pertinent question that comes up is, should buildDeferredFunction itself use a LoanWrapper to do what it's currently doing?

[andreak]

The use-case is that I often want some parts of the page to load async, that given. Sometimes these parts are part of an AJAX-response, so I have a custom version OrigoLazyLoad which works for AJAX-responses in 2.6, which this PR obsoletes.

What my OrigoLazyLoad also does is provide an option to pass a list of LoanWrappers as the last argument. I use these to set SpringSecurityContext thread-local vars, log4j MDC(Mapped Diagnostic Context) and other custom stuff (also ThreadLocals) which my services (which provides data for the snippet to render) require.
It's also important for me that the first argument (xhtml: NodeSeq) is by name so it's evaluated lazily, in my prev example someCssSelFunc(ns) is then evaluated async, in the same thread as the func is executed. This of course is more useful when calling OrigoLazyLoad progamatically as the RHS of a CssSelector Transform (which is the only way I'm using it). Using LoanWrapper this way is analogous to the way SpecializedLiftActor uses loan-wrappers, which I use in all my comet-actors, like this:

trait OrigoCometActor extends CometActor with CometListener with Loggable {
    lazy val clientContext = theSession.httpSession.openOrThrowException("andreak: httpSession should always return Full in comet-actor").attribute(Keys.CLIENT_CONTEXT).asInstanceOf[ClientContext]
    lazy val messages = LiftRules.context.attribute("uno.messages").openOrThrowException("No uno.messages found in servlet-context").asInstanceOf[MessageResources]
    var authentication: Box[Authentication] = Empty
// .......
//  .......
override protected def aroundLoans: List[CommonLoanWrapper] = {
    new LoanWrapper  {
        def apply[T](f: => T) : T =  {
            authentication.foreach(auth => SecurityContextHolder.getContext.setAuthentication(auth))
            ClientContext.setInstance(clientContext)
            val userName = clientContext.getOrigoPerson.getUser.getUserName
            ThreadLocalContext.setUserName(userName)
            ThreadLocalContext.setMessages(messages)
            MDC.put(Constants.LOG_MDC_USERNAME_KEY, userName)
            ?(clientContext.getRequestKey).foreach(MDC.put(Constants.LOG_MDC_REQUEST_KEY, _))
            //              debug("Message received")
            Constants.logCreatedMDC(log)
            try {
                f
            }
            finally {
                Constants.logRemovingMDC(log)
                MDC.clear()
                SecurityContextHolder.clearContext()
                ClientContext.reset()
                ThreadLocalContext.reset()
            }
        }
    } :: Nil
}

I think pushing LoanWrapper all the way into buildDeferredFunction, as an optional arg, is a great idea!

I also prefer placeholderNS (the NodeSeq to render which waiting for the data) to be an explicit argument.

Let me know if the use-case still isn't clear.

Lift-3 really shines with its improved AJAX/async-support, which we use quite heavily but had to hack a bit to accomplish in pre-3.

[Shadowfiend]

Are the same LoanWrappers also used in LiftRules.allAround? Would it make sense for buildDeferredFunction to wrap its deferred functions in LiftRules.allAround? Would that solve the issues without requiring additional parameters? Trying to explore the solution space a bit to see if we can avoid make the API more complex while still providing the functionality in a way that makes sense :)

As a side note, I'll be replying to comments/thinking about solutions, but I probably won't circle back to make code changes until the end of the week.

EDIT: My thinking on allAround, btw, is that the purpose of buildDeferredFunction is to execute the function in as close to the same context as when it was deferred as possible—so it seems like maybe that should be within the LoanWrappers as well.

[andreak]

No, the list of LWR are not the same as in allAround and do vary depending on what kind of work is getting done async (calling some WebServices require other context etc.), and is often not the same as what's being done in the HTTP-thread.

[Shadowfiend]

Got it. Perhaps something like def buildDeferredFunction[T](f: () => T, loanWrappers: List[LoanWrapper] = LiftRules.allAround): () => T = { where we default to LiftRules.allAround but it can be replaced/revised?

[andreak]

Excellent suggestion!

[Shadowfiend]

Cool. I'll be seeing @dpp the next couple of days so I'll probably pick his brain some more on LoanWrappers and whether it would make sense for deferred functions to be wrapped in allAround by default.

[Shadowfiend]

Okay, chatted with David a bit about this… It seems like for now your use case is still a bit unique to add to this API. buildDeferredFunction already runs LiftRules.allAround, so that part is already taken care of as it is. If the client code wants to run additional LoanWrappers, that can happen in the function passed to AsyncRenderComet.asyncRender, and they will in turn be wrapped in a deferred function. That should let you cut down your custom lazy-load comet to something fairly simple.

To simplify it further, I'm thinking of providing an overload of the LazyLoad singleton's render method that takes a function; that function will be run instead of the Replace stuff here: https://github.com/lift/framework/pull/1607/files#diff-bd3940c5f8f059b04a9dde01e1a52a8fR67 . That will mean that we'll run the function you pass in and send its results down to the client; the JsCmd that your function produces would be in charge of replacing the placeholder. Signature could be:

  def render(xhtml: NodeSeq, renderer: (String)=>JsCmd): NodeSeq

The String passed to the renderer would be the id of the placeholder element.

Thoughts?

[andreak]

Hm, I was hoping to avoid having to make any custom LazyLoad object.

I'm quite surprised that my use-case is thought of as a bit unique.

For me these things are important API wise:

1. The xhtml passed to render is by name so it's evaled lazily, in the async deferred function
2. I must be able to pass a list of LoanWrappers
3. I mus be able to pass the placeholder-NodeSeq as an argument, not a context-var in S.attr

AFAICS what you propose makes none of this possible. However it does make creating a custom LazyLoad object much easier.

[Shadowfiend]

Sorry, yeah, I over-focused on the LoanWrapper stuff. I think making render take a call-by-name is something we can do now, and I think passing the placeholder is something we can do now. I think the LoanWrapper stuff is better left as a “you can pass your own deferred function” bit, so that if someone else wants to wrap things in something that isn't a LoanWrapper, we're not proscribing that in the API.

[andreak]

Nice!
I can live with passing my own deferred-function:-)

Rock on!

[Shadowfiend]

Okay, see if that works a little bit better.

[andreak]

Great, I'll check it out!

[andreak]

Works as expected!

My custom lazy-load snippet becomes much more readable now:

object OrigoLazyLoad {

    private def around[R](lw: List[LoanWrapper])(f: => R): R = lw match {
        case Nil => f
        case xs => CommonLoanWrapper(xs)(f)
    }

    def render(xhtml: => NodeSeq, placeholderTemplate: NodeSeq, lw: List[LoanWrapper]): NodeSeq = {
        S.session.map { session =>
            LazyLoad.render(session.buildDeferredFunction((domId: String) => around(lw)(Replace(domId, xhtml)))
                , Full(placeholderTemplate))
        }.openOr(Comment("Asynchronous rendering requires a session context."))
    }

}

I still think this should be a part of Lift so other people get this out of the box, but won't argue more about it.

[Shadowfiend]

Although I just realized, your around stuff runs inside the deferred function. That means we can make the render overload that takes a (String)=>NodeSeq handle the deferred function wrapping, and make the situation where people need to do their own wrapping a little harder (directly using AsyncRenderComet.asyncRenderDeferred). That would make your snippet look like:

object OrigoLazyLoad {

    private def around[R](lw: List[LoanWrapper])(f: => R): R = lw match {
        case Nil => f
        case xs => CommonLoanWrapper(xs)(f)
    }

    def render(xhtml: => NodeSeq, placeholderTemplate: NodeSeq, lw: List[LoanWrapper]): NodeSeq = {
      LazyLoad.render((domId: String) => around(lw)(Replace(domId, xhtml)), Full(placeholderTemplate))
    }
}

[andreak]

That'd be great!

[Shadowfiend]

Okies, there we go.

[andreak]

Nice!

BTW; Should't AsyncRenderComet's asyncRenderDeferred and asyncRender swap names, as asycnRender is the one that actually calls session.buildDeferredFunction?

[Shadowfiend]

I've been thinking about that myself. I'm a little torn—asyncRenderDeferred is saying that it's an async render for an already deferred function, while asyncRender is doing an async render and taking care of deferring it internally.

I'm not particularly satisfied with the names as they stand, but I haven't come up with better naming. The reverse is also confusing for me. Would love to hear some impressions from other folks.

[andreak]

I think it's "clean code" to name a method after what it's intention is, what it's doing. What asyncRender is doing is to process a function which it "defers" asynchronously, so IMO asyncRenderDeferred is cleaner. The name "asyncRender" implies that it's async, not that it's the caller's job to make it async...

[Shadowfiend]

Heh, well the trouble is both of them handle the asynchronous part of the render. buildDeferredFunction doesn't handle the asynchronicity, it handles the restoration of state whenever the function is ultimately invoked :/

[andreak]

Yes, I know. That's why I think asyncRenderDeferred is a better name for the one handling both the asynchronicity and restoration of state. Maybe asyncRenderSnapshotted is better, the state is snapshotted and then restored when run.

[Shadowfiend]

Maybe asyncDeferAndRender vs asyncRender? I agree about snapshotted, as well, and it's worth bringing that up separately on the list, since doing that rename properly would involve renaming buildDeferredFunction to buildSnapshottedFunction.

[andreak]

I don't think any of this naming-issues should hold this PR from getting merged. We should discuss on the list and open a separate PR I think.

[Shadowfiend]

So is that a 👍 ? :)

[andreak]

+1 for merge:-)

[Shadowfiend]

Anyone want to do the honors? O:-)

[andreak]

Rock and roll!