Question - Serverless Integration with a generic stateful endpoint.

[jamesvillarrubia]

Really interested in this library, but, as I understand it, it maintains state as property within the Class object. This is fine server-side, if I can share that Class object across multiple requests, but becomes a multiplicative problem in a high-concurrency, stateless environment like AWS Lambda.

1. Is there a pattern you recommend for serverless environments or sharing object state across containers that I'm not seeing in the docs?

2. If not, would that be a plugin/addition/alternative that the nodeshift team would be interested in?

[lance]

@jamesvillarrubia I've also been thinking about this module in a stateless environment. As you mention, in an environment where instances are constantly being spun up/down, state doesn't really make much sense. It almost seems as though there should be some kind of proxy service that all of the stateless functions are pointing at. In this proxy service is where the circuit breaker lives. Of course, that doesn't help when the proxy is down. Functions, being inherently stateless, just can't maintain any information about the status of the remote endpoint - or at least not enough to be of much use.

Not sure this helps much...

If not, would that be a plugin/addition/alternative that the nodeshift team would be interested in?

I can't speak for the team any more, but if you want to elaborate a bit on what you are thinking that would help.

[mlevkovsky]

@lance hopping on this thread.
It would be cool if we could have a feature where we can configure a remote store for the state of the CB. For example, https://github.com/gunnargrosch/circuitbreaker-lambda, allows the user to configure dynamodb to maintain the state of the circuit breaker.
Ideally, if you pass in some sort of flag, (a connection string and key for example) then this package would know to go retrieve it's information from a remote source.

[jamesvillarrubia]

@mlevkovsky Yes that's exactly what I was imagining. We just ported that same library you linked to firestore in GCP, but a generic storage API (with some adapters) would make me switch back to opossum, with all nice bells and whistles that it has.

[lholmquist]

@mlevkovsky interesting concept. I would want to try and keep the main Opossum module as stateless as possible, so perhaps this is something that either plugins into opossum or that opossum can send its info too. We've done something similar with our opossum-promethesus module, https://github.com/nodeshift/opossum-prometheus .

[mlevkovsky]

@lholmquist that makes sense. Thanks for linking me to that repo! much appreciated.

@jamesvillarrubia Further more, I actually made an interesting discovery about lambda, that lets opossum works right out of the box.

If you define something outside of the handlers in lambdas, AWS will “freeze” those resources between invocations (more information here). For an often used lambda, this means, that you can essentially, sort of be stateful.
So if you initialize your circuit breaker outside of the handle method, then this will work just fine. I tested it today and it kept the state between lambda runs.

[github-actions]

This issue is stale because it has been open 30 days with no activity.

[SethThomas]

Landed on this thread with the same use-case outlined by @jamesvillarrubia. I am implementing a circuit breaker pattern in a serverless environment to control access to 3rd party APIs. I'd prefer not to roll my own implementation, as Opposum looks like a feature-rich solution that I'd prefer to use.

The circuit-breaker lambda example provides one way this could be implemented in a serverless environment. While this is a great example of how this can be done, it tightly couples the persistence layer with the circuit breaker mechanism.

Adding a persistence layer to this project seems outside of it's scope. I personally don't mind writing my own wrapper around opossum to persist state in a way that makes sense in my environment (e.g. DynamoDB, Redis, Postgres, etc). In order for this to be an option, Opposum would need to provide a way to initialize the internal state. I see we can get the circuit breaker status, which I could persist in my environment. However, I don't see a mechanism to re-initialize the same state.

I haven't done a deep dive into this project yet, but does that functionality to initialize the internal state currently exist?

[lance]

I haven't done a deep dive into this project yet, but does that functionality to initialize the internal state currently exist?

It does not. I looked into this a couple of months ago but have not done anything to address it, since my work on this project is now mostly part-time, and I haven't had the time to do much more than look into what it would take. We do, of course, accept pull requests.

[lance]

@SethThomas fwiw, I think your approach sounds solid, and you are correct that a persistence layer is outside of the scope of opossum. Providing an API to support state initialization is the key.

[mlevkovsky]

@SethThomas I know that AWS keeps some information preloaded to avoid the slow cold boot issue.
If you initialize your circuit breaker, outside of the function and in the global context, then you actually won't lose the cb state between executions and it will be re-used.
I tested this myself and it seemed to work just fine.
Check this article

[SethThomas]

@mlevkovsky This is a good thing to keep in mind, and certainly can help improve performance. However, be mindful that Lambda provides no guarantee how long the execution environment will stay "warm" between executions. If your application relies on CB state from a prior lambda execution, you'll probably want to manage the persistence yourself.

[mlevkovsky]

@SethThomas yes 100%
Thankfully it was a case where, it's ok if the lambda loses the state of the cb and I was able to afford that.
The thing about using an external resource, is that now you have to potentially have a circuit breaker for that external dependancy 😅

[lholmquist]

i've been working on an "import" for the stats that I'll create a PR for later today

[lholmquist]

I started a draft PR here: #568

[github-actions]

This issue is stale because it has been open 30 days with no activity.

[lholmquist]

Closing this since this functionality was just released in 6.2.0, https://www.npmjs.com/package/opossum/v/6.2.0