[#152269178] Migrate ComputeVisibleServices to runtime v2 #2
Conversation
CreateService/handler.ts
Outdated
| updatedAt: new Date().getTime() | ||
| }; | ||
| const dfClient = df.getClient(context); | ||
| await dfClient.startNew("UpsertServiceOrchestrator", undefined, event); |
There was a problem hiding this comment.
I've found that relying on durable functions to serialize/deserialize parameters is too error prone - it's safer to explicitly encode/decode parameters with io-ts, see for example pagopa/io-functions-services#17
| visibleService: VisibleService | ||
| }); | ||
|
|
||
| const Input = t.union([AddVisibleServiceInput, RemoveVisibleServiceInput]); |
There was a problem hiding this comment.
you want a taggedUnion here to be able to encode the input from the orchestrator
There was a problem hiding this comment.
I was using union because taggedUnion is now deprecated (union is now able to detect and optimize tagged unions). The problem is that this new feature is available only from v1.9.0 (we are using v1.8.6 that is just before v1.9.0) so i am going to revert the code to use taggedUnion as you suggested.
| currentVisibleService !== undefined && | ||
| currentVisibleService.version >= visibleService.version | ||
| ) { | ||
| // A newer version is already stored in the blob, so skip the remove/update |
There was a problem hiding this comment.
when could this scenario happen? it looks like a conflict (another process updated this data concurrently), perhaps we need to error in this case?
| } | ||
| ); | ||
|
|
||
| // Map None to empty object |
There was a problem hiding this comment.
| // Map None to empty object | |
| // Default to an empty object when the blob does not exist yet |
| ): Promise<Either<Error, true>> { | ||
| // Retrieve the current visibleServices blob using the leaseId | ||
| const errorOrMaybeVisibleServices = await getBlobAsObject( | ||
| t.dictionary(ServiceId, VisibleService), |
There was a problem hiding this comment.
for clarity, give a name and declare this type outside this function
| if (isLeft(errorOrVisibleServices)) { | ||
| return left( | ||
| Error( | ||
| `UpdateVisibleServicesActivity|Cannot decode blob|ERROR=${errorOrVisibleServices.value}` |
There was a problem hiding this comment.
if these are validation errors, you can use readableReport to format them in a way that is suitable for logging
| VISIBLE_SERVICE_CONTAINER, | ||
| VISIBLE_SERVICE_BLOB_ID, | ||
| { | ||
| leaseDuration: LEASE_DURATION |
There was a problem hiding this comment.
what happens if this function takes more than LEASE_DURATION to release the lease?
are we protected from the scenarios described in "Protecting a resource with a lock" section from this article?
There was a problem hiding this comment.
Yes we are protected because when you WRITE you have to pass the leaseId, and if the leaseId is expired no write happens.
There was a problem hiding this comment.
@cloudify it is exactly like the "Making the lock safe with fencing" paragraph but the token is the leaseId.
utils/UpsertServiceEvent.ts
Outdated
| export const UpsertServiceEvent = t.intersection([ | ||
| t.interface({ | ||
| newService: RetrievedService, | ||
| updatedAt: t.number |
There was a problem hiding this comment.
you can use UTCISODateFromString to serialize/deserialize Dates
Affected stories
New dependencies added: azure-storageAuthor: Microsoft Corporation Description: Microsoft Azure Storage Client Library for Node.js Homepage: http://github.com/Azure/azure-storage-node
durable-functionsAuthor: kashimiz Description: Durable Functions library for Node.js Azure Functions Homepage: https://github.com/Azure/azure-functions-durable-js#readme
|
| Branch | Status |
|---|---|
| master | |
| dev |
Durable Functions for Node.js
The durable-functions npm package allows you to write Durable Functions for Node.js. Durable Functions is an extension of Azure Functions that lets you write stateful functions and workflows in a serverless environment. The extension manages state, checkpoints, and restarts for you. Durable Functions' advantages include:
- Define workflows in code. No JSON schemas or designers are needed.
- Call other functions synchronously and asynchronously. Output from called functions can be saved to local variables.
- Automatically checkpoint progress whenever the function schedules async work. Local state is never lost if the process recycles or the VM reboots.
You can find more information at the following links:
A durable function, or orchestration, is a solution made up of different types of Azure Functions:
- Activity: the functions and tasks being orchestrated by your workflow.
- Orchestrator: a function that describes the way and order actions are executed in code.
- Client: the entry point for creating an instance of a durable orchestration.
Durable Functions' function types and features are documented in-depth here.
Getting Started
You can follow the Visual Studio Code quickstart to get started with a function chaining example, or follow the general checklist below:
-
Install prerequisites:
- Azure Functions Core Tools version 2.x
- Azure Storage Emulator (Windows) or an actual Azure storage account (Mac or Linux)
- Node.js 8.6.0 or later
-
Create an Azure Functions app. Visual Studio Code's Azure Functions plugin is recommended.
-
Install the Durable Functions extension
Run this command from the root folder of your Azure Functions app:
func extensions install -p Microsoft.Azure.WebJobs.Extensions.DurableTask -v 1.7.0durable-functions requires Microsoft.Azure.WebJobs.Extensions.DurableTask 1.7.0 or greater.
- Install the
durable-functionsnpm package at the root of your function app:
npm install durable-functions- Write an activity function (see sample):
module.exports = async function(context) {
// your code here
};- Write an orchestrator function (see sample):
const df = require('durable-functions');
module.exports = df.orchestrator(function*(context){
// your code here
});Note: Orchestrator functions must follow certain code constraints.
- Write your client function (see sample):
module.exports = async function (context, req) {
const client = df.getClient(context);
const instanceId = await client.startNew(req.params.functionName, undefined, req.body);
context.log(`Started orchestration with ID = '${instanceId}'.`);
return client.createCheckStatusResponse(context.bindingData.req, instanceId);
};Note: Client functions are started by a trigger binding available in the Azure Functions 2.x major version. Read more about trigger bindings and 2.x-supported bindings.
Samples
The Durable Functions samples demonstrate several common use cases. They are located in the samples directory. Descriptive documentation is also available:
- Function Chaining - Hello Sequence
- Fan-out/Fan-in - Cloud Backup
- Monitors - Weather Watcher
- Human Interaction & Timeouts - Phone Verification
const df = require("durable-functions");
module.exports = df.orchestrator(function*(context){
context.log("Starting chain sample");
const output = [];
output.push(yield context.df.callActivity("E1_SayHello", "Tokyo"));
output.push(yield context.df.callActivity("E1_SayHello", "Seattle"));
output.push(yield context.df.callActivity("E1_SayHello", "London"));
return output;
});How it works
Durable Functions
One of the key attributes of Durable Functions is reliable execution. Orchestrator functions and activity functions may be running on different VMs within a data center, and those VMs or the underlying networking infrastructure is not 100% reliable.
In spite of this, Durable Functions ensures reliable execution of orchestrations. It does so by using storage queues to drive function invocation and by periodically checkpointing execution history into storage tables (using a cloud design pattern known as Event Sourcing). That history can then be replayed to automatically rebuild the in-memory state of an orchestrator function.
Read more about Durable Functions' reliable execution.
Durable Functions JS
The durable-functions shim lets you express a workflow in code as a generator function wrapped by a call to the orchestrator method. orchestrator treats yield-ed calls to your function context's df object, like context.df.callActivity, as points where you want to schedule an asynchronous unit of work and wait for it to complete.
These calls return a Task or TaskSet object signifying the outstanding work. The orchestrator method appends the action(s) of the Task or TaskSet object to a list which it passes back to the Functions runtime, plus whether the function is completed, and any output or errors.
The Azure Functions extension schedules the desired actions. When the actions complete, the extension triggers the orchestrator function to replay up to the next incomplete asynchronous unit of work or its end, whichever comes first.
Generated by 🚫 dangerJS
Codecov Report
@@ Coverage Diff @@
## master #2 +/- ##
=========================================
+ Coverage 85.84% 86.5% +0.65%
=========================================
Files 6 7 +1
Lines 106 126 +20
Branches 7 8 +1
=========================================
+ Hits 91 109 +18
- Misses 15 17 +2 |
|
@cloudify code updated.
and the blob is always correctly updated to the last version. |
|
@francescopersico excellent, thank you! |
CreateService/handler.ts
Outdated
|
|
||
| const createdService = errorOrCreatedService.value; | ||
|
|
||
| const errorOrUpsertServiceEvent = UpsertServiceEvent.decode({ |
There was a problem hiding this comment.
we should actually encode the typed object to an untyped one, this should be something like:
const upsertServiceOrchestratorInput = UpsertServiceEvent.encode({
newService: createdService,
updatedAt: new Date()
});
const dfClient = df.getClient(context);
await dfClient.startNew(
"UpsertServiceOrchestrator",
undefined,
upsertServiceOrchestratorInput
);There was a problem hiding this comment.
Done also for the Activity input.
CreateService/handler.ts
Outdated
|
|
||
| const upsertServiceEvent = UpsertServiceEvent.encode({ | ||
| newService: createdService, | ||
| updatedAt: new Date().getTime() |
There was a problem hiding this comment.
we usually encode dates as UTC ISO strings, you should use UTCISODateFromString in the definition of UpsertServiceEvent (instead of t.integer)
There was a problem hiding this comment.
Repository io-functions-app is using t.integer for events https://github.com/teamdigitale/io-functions-app/blob/master/utils/UpdatedProfileEvent.ts#L14.
I am going to change this PR to use UTCISODateFromString anyway.
There was a problem hiding this comment.
thanks for noticing, can you also fix the functions app? thanks
| } | ||
| ); | ||
|
|
||
| const event = UpsertServiceEvent.encode({ |
There was a problem hiding this comment.
for consistency
| const event = UpsertServiceEvent.encode({ | |
| const upsertServiceEvent = UpsertServiceEvent.encode({ |
UpdateService/handler.ts
Outdated
| retrievedServiceToApiService(maybeUpdatedService.value) | ||
| const updatedService = maybeUpdatedService.value; | ||
|
|
||
| const errorOrUpsertServiceEvent = UpsertServiceEvent.encode({ |
There was a problem hiding this comment.
| const errorOrUpsertServiceEvent = UpsertServiceEvent.encode({ | |
| const upsertServiceEvent = UpsertServiceEvent.encode({ |
cloudify
deleted the
152269178-migrate-computevisibleservices-to-runtime2
branch
No description provided.