IPLD Transformations and Authenticated Pipelines

[nicola]

The idea is to have an authenticated "Dataflow" (a la Google).

In other words, imagine having some source of data and a pipeline, a series of transformations, ordered in a DAG. How can we make this authenticated so that we can safely distribute these pipelines?

Useful links:

• Apache Beam project
• Google Cloud Dataflow
• Dataconomy on Dataflow

Reading list:

• Stream 101
• Stream 102
• Spark vs Dataflow
• Dataflow pipelines
• Dataflow transforms
• Dataflow paper
• Processing on IOT

Authentication:

• IntegriDB

• Dont trust. Verify
• SMR work documented here: State Machine replication #28

Relevant to ipfs/specs#115 (IPLD transformations) and may need ipld/specs#4 (IPLD selectors)