0.10.0

Release notes

Alongside yagna 0.12, we’re releasing a new version - 0.10 - of yapapi, Golem’s Python API.
This new update comes with a set of both gradual improvements and important features that expand
the range of applications that can be developed on Golem and at the same time make the process
easier by adding to the toolset the developers have at their disposal.

New features

Experimental outbound network access

We’ve been aiming for some time to enable payloads to access external locations, knowing it would
open a vast world of new possibilities regarding the types of apps that can be built on Golem.
Therefore, one of the important features that yagna 0.12 and yapapi 0.10 introduce is the
experimental support for outbound internet access from Golem VM runtimes.

As we’re very consciously aware that giving access to all locations without any limits would make
Golem a dream-come-true platform for botnets of all kinds, the current release makes this feature
experimentally available in two flavours. Firstly, we’re adding a curated whitelist of locations
that the providers will have set up by default and which can be accessed by any VM payload.
Secondly, we’re enabling the VM images to be accompanied by application manifests that enable only
specific addresses to be accessed. Each of such manifests must be signed by the requestor's private
key and must carry a certificate that confirms the requestor has the privilege of running such apps on Golem.

Alongside the feature itself, comes a new example - external-api-request - that showcases the
latter type of usage.

This is the feature many of you have been asking about for a long time now and with it in place,
we cannot wait to see what kinds of new apps our community creators will be able to develop!

Generic TCP socket proxy

A few releases back, we have made the local http proxy available to enable HTTP-based services to be
accessed through a local port. In the meantime, it has become clear that other types of services
would also benefit from such access.

Consequently, in this release, we’re adding SocketProxy - a module that does exactly that, in
other words, it exposes a local port on a requestor’s machine and forwards all the traffic
to and from a remote port on a provider node through it.

In effect, any type of service - be it an SSH server, a remote SQL database or, say, redis
cache - as long as it’s a TCP service running inside a VM, can be very easily accessed through
a port on the requestor’s machine without any additional, intermediary tools.

We have also updated our SSH example to show the usage of this feature.

This does not yet enable direct access to those ports on the provider node’s IP addresses and
this is a feature we’ll make available in some future release.

Minor features

IP address removal and recycling

While our initial support for Golem VPN allowed a user to define the IP addresses of launched
services, it only allowed those addresses to be assigned once when a given service instance was
started.

Even though it works well in a happy-path scenario when each of the commissioned services launches
successfully the first time around, it becomes problematic when there is a need to restart an
instance, either because we want to modify it or just because it failed to initialise properly, and
we want to relaunch it on another provider node.

The new update enables the user to explicitly remove the IP address mapping for each of the nodes in
the network, but it also automatically frees all the IP addresses when instances are terminated.
Additionally, failing services with statically-assigned IPs are now able to be restarted correctly.

More convenient Service restart hooks

In the previous release, we have added the Service.reset hook that allowed the application authors
to define operations needed for Service to be brought back to an initial state when an instance
was about to be restarted.

That said, the engine didn't give the user a convenient way to define the exact scenarios
when instances should be restarted - the previous, respawn_condition hook was not easily
available outside the API itself.

Because of that, we have decided to define an explicit location to enable such customised behaviour,
namely the Service.restart_condition property. The default condition is consistent with the
previous, built-in behaviour of the engine which triggers a restart only if the instance it had not
been successfully started. Application authors can easily now override this default to provide
their own, tailored criteria for restart.

Customised debit note and invoice acceptance criteria

Up until now, the requestor agent always accepted the full amount specified in the invoices and
debit notes coming from the providers.

The new update adds the debit_note_accepted_amount and invoice_accepted_amount hooks to
MarketStrategy, enabling the requestor agent authors to define their own criteria and calculations
that determine whether incoming debit notes and invoices are accepted and whether they should be
accepted in full or partially.

Automatic service activity health-check

Previously, the requestor agent had no way of knowing that a given service stopped being available,
unless the launched Service instance actively interacted with the underlying activity running on the
provider node.

In this release, we have added a background task periodically querying the activity for its state.
Thanks to that, the requestor will be notified of a given service’s failure soon after it happens.
This allows the application authors to program recovery behaviour into their code more efficiently.

Full changelog

New features

• Add experimental support for manifest payloads (#991 / #994 / #1008)
  • Add a new external-api-request example that uses the manifests payload to enable outbound traffic
• Add yapapi.contrib.SocketProxy - a generic proxy to facilitate connections to any TCP-based services running on providers (#1027)
  • Remove dependency on websocat in the ssh example
• Add support for removal of IP addresses from a VPN (#1054)
  • Recycle IP addresses of failed Service instances so they can be restarted on a new node (#1054)
• Enable Service to implement logic to decide whether instances should be restarted (#1022)
  • new Service.restart_condition hook
• Allow MarketStrategy classes to define their own criteria for debit note and invoice acceptance (#954)
  • new MarketStrategy.invoice_accepted_amount hook
  • new MarketStrategy.debit_note_accepted_amount hook
• Add a health-check task to continually verify the state of running Service instances and detect defunct ones (#1033)
  • Add Service.is_activity_responsive method

Backward-incompatible changes

• Remove components deprecated in 0.6 and 0.7 (#919)
  • Remove driver and network arguments of the Golem class
    • plus the respective driver and network properties
  • remove yapapi.package.vm module (use yapapi.payload.vm instead)
    • Remove exescript helpers from yapapi.ctx and make usage of the Script explicit:
      • yapapi.ctx.CommandContainer
      • WorkContext.commit
      • command methods from WorkContext, e.g. ctx.run()
        • client code should just yield Script objects from task / service handlers instead
  • Remove yapapi.executor.ctx, yapapi.executor.events, yapapi.executor.strategy
    • these had been moved to yapapi.engine
  • Remove yapapi.props.inf.RuntimeType enum,
    • Use yapapi.props.inf.RUNTIME_* constants directly
  • Remove yapapi.props.inf.InfVm,
    • those properties had been included into yapapi.payload.vm
• Replace ServiceStarted event with ServiceStateChanged (#958)

Minor updates

• Change the default subnet to public (#1048)

Fixes

• Capture and handle errors in Strategy.score_offer (#908)
• Enable the LocalHTTPProxy to properly handle requests larger than 64kbytes (#1013)
  • Add a modified example web app example that uses file uploads to test larger requests
• Allow the engine to accept one additional DebitNote when activities are terminated (#1030)
• Add handling for remote response timeouts in the local http proxy (#1050)
• Fix handling and reporting of IP address collisions in the VPN (#1053)

Internal changes

• Overhaul the invoice processing and introduce InvoiceManager to encapsulate all logic related to processing of invoices from the providers. #886
• Add concurrent DebitNote handling (#902)
• Split proposal-receiving and proposal-scoring (#968)
• Use updated yagna REST API bindings (#944)
• Copy distutils.util.strtobool to yapapi.utils because of deprecation (#1009)
• Remove pre-commit configuration and update the README (#1018)
• Add isort to project dependencies and checks to enforce imports ordering (#1034)

CI and Testing

• Add an integration test for the webapp example (#974)
• Add an integration test for the scan example (#977)
• Update goth to newest version
• Various minor fixes improving the reliability of the integration tests

0.10.0-alpha.1

What's Changed

• use public as the default subnet by @shadeofblue in #1048
• add handling for remote response timeout in http proxy by @shadeofblue in #1050
• fix webapp-fileupload by @shadeofblue in #1052
• Blue/0.10 ip address colission reporting by @shadeofblue in #1053
• recycle network IPs by @shadeofblue in #1054

Full Changelog: 0.10.0-alpha.0...0.10.0-alpha.1

0.10.0-alpha.0

What's Changed

• Invoice managment refactoring by @johny-b in #886
• Concurrent debit note handling by @filipgolem in #902
• fix golem nightly by @shadeofblue in #833
• Score error fix by @scx1332 in #908
• Remove components deprecated in 0.6 and 0.7 by @filipgolem in #919
• Use updated yagna REST API bindings by @filipgolem in #944
• Assert that websocat is installed in SSH goth test by @kmazurek in #949
• ServiceStateChanged event by @johny-b in #958
• Accepted amount for a debit note/invoice is decided in the MarketStrategy by @johny-b in #954
• Split proposal-receiving and proposal-scoring by @johny-b in #968
• bump goth version to 0.11 by @johny-b in #972
• Webapp example test by @shadeofblue in #974
• add goth test for the scan example by @shadeofblue in #977
• bump goth to 0.12 by @shadeofblue in #983
• Manifest support by @pwalski in #991
• fix the goth runs by @shadeofblue in #995
• bump checkout and python installer action versions by @shadeofblue in #997
• Yagna outbuond network Goth test related changes by @pwalski in #993
• align external_api_request example with the others (a bit) by @shadeofblue in #1005
• Optional manifest singature by @pwalski in #994
• copy distutils.util.strtobool to yapapi.utils because of deprecation by @shadeofblue in #1009
• use a specific version of yagna in the tests by @shadeofblue in #1015
• LocalHTTPProxy fix for larger request sizes + modified webapp -> file upload example by @shadeofblue in #1013
• Update pre-commit configuration by @lucekdudek in #1018
• 1007: increased timeout when resolving image repo src by @lucekdudek in #1017
• Added ssh and websocat goth requirements to README, misc by @lucekdudek in #1023
• #986: Move restart required logic to Service by @lucekdudek in #1022
• Add a SocketProxy by @shadeofblue in #1027
• #1021 Increase timeouts for goth test by @lucekdudek in #1032
• Resolves #1014 Adjust engine termination reason logic so we accept one… by @lucekdudek in #1030
• Added isort by @lucekdudek in #1034
• ServiceRunner's health check task by @shadeofblue in #1033
• Computation Payload Manifest documentation by @pwalski in #1008

New Contributors

• @scx1332 made their first contribution in #908
• @pwalski made their first contribution in #991
• @lucekdudek made their first contribution in #1018

Full Changelog: 0.9.3...0.10.0-alpha.0

0.9.3

What's Changed

• fix the hashcat test by @shadeofblue in #978
• Merge pull request #983 from golemfactory/bump-goth-to-0.12 by @shadeofblue in #989
• Blue/0.9 fix goth runs by @shadeofblue in #998
• fix the agreement termination race condition keyerror by @shadeofblue in #1000
• add option to specify the goth config file in the integration tests by @shadeofblue in #1002
• 0.9/integration test fixes by @shadeofblue in #1016

Full Changelog: 0.9.2...0.9.3

0.9.3-alpha.0

What's Changed

• fix the hashcat test by @shadeofblue in #978
• Merge pull request #983 from golemfactory/bump-goth-to-0.12 by @shadeofblue in #989
• Blue/0.9 fix goth runs by @shadeofblue in #998
• fix the agreement termination race condition keyerror by @shadeofblue in #1000

Full Changelog: 0.9.2...0.9.3-alpha.0

0.9.2

What's Changed

• Failed to spawn instance fix (assign to agreement variable before creating activity) by @filipgolem in #929
• 0.9 fix wiezzel's issues by @shadeofblue in #953
• Catch KeyError instead of IndexError when accessing self._agreements which is a dictionary (to b0.9) by @filipgolem in #956
• Invalid image hash -> an exception (in services) by @johny-b in #959
• Next step towards mypy --strict by @johny-b in #960
• Blue/activity start time by @shadeofblue in #964
• bump goth version to 0.11 by @johny-b in #971

Full Changelog: 0.9.1...0.9.2

0.9.2-alpha.0

What's Changed

• Failed to spawn instance fix (assign to agreement variable before creating activity) by @filipgolem in #929
• 0.9 fix wiezzel's issues by @shadeofblue in #953
• Catch KeyError instead of IndexError when accessing self._agreements which is a dictionary (to b0.9) by @filipgolem in #956
• Invalid image hash -> an exception (in services) by @johny-b in #959
• Next step towards mypy --strict by @johny-b in #960
• Blue/activity start time by @shadeofblue in #964

Full Changelog: 0.9.1...0.9.2-alpha.0

0.9.1

What's Changed

• No allocation timeout by @filipgolem in #926
• fixes 927 so ProviderFilter handles coroutine callback by @krunch3r76 in #930

Full Changelog: 0.9.0...0.9.1

0.9.0

Release notes

We're happy to announce a major release of yapapi - Golem's Python API 0.9.

Events

This update adds a completely overhauled events subsystem that makes it much easier to implement more complicated functionality on top of yapapi. Additionally, it allows those components to be better decoupled and more independent from each other, increasing their composability and reusability.

Also, the new events now carry all the objects they pertain to instead of just the identifiers of those objects, as was the case previously. This saves the developers the hurdle of having to take care of storing, handling and retrieving those objects in their own code.

On top of that, the new events interface allows the developers to easily write event consumers that listen only to certain types of events and which can also handle custom events defined outside of yapapi.

Immediately available Service instance handlers

Another important addition to the API is the ability to get handles to Service instances immediately after those instances are requested through a Golem.run_service call.

Thanks to this change, the client code can start explicitly tracking what is happening with the commissioned Service instances without having to "guess" their status.

General motivation behind this change was better management of the services deployed through yapapi and simplification of code that wishes to extend this functionality further.

Yapapi Contrib

We have decided to introduce a new part of yapapi, aimed less at implementing the functionality of the library itself and more at adding additional components that are build on top of yapapi.

This part of yapapi repository serves to provide requestor agent application authors with reusable components that we or third-party developers found useful while working on examples and apps that use our Python API.

They’re not strictly part of the high level API library itself and while we do intend to keep them in sync with the subsequent releases, they should be treated as experimental and their interface may change without a proper deprecation process.

The first components that you can already use are the ProviderFilter market strategy helper and an http proxy component for network-enabled services.

ProviderFilter

ProviderFilter is a market strategy wrapper that enables easy exclusion of offers from certain providers using a simple boolean condition, while preserving correct scoring of the remaining offers by the base strategy.

It can be used to construct your own custom deny lists based on a simple registry or it can be extended through the new events interface to construct such a deny list dynamically based on some events (e.g. failure of an activity on a specific provider node).

LocalHTTPProxy

This component was previously part of the HTTP proxy example which we made available in our last major release.

Now, we have extracted it and we're making it available to all yapapi developers as a stand-alone module that you can include in the apps that you build on top of yapapi.

The local HTTP proxy enables easy connections to any VPN-enabled, HTTP-based services launched on Golem providers using yapapi’s Services API.

Accompanying it is a new example - a web application composed of a simple web server and a separate database node.

Mid-agreement payments

To allow longer agreement times, mainly in order to enable services running on Golem for durations of e.g. weeks and months as opposed to minutes or hours, we have added the mid-agreement payments support which works with the analogous update to newest version of yagna's provider agent.

This functionality relies on additional negotiation between the providers and requestors, through which the requestor agrees to pay the provider in set intervals within the duration of an agreement.

Before this change, the payments had only been made after an invoice was issued at the end, when the agreement has been terminated. Now, the agreement can persist indefinitely as long as the provider stays active and as long as the requestor continues to pay for that activity.

More on that in our handbook..

Early break from Task API's worker function

The last small but important feature we're making available in this release in yapapi is the ability to cleanly exit from Task API's worker function when new tasks are still available.

We've found this useful in various scenarios where, for any reason, we don't want to just spread our task over the Golem network to distribute the computational load but are rather focused on the executing nodes themselves.

This might include needs such as e.g. running a benchmark on as many nodes as possible to construct a list that we later use to run proper computations, or running a task that takes advantage of geographical distribution of Golem's nodes.

This feature is presented through our new node scanner example.

New examples

As already mentioned above, we've prepared two new examples showcasing new yapapi features.

Node scanner

The node scanner example presents a clean and simple way of using yapapi's Task API to run a single task per a provider node to find what CPU brand is each provider using.

While not very useful on its own, it serves as a template for any scenario where a single (or a specific number of) task on a single node is needed - or - any scenario where we'd like to stop executing tasks on a specific node for whatever other reason.

Web application

The last but not the least, we're presenting you with a web application example, which utilizes the Service API, yagna's VPN capability alongside the associated Network API in yapapi and finally the aforementioned Local HTTP Proxy module to run a proper web application on Golem.

The app is composed of an extremely simple HTTP server written in Python and using Flask and SQLAlchemy and of a database server running on rqlite (which is a replication-capable db back-end based on SQLite).

Both components are running on separate provider nodes and are connected by a virtual private network (VPN) both to themselves and to the requestor node.

Finally, thanks to the local HTTP proxy, we're able to open a port on requestor's machine and route all traffic from that port to the HTTP server on a provider.

Miscellanea

Apart from that, we're including bunch of other, minor updates including fixes and small improvements here and there. Please have a look at the detailed changelog below.

Detailed changelog

Backward-incompatible changes

• Any functionality relying on the API's events will need to be updated to use the new events instead.

Major updates

• Overhauled events subsystem - Events API Reference
  • Add Golem.add_event_consumer method (#755)
  • ComputationStarted/Finished events in services API (#761)
  • Services now emit Service[Started|Finished] instead of Task[Started|Finished] (#790)
  • Replace PaymentAccepted with (Invoice|DebitNote)Accepted (#792)
  • Emit events using actual objects of interest instead of just their ids (#783)
  • New events, with full objects as attributes (e.g. agr_id -> Agreement) (#801)
  • Change the exception interface for events (#798)
  • Improve event-related type hints (#802)
  • Add custom repr and str methods to Event class (#806)
  • Event-based DecreaseScoreForUnconfirmedAgreement strategy (#805)
  • Add reprs to script event objects
  • General cleanup of the Events-related functionality (#840)
  • Event-related cleanup of yapapi.log.SummaryLogger (#847)
  • Golem.add_event_consumer is no longer async. (#866)
  • Start consumers only when Golem is operative
  • Event pre-filtering in Golem.add_event_consumer. (#868)
• Separate ServiceRunner from the Cluster and expose immediately available handles to Service instances (#750)
  • yapapi.services split to separate files
  • yapapi.services.Cluster split to ServiceRunner and a Cluster
  • Add Service.reset() method, called when a service is restarted, since handlers of restarted instances can be called more than once now
• Introduce yapapi.contrib - a helper library of reusable components
  • New ProviderFilter helper strategy (#811) - see ProviderFilter API reference
  • Make the local http proxy a reusable component (#860) - see LocalHttpProxy API reference
    • make the proxy more robust
    • better error handling in HttpProxyService
• Implement the mid-agreement payments mechanism enabling indefinitely long agreements between the providers and requestors (#831) and (#864) - [ TODO add a link to the new Payments article ], GAP-3 Mid-agreement payments
  • Add a general mechanism to handle negotiable properties
• add a new WrappingMarketStrategy interface and convert all current wrapping strategies to WrappingMarketStrategy (#877) - [TODO add a link to the API reference]
• Provide a clean way of an early break from a worker function (#83...

0.9.0-alpha.4

bump version to 0.9.0-alpha.4