Switch to mainline ponyc 0.19.next_release #1615

dipinhora · 2017-10-05T21:02:31Z

Switch to mainline ponyc 0.19.next_release (because it includes a critical bugfix).

Note: this PR is the same as @jtfmumm's PR #1542 but rebased onto the latest master with updates for the new Makefile conventions.

We used our own helper methods for AsioEvent in our version of ponyc. Now that we're moving over to mainline ponyc, I've broken these out into a helper primitive in wallaroo_labs.

ponyc 0.10.0 changed behaviors to return None instead of the actor. This means that in multiple places throughout the code we need to now use the chaining notation or directly return the actor when we expect a tag as a return type instead of None.

This is required for mainline ponyc

In order to upgrade to mainline ponyc, we need to use the new approach to chaining.

We don't need wrapper methods for the FFI calls `@pony_asio_event_set_readable` and `@pony_asio_event_set_writeable`. Call them directly as in mainline ponyc.

ponyc 0.16.0 requires that partial calls are explicit, using x()? instead of x()

Simplifies transformation of missing files into String by removing use of Iter that is not compatible with ponyc 0.18.0.

Before one-shot changes to ponyc, we followed a different approach for Linux and OSX, but now we can follow our Linux approach for both given our switch to ponyc 0.19.1.

Update docs and .travis.yml to reflect the fact that we are now using mainline ponyc.

Update to use new partial call syntax.

Update Sender and Receiver to use new approach to chaining and explicit partial call syntax.

A number FFI signatures needed to be updated for compatibility with ponyc 0.19.1.

Now that we're switching to mainline ponyc, we no longer need to instruct users to install llvm and related dependencies.

We do not currently support the C++ language bindings, so this removes C++ test builds from CI.

Added explicit partial calls.

A few of the example Pony apps used helper applications that needed updates to use explicit partial calls.

We need pony-kakfa 0.2.0 for machida and celsius-kafka in order to build with ponyc 0.19.2.

C++ API is no longer supported and C++ apps no longer build.

Outputs to stdout did not distinguish between control and data channels when reporting that all workers had identified their channels.

When we initialize control connections, we need to make sure we don't try to write to the TCPConnection until it's connected. The ControlConnection wrapper buffers data until the connection is ready.

A few WallarooLabs libs not used by Wallaroo itself still needed updates for explicit partial calls and new chaining approach.

Topology Layout test Pony programs needed to use new explicit partial calls.

Wesley tests needed explicit partial calls and new approach to chaining.

Backport ponylang/ponyc#1578 and ponylang/ponyc#1626 to net actors because the help resolve edge cases around the networking code with direct impacts to the stability of the network tests.

Previously, as part of commit 2feeb40, we switched from sending `Array[ByteSeq] val` to `Array[ByteSeq] iso` to work around a `val` tracing bug. As part of the update to the latest ponylang ponyc, that workaround is no longer required and this commit changes the `Array[ByteSeq] iso` back to `Array[ByteSeq] val`.

As part of #1125, `_wb.reserve` calls were added to Runner in order to avoid a data/memory corruption bug in the WallarooLabs ponyc compiler. As part of the update to ponylang ponyc, the workaround is no longer required. Resolves #1612

jtfmumm · 2017-10-12T09:45:57Z

@JONBRWN @dipinhora Are we still waiting on something before merging this PR? My understanding from the sync meeting was that we accept the performance regression and there are no changes to documentation required.

JONBRWN · 2017-10-12T13:04:55Z

@jtfmumm wanted to bring up a few things with @SeanTAllen in a meeting I have with him today before merging, will merge after that.

jtfmumm · 2017-10-12T13:05:40Z

@JONBRWN Ok sounds good.

SeanTAllen · 2017-10-12T14:10:04Z

.travis.yml

    fi;
  - if [ "${TRAVIS_OS_NAME}" = "linux" ];
    then
-      sudo apt-get -fy install cpuset;


this shouldnt be removed

SeanTAllen · 2017-10-12T14:10:34Z

.travis.yml

-    git clone https://github.com/WallarooLabs/ponyc.git;
-    cd ponyc;
-    make CC=$CC1 CXX=$CXX1;
-    sudo make install;
  - echo "Installing pony-stable";
    cd /tmp;
    git clone https://github.com/ponylang/pony-stable;


this should be installed via apt-get

SeanTAllen · 2017-10-12T14:10:58Z

.travis.yml

@@ -48,55 +48,30 @@ install:
  # return to our starting directory, then our `script` step will fail because
  # its in the wrong directory.
  - export INSTALL_STARTED_AT=`pwd`
+  - echo "Installing WallarooLabs/ponyc";


we dont install our ponyc. this can be changed.

Updates pony-stable to be installed via brew and apt-get. Re-inserts cpuset installation on linux. Changes output to only say "Installing ponyc"

jtfmumm and others added 27 commits October 5, 2017 18:53

Update arrays to use sequence notation

b1e3828

Rename instances of IPAddress to NetAddress

817aac8

Add AsioEventHelper

2ce8d62

We used our own helper methods for AsioEvent in our version of ponyc. Now that we're moving over to mainline ponyc, I've broken these out into a helper primitive in wallaroo_labs.

Update TCPConnectNotifiers to implement connect_failed

6e4ab8e

This is required for mainline ponyc

Update machida to use new approach to chaining

e2766c7

In order to upgrade to mainline ponyc, we need to use the new approach to chaining.

Use direct calls instead of AsioEventHelper

2accfe4

We don't need wrapper methods for the FFI calls `@pony_asio_event_set_readable` and `@pony_asio_event_set_writeable`. Call them directly as in mainline ponyc.

Update to use new explicit partial calls

1d215e4

ponyc 0.16.0 requires that partial calls are explicit, using x()? instead of x()

Remove outdated use of Iter

40e9766

Simplifies transformation of missing files into String by removing use of Iter that is not compatible with ponyc 0.18.0.

Update OSX event handling to follow Linux approach

a296a68

Before one-shot changes to ponyc, we followed a different approach for Linux and OSX, but now we can follow our Linux approach for both given our switch to ponyc 0.19.1.

Update installation docs and .travis.yml

ed6378c

Update docs and .travis.yml to reflect the fact that we are now using mainline ponyc.

Update cluster shutdown tool for ponyc 0.19.1

ccef9dd

Update to use new partial call syntax.

Update Giles to compile with ponyc 0.19.1

a4f27f4

Update Sender and Receiver to use new approach to chaining and explicit partial call syntax.

Fix FFI incompatibilities

80ffc41

A number FFI signatures needed to be updated for compatibility with ponyc 0.19.1.

Remove llvm install instructions from docs

d4671b0

Now that we're switching to mainline ponyc, we no longer need to instruct users to install llvm and related dependencies.

Disable C++ builds for CI

bc78b89

We do not currently support the C++ language bindings, so this removes C++ test builds from CI.

Update gen.pony to build with ponyc 0.19.1

7072afc

Added explicit partial calls.

Update helper Pony apps to compile with 0.19.2

6ffbcf3

A few of the example Pony apps used helper applications that needed updates to use explicit partial calls.

Update apps to use pony-kafka 0.2.0

f302b9a

We need pony-kakfa 0.2.0 for machida and celsius-kafka in order to build with ponyc 0.19.2.

Update README to explain new C++ API status

3e6a57f

C++ API is no longer supported and C++ apps no longer build.

Improve ClusterInitializer messages

068a999

Outputs to stdout did not distinguish between control and data channels when reporting that all workers had identified their channels.

Add ControlConnection wrapper for TCPConnection

05f006b

When we initialize control connections, we need to make sure we don't try to write to the TCPConnection until it's connected. The ControlConnection wrapper buffers data until the connection is ready.

Update WallarooLabs libs to compile with 0.19.2

28c090b

A few WallarooLabs libs not used by Wallaroo itself still needed updates for explicit partial calls and new chaining approach.

Update testing tools to compile with 0.19.2

c7514bc

Update topology_layout programs for 0.19.2

2939bde

Topology Layout test Pony programs needed to use new explicit partial calls.

Upgrade Wesley tests to 0.19.2

62caed7

Wesley tests needed explicit partial calls and new approach to chaining.

Backport ponylang/ponyc#1578 and ponylang/ponyc#1626 to net actors

34d192c

Backport ponylang/ponyc#1578 and ponylang/ponyc#1626 to net actors because the help resolve edge cases around the networking code with direct impacts to the stability of the network tests.

dipinhora added the DO NOT MERGE label Oct 5, 2017

dipinhora added 2 commits October 5, 2017 21:25

Remove _wb.reserve calls from Runner

9fe46ea

As part of #1125, `_wb.reserve` calls were added to Runner in order to avoid a data/memory corruption bug in the WallarooLabs ponyc compiler. As part of the update to ponylang ponyc, the workaround is no longer required. Resolves #1612

SeanTAllen reviewed Oct 12, 2017

View reviewed changes

Update travis yml file

4ae7990

Updates pony-stable to be installed via brew and apt-get. Re-inserts cpuset installation on linux. Changes output to only say "Installing ponyc"

JONBRWN force-pushed the ponyc-update-0.19.next_release branch from 89ce9b8 to 4ae7990 Compare October 12, 2017 16:49

JONBRWN removed the DO NOT MERGE label Oct 12, 2017

JONBRWN merged commit 24112c2 into master Oct 12, 2017

JONBRWN deleted the ponyc-update-0.19.next_release branch October 12, 2017 19:34

JONBRWN mentioned this pull request Oct 12, 2017

Switch to mainline ponyc 0.19.2 #1542

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to mainline ponyc 0.19.next_release #1615

Switch to mainline ponyc 0.19.next_release #1615

dipinhora commented Oct 5, 2017

jtfmumm commented Oct 12, 2017

JONBRWN commented Oct 12, 2017

jtfmumm commented Oct 12, 2017

SeanTAllen Oct 12, 2017

SeanTAllen Oct 12, 2017

SeanTAllen Oct 12, 2017

Switch to mainline ponyc 0.19.next_release #1615

Switch to mainline ponyc 0.19.next_release #1615

Conversation

dipinhora commented Oct 5, 2017

jtfmumm commented Oct 12, 2017

JONBRWN commented Oct 12, 2017

jtfmumm commented Oct 12, 2017

SeanTAllen Oct 12, 2017

Choose a reason for hiding this comment

SeanTAllen Oct 12, 2017

Choose a reason for hiding this comment

SeanTAllen Oct 12, 2017

Choose a reason for hiding this comment