Fix errors in Python decorator API documentation #2124

SeanTAllen · 2018-03-22T16:42:46Z

It doesn't exist and using it causes a syntax error.

[skip ci]

It doesn't exist and using it causes a syntax error. [skip ci]

The `Wallaroo API` section of the Python API docs was not updated to use the latest API and referenced old and unused decorators. This change updates the file to remove unused decorators and update the naming/formatting of the ones which still exist. [skip ci]

nisanharamati · 2018-03-22T17:19:23Z

book/python/api.md

@@ -229,21 +229,21 @@ The TCP Source Decoder is responsible for two tasks:
 1. Telling Wallaroo _how many bytes to read_ from its input connection.
 2. Converting those bytes into an object that the rest of the application can process.

-Your function must be decorated with `@decoder(header_length, fmt)`.
+Your function must be decorated with `@wallaroo.decoder(header_length=header_length, length_fmt="length_fmt")`.


length_fmt doesn't take a name string here. It takes an argument that is passed to struct.unpack following https://docs.python.org/2/library/struct.html#format-strings

That's described in the #### length_fmt` section below, but the example probably shouldn't use an invalid argument either?

any thoughts on what you'd propose as an optimal replacement to "length_fmt"?

I'm tempted to use real values from our examples, like @wallaroo.decoder(header_length=4, length_fmt=">I")

nisanharamati · 2018-03-22T17:21:14Z

book/python/api.md

@@ -254,8 +254,8 @@ Return a python a python object of the type the next step in the pipeline expect
 A complete TCPSourceDecoder example that decodes messages with a 32-bit unsigned integer _payload\_length_ and a character followed by a 32-bit unsigned int in its _payload_:

 ```python
-@decoder(header_length=4, length_fmt=">L")
-def decode(self, bs):
+@wallaroo.decoder(header_length=4, length_fmt=">L")


We've been using ">I" in our examples, so this should probably match (they're functionally identical)

nisanharamati · 2018-03-22T19:43:45Z

book/python/api.md

@@ -187,15 +187,15 @@ Partition Keys must correctly support the `__eq__` (`==`) and `__hash__` operato

 ### PartitionFunction

-A partition function class must be wrapped in the `@partition_function` decorator and return the appropriate [Key](#key) for `data`.
+A partition function class must be wrapped in the `@wallaroo.partition` decorator and return the appropriate [Key](#key) for `data`.


Not a class anymore!
should be
A partition function must be decorated with ... (also note decorated with instead of wrapped in)

nisanharamati · 2018-03-22T19:44:12Z

book/python/api.md

@@ -187,15 +187,15 @@ Partition Keys must correctly support the `__eq__` (`==`) and `__hash__` operato

 ### PartitionFunction


Title should probably just be Partition

nisanharamati · 2018-03-22T19:46:03Z

book/python/api.md


-##### `encode(data)`
+##### `encoder(data)`


This should stay as encode as it's a function, not a class. It should be named after the action it performs, not the object performing it

Now that I think about it, this section is weird. It looks like it's left over from when encode was a field of the Encoder class, but isn't anymore. Maybe the entire section should be revised to reflect that it's just a decorated function?

all of our examples use encoder, should we update all of those as well?

nisanharamati · 2018-03-22T19:46:27Z

book/python/api.md

@@ -218,7 +218,7 @@ A complete `TCPSinkEncoder` example that takes a list of integers and encodes it

 ```python
 @encoder
-def encode(self, data):
+def encoder(data):


ditto here: should be def encode(data)

Removes usage of classes when objects can now be functions. Removes usage of TCPSinkEncoder and TCPSourceDecoder which no longer exist. [skip ci]

JONBRWN · 2018-03-23T15:14:13Z

@nisanharamati made some updates, review it when you have a chance. Thanks!

nisanharamati · 2018-03-27T18:29:35Z

book/python/api.md

@@ -141,7 +140,7 @@ Return the complete list of topology tuples. This is the topology structure Wall

 A stateless computation is a simple function that takes input, returns an output, and does not modify any variables outside of its scope. A stateless computation has _no side effects_.

-A `computation` class must be wrapped by the `@computation(name)` decorator, which takes the name of the computation as an argument.
+A `computation` function must be decorated with the `@wallaroo.computation(name="computation name")` decorator, which takes the name of the computation as an argument.


I think this should also include @wallaroo.computation_multi as an option.

nisanharamati · 2018-03-27T18:30:38Z

book/python/api.md

@@ -167,7 +166,7 @@ def compute(data):
 A Computation that returns both its input integer and double that value. If the incoming data isn't an integer, we filter (drop) the message by returning `None`.

 ```python
-@computation_multi("doubledouble"):
+@wallaroo.computation_multi(name="doubledouble"):


No colon at the end of a decorator

nisanharamati · 2018-03-27T23:20:30Z

@JONBRWN Please give this a pass?
If you don't like these changes, feel free to overrule them.

I did the following:

Organize everything related to Sources, Sinks, and Computations in groups on the page.
Within the sources and sinks groups, I further reorganized to use the following hierarchy: Source -> TCPSource | KafkaSource -> Source Decoder -> Example TCP | Example Kafka and similarly for sinks
Within the Computations group, I organized it so that the decorators are subheaders of the Computation and StateComputation levels respectively, and each with their own example.

nisanharamati · 2018-03-27T23:32:58Z

Still missing:
something about tcp_parse_input_addrs, tcp_parse_output_addrs, kafka_parse_source_options, and kafka_parse_sink_options

aturley · 2018-03-28T16:53:45Z

@SeanTAllen title says "Fix errors in Python generator API documentation", but I think you meant "Fix errors in Python decorator API documentation", correct?

SeanTAllen · 2018-03-28T17:13:09Z

omg, i cant believe i keep typing generator rather than decorator.

JONBRWN · 2018-03-28T17:20:26Z

@nisanharamati this looks good to me but I don’t have as much of an opinion on documentation as others.

@aturley any thoughts on the changes?

aturley · 2018-04-03T00:52:15Z

This looks good for now.

Remove references to @State decorator from Python docs

37cb548

It doesn't exist and using it causes a syntax error. [skip ci]

SeanTAllen requested a review from JONBRWN March 22, 2018 16:42

JONBRWN requested a review from nisanharamati March 22, 2018 17:16

nisanharamati reviewed Mar 22, 2018

View reviewed changes

SeanTAllen added the changelog - fixed label Mar 22, 2018

nisanharamati reviewed Mar 22, 2018

View reviewed changes

SeanTAllen changed the title ~~Remove references to @state decorator from Python docs~~ Fix errors in Python generator API documentation Mar 23, 2018

Further updates/cleanup to Python API doc

193ebad

Removes usage of classes when objects can now be functions. Removes usage of TCPSinkEncoder and TCPSourceDecoder which no longer exist. [skip ci]

nisanharamati reviewed Mar 27, 2018

View reviewed changes

nisanharamati added 3 commits March 27, 2018 16:08

Overhaul Sink, Source and Computations

098c01b

Fix application builder link

dfd9bdf

Update description of new_pipeline and to_sinks

e7201dd

SeanTAllen changed the title ~~Fix errors in Python generator API documentation~~ Fix errors in Python decorator API documentation Mar 28, 2018

aturley merged commit 919c00c into master Apr 3, 2018

wallaroolabs-wally added a commit that referenced this pull request Apr 3, 2018

Update CHANGELOG for PR #2124 [skip ci]

06a7bb6

nisanharamati deleted the no-state-decorator branch April 3, 2018 02:20

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix errors in Python decorator API documentation #2124

Fix errors in Python decorator API documentation #2124

SeanTAllen commented Mar 22, 2018

nisanharamati Mar 22, 2018

nisanharamati Mar 22, 2018

JONBRWN Mar 22, 2018

nisanharamati Mar 22, 2018

nisanharamati Mar 22, 2018

nisanharamati Mar 22, 2018

nisanharamati Mar 22, 2018

nisanharamati Mar 22, 2018

nisanharamati Mar 22, 2018

JONBRWN Mar 22, 2018

nisanharamati Mar 22, 2018

JONBRWN commented Mar 23, 2018

nisanharamati Mar 27, 2018

nisanharamati Mar 27, 2018

nisanharamati commented Mar 27, 2018

nisanharamati commented Mar 27, 2018

aturley commented Mar 28, 2018

SeanTAllen commented Mar 28, 2018

JONBRWN commented Mar 28, 2018

aturley commented Apr 3, 2018

		@@ -187,15 +187,15 @@ Partition Keys must correctly support the `__eq__` (`==`) and `__hash__` operato

		### PartitionFunction


		##### `encode(data)`
		##### `encoder(data)`

Fix errors in Python decorator API documentation #2124

Fix errors in Python decorator API documentation #2124

Conversation

SeanTAllen commented Mar 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JONBRWN commented Mar 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nisanharamati commented Mar 27, 2018

nisanharamati commented Mar 27, 2018

aturley commented Mar 28, 2018

SeanTAllen commented Mar 28, 2018

JONBRWN commented Mar 28, 2018

aturley commented Apr 3, 2018