Skip to content

Commit

Permalink
merge main
Browse files Browse the repository at this point in the history
  • Loading branch information
timkpaine committed Nov 22, 2021
2 parents 2ccfc18 + 8e6901f commit 91c790a
Show file tree
Hide file tree
Showing 36 changed files with 2,993 additions and 580 deletions.
6 changes: 3 additions & 3 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:
strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
python-version: [3.7, 3.8]
python-version: [3.7, 3.9]

steps:
- uses: actions/checkout@v2
Expand All @@ -35,12 +35,12 @@ jobs:
- name: Test
run: |
make tests
make testsci
if: ${{ matrix.os != 'ubuntu-latest' }}

- name: Test
run: |
make dockerup && make tests && make dockerdown
make dockerup && make testsci && make dockerdown
if: ${{ matrix.os == 'ubuntu-latest' }}


Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
python-version: [3.7, 3.8]
python-version: [3.7, 3.8, 3.9]

steps:
- uses: actions/checkout@v2
Expand All @@ -34,12 +34,12 @@ jobs:
- name: Test
run: |
make tests
make testsci
if: ${{ matrix.os != 'ubuntu-latest' }}

- name: Test
run: |
make dockerup && make tests && make dockerdown
make dockerup && make testsci && make dockerdown
if: ${{ matrix.os == 'ubuntu-latest' }}

- name: Build and publish
Expand Down
125 changes: 125 additions & 0 deletions CATALOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
# Sources and Sinks
## Sources
- Python Function/Generator/Async Function/Async Generator
- Curve - yield through an iterable
- Const - yield a constant
- Timer - yield on an interval
- Random - generates a random dictionary of values
- File - streams data from a file, optionally loading each line as a json
- HTTP - polls a url with GET requests, streams data out
- HTTPServer - runs an http server and streams data sent by clients
- Websocket - strams data from a websocket
- WebsocketServer - runs a websocket server and streams data sent by clients
- SocketIO - streams data from a socketIO connection
- SocketIOServer - streams data from a socketIO connection
- SSE - streams data from an SSE connection
- Kafka - streams data from kafka
- Postgres - streams data from postgres

## Sinks
- Foo - data to a python function
- File - data to a file
- HTTP - POSTs data to an url
- HTTPServer - runs an http server and streams data to connections
- Websocket - streams data to a websocket
- WebsocketServer - runs a websocket server and streams data to connections
- SocketIO - streams data to a socketIO connection
- SocketIOServer - runs a socketio server and streams data to connections
- SSE - runs an SSE server and streams data to connections
- Kafka - streams data to kafka
- Postgres - streams data to postgres
- Email - streams data and sends it in emails
- TextMessage - streams data and sends it via text message

# Transforms
## Modulate
- Delay - Streaming wrapper to delay a stream
- Throttle - Streaming wrapper to only tick at most every interval
- Debounce - Streaming wrapper to only tick on new values
- Apply - Streaming wrapper to apply a function to an input stream
- Window - Streaming wrapper to collect a window of values
- Unroll - Streaming wrapper to unroll an iterable stream
- UnrollDataFrame - Streaming wrapper to unroll a dataframe into a stream
- Merge - Streaming wrapper to merge 2 inputs into a single output
- ListMerge - Streaming wrapper to merge 2 input lists into a single output list
- DictMerge - Streaming wrapper to merge 2 input dicts into a single output dict. Preference is given to the second input (e.g. if keys overlap)
- Reduce - Streaming wrapper to merge any number of inputs
- FixedMap - Map input stream to fixed number of outputs
- Subprocess - Open a subprocess and yield results as they come. Can also stream data to subprocess (either instantaneous or long-running subprocess)


## Calculations
Note that `tributary` can also be configured to operate on **dual numbers** for things like lazy or streaming autodifferentiation.

### Arithmetic Operators
- Noop (unary) - Pass input to output
- Negate (unary) - -1 * input
- Invert (unary) - 1/input
- Add (binary) - add 2 inputs
- Sub (binary) - subtract second input from first
- Mult (binary) - multiple inputs
- Div (binary) - divide first input by second
- RDiv (binary) - divide second input by first
- Mod (binary) - first input % second input
- Pow (binary) - first input^second input
- Sum (n-ary) - sum all inputs
- Average (n-ary) - average of all inputs
- Round (unary)
- Floor (unary)
- Ceil (unary)

### Boolean Operators
- Not (unary) - `Not` input
- And (binary) - `And` inputs
- Or (binary) - `Or` inputs

### Comparators
- Equal (binary) - inputs are equal
- NotEqual (binary) - inputs are not equal
- Less (binary) - first input is less than second input
- LessOrEqual (binary) - first input is less than or equal to second input
- Greater (binary) - first input is greater than second input
- GreaterOrEqual (binary) - first input is greater than or equal to second input

### Math
- Log (unary)
- Sin (unary)
- Cos (unary)
- Tan (unary)
- Arcsin (unary)
- Arccos (unary)
- Arctan (unary)
- Sqrt (unary)
- Abs (unary)
- Exp (unary)
- Erf (unary)

### Financial Calculations
- RSI - Relative Strength Index
- MACD - Moving Average Convergence Divergence

## Converters
- Int (unary)
- Float (unary)
- Bool (unary)
- Str (unary)

## Basket Functions
- Len (unary)
- Count (unary)
- Min (unary)
- Max (unary)
- Sum (unary)
- Average (unary)

## Rolling
- RollingCount - Node to count inputs
- RollingMin - Node to take rolling min of inputs
- RollingMax - Node to take rolling max of inputs
- RollingSum - Node to take rolling sum inputs
- RollingAverage - Node to take the running average
- SMA - Node to take the simple moving average over a window
- EMA - Node to take an exponential moving average over a window

## Node Type Converters
- Lazy->Streaming
21 changes: 9 additions & 12 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,21 @@ build: ## Build the repository
tests: ## Clean and Make unit tests
python -m pytest tributary --cov=tributary --junitxml=python_junit.xml --cov-report=xml --cov-branch

testsci: ## Clean and Make unit tests
CI=true python -m pytest tributary --cov=tributary --junitxml=python_junit.xml --cov-report=xml --cov-branch

testsv: ## Clean and Make unit tests
python -m pytest -vvv tributary --cov=tributary --junitxml=python_junit.xml --cov-report=xml --cov-branch

testsnocov: ## Clean and Make unit tests
python -m pytest -v tributary -x

dockerup:
docker-compose -f ci/docker-compose.yml up -d

dockerdown:
docker-compose -f ci/docker-compose.yml down

notebooks: ## test execute the notebooks
./scripts/test_notebooks.sh

Expand All @@ -19,12 +28,6 @@ lint: ## run linter
fix: ## run black fix
python -m black tributary/ setup.py

annotate: ## MyPy type annotation check
python -m mypy -s tributary

annotate_l: ## MyPy type annotation check - count only
python -m mypy -s tributary | wc -l

clean: ## clean the repository
find . -name "__pycache__" | xargs rm -rf
find . -name "*.pyc" | xargs rm -rf
Expand Down Expand Up @@ -56,10 +59,4 @@ help:
print-%:
@echo '$*=$($*)'

dockerup:
docker-compose -f ci/docker-compose.yml up -d

dockerdown:
docker-compose -f ci/docker-compose.yml down

.PHONY: clean build run test tests help annotate annotate_l docs dist dockerup dockerdown
148 changes: 23 additions & 125 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,128 +61,26 @@ Here green indicates executing, yellow indicates stalled for backpressure, and r

Here green indicates executing, and red indicates that the node is dirty. Note the the determination if a node is dirty is also done lazily (we can check with `isDirty` whcih will update the node's graph state.

# Sources and Sinks
## Sources
- Python Function/Generator/Async Function/Async Generator
- Curve - yield through an iterable
- Const - yield a constant
- Timer - yield on an interval
- Random - generates a random dictionary of values
- File - streams data from a file, optionally loading each line as a json
- HTTP - polls a url with GET requests, streams data out
- HTTPServer - runs an http server and streams data sent by clients
- Websocket - strams data from a websocket
- WebsocketServer - runs a websocket server and streams data sent by clients
- SocketIO - streams data from a socketIO connection
- SocketIOServer - streams data from a socketIO connection
- SSE - streams data from an SSE connection
- Kafka - streams data from kafka
- Postgres - streams data from postgres

## Sinks
- Foo - data to a python function
- File - data to a file
- HTTP - POSTs data to an url
- HTTPServer - runs an http server and streams data to connections
- Websocket - streams data to a websocket
- WebsocketServer - runs a websocket server and streams data to connections
- SocketIO - streams data to a socketIO connection
- SocketIOServer - runs a socketio server and streams data to connections
- SSE - runs an SSE server and streams data to connections
- Kafka - streams data to kafka
- Postgres - streams data to postgres
- Email - streams data and sends it in emails
- TextMessage - streams data and sends it via text message

# Transforms
## Modulate
- Delay - Streaming wrapper to delay a stream
- Throttle - Streaming wrapper to only tick at most every interval
- Debounce - Streaming wrapper to only tick on new values
- Apply - Streaming wrapper to apply a function to an input stream
- Window - Streaming wrapper to collect a window of values
- Unroll - Streaming wrapper to unroll an iterable stream
- UnrollDataFrame - Streaming wrapper to unroll a dataframe into a stream
- Merge - Streaming wrapper to merge 2 inputs into a single output
- ListMerge - Streaming wrapper to merge 2 input lists into a single output list
- DictMerge - Streaming wrapper to merge 2 input dicts into a single output dict. Preference is given to the second input (e.g. if keys overlap)
- Reduce - Streaming wrapper to merge any number of inputs
- FixedMap - Map input stream to fixed number of outputs
- Subprocess - Open a subprocess and yield results as they come. Can also stream data to subprocess (either instantaneous or long-running subprocess)


## Calculations
Note that `tributary` can also be configured to operate on **dual numbers** for things like lazy or streaming autodifferentiation.

### Arithmetic Operators
- Noop (unary) - Pass input to output
- Negate (unary) - -1 * input
- Invert (unary) - 1/input
- Add (binary) - add 2 inputs
- Sub (binary) - subtract second input from first
- Mult (binary) - multiple inputs
- Div (binary) - divide first input by second
- RDiv (binary) - divide second input by first
- Mod (binary) - first input % second input
- Pow (binary) - first input^second input
- Sum (n-ary) - sum all inputs
- Average (n-ary) - average of all inputs
- Round (unary)
- Floor (unary)
- Ceil (unary)

### Boolean Operators
- Not (unary) - `Not` input
- And (binary) - `And` inputs
- Or (binary) - `Or` inputs

### Comparators
- Equal (binary) - inputs are equal
- NotEqual (binary) - inputs are not equal
- Less (binary) - first input is less than second input
- LessOrEqual (binary) - first input is less than or equal to second input
- Greater (binary) - first input is greater than second input
- GreaterOrEqual (binary) - first input is greater than or equal to second input

### Math
- Log (unary)
- Sin (unary)
- Cos (unary)
- Tan (unary)
- Arcsin (unary)
- Arccos (unary)
- Arctan (unary)
- Sqrt (unary)
- Abs (unary)
- Exp (unary)
- Erf (unary)

### Financial Calculations
- RSI - Relative Strength Index
- MACD - Moving Average Convergence Divergence

## Converters
- Int (unary)
- Float (unary)
- Bool (unary)
- Str (unary)

## Basket Functions
- Len (unary)
- Count (unary)
- Min (unary)
- Max (unary)
- Sum (unary)
- Average (unary)

## Rolling
- RollingCount - Node to count inputs
- RollingMin - Node to take rolling min of inputs
- RollingMax - Node to take rolling max of inputs
- RollingSum - Node to take rolling sum inputs
- RollingAverage - Node to take the running average
- SMA - Node to take the simple moving average over a window
- EMA - Node to take an exponential moving average over a window

## Node Type Converters
- Lazy->Streaming
## Catalog
See the [CATALOG](CATALOG.md) for a full list of functions, transforms, sources, and sinks.

## Support / Contributors
Thanks to the following organizations for providing code or financial support.


<a href="https://nemoulous.com"><img src="https://raw.githubusercontent.com/timkpaine/tributary/main/docs/img/nem.png" width="50"></a>

<a href="https://nemoulous.com">Nemoulous</a>

## License
This software is licensed under the Apache 2.0 license. See the [LICENSE](LICENSE) file for details.

## Alternatives
Here is an incomplete list of libraries which implement similar/overlapping functionality

- [man-group/mdf](https://github.com/man-group/mdf)
- [cedricleroy/pyungo](https://github.com/cedricleroy/pyungo)
- [python-streamz/streamz](https://github.com/python-streamz/streamz)
- [EntilZha/pyfunctional](https://github.com/EntilZha/PyFunctional)
- [stitchfix/hamilton](https://github.com/stitchfix/hamilton)

2 changes: 1 addition & 1 deletion binder/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
ipydagred3==0.2.5
ipyregulartable==0.1.2
ipywidgets==7.5.1
jupyterlab==2.2.8
jupyterlab==2.2.10
pyEX==0.2.5
requests==2.24.0
tributary==0.1.4
Loading

0 comments on commit 91c790a

Please sign in to comment.