Skip to content

Commit

Permalink
♊ fix(data): stop using eod ohlc path for standardizing intraday data…
Browse files Browse the repository at this point in the history
… ♊ (#78)

* 🎬 chore: init function 🎬 (#7)

* new files

* create symbol / instrument lookup tables

* init or new?

* abstract a few functions out

* historical data

* update readme to reference roadmap

* update reqs

* newline for function

* 🏹 chore: add logo 🏹 (#8)

* add logo

* adding logo

* update gitignore

* update readme

* Create python-app.yml

* organization and linting

* pytest-check

* flatten tests

* flatten tests

* new tests

* test command update

* just run pytest

* init and copy token bash script

* env vars

* workflow

* secrets

* progress towards automation

* handled non ci case

* update token script almost done

* chmod +x

* make tokens dir

* split prebuild and postbuild scripts

* fix workflow

* script file path in workflow

* housecleaning

* get windows username on wsl

* new workflow

* rename workflows

* yml syntax

* schedule

* try again

* try again

* daily update

* update todo

* code coverage

* import sys

* lower coverage threshold

* test_load_portfolio

* auto versioning

* writing last tests

* tests done

* locked reqs

* wrong dotenv

* try again

* update function name

* fix interval def

* comment out csv creation

* yml descriptions

* 🐞 Bug: Commit New OAuth 🐞 (#12)

* debug workflow

* add email and password env vars

* debug echo

* more debugging

* single line

* see if token exists in project dir

* try quotes

* fixed?

* diff to cmp

* double login

* log cmp result

* use quotes

* fixed issue

* try again

* whyyy

* unsilence gpg

* encrypt in root dir

* pass ref

* force=true

* try different action

* try different action

* [AUTO] OAuth token refresh #patch

* remove comments

* force push option

* [AUTO] OAuth token refresh #patch

* Update update.yml

* 📱 fix(broker): auth flow / 2FA 📱 (#16)

* change env var names

* new auth flow

* fix 2fa

* add 2fa env var to workflow

* readme and auto release

* 💰 feat(data): dividends 💰 (#14)

* script updates

* stricter branch cov & csv utils

* csv tests

* data dir

* typo

* fix test

* fix test data object

* uncomment tests

* yfinance import

* add local token update script

* symbols.csv

* script changes

* flake

* update cron schedule

* fix hist test?

* [AUTO] OAuth token refresh #patch

* comment out print statement

* only save hist purposefully

* only auto commit token

* yaml

* token from master

* readme

* readme again

* create datareader and datawriter classes

* update login script

* broker superclass tangent

* reorganization

* flake8

* dividend function done

* try auto symbol

* try again

* [AUTO] Symbol update #patch

* #minor commit

* auto dividend

* fixed robinhood dot to yahoo dash bug

* add multiprocessing

* fixed filtering for dividend and splits tables

* fixed csv update bug + more robust dividend saving

* div update

* div update iex

* constants

* debug

* working on tests

* working broker tests

* remove yfinance dep

* broker tests done

* fix naming issue

* import

* working on marketdata tests

* finished marketdata tests

* fileops tests almost done

* done

* [AUTO] Symbol update #patch

* [AUTO] Symbol update #patch

* [AUTO] Symbol update #patch

* [AUTO] Symbol update #patch

* 💽 feat(data): s3 integration 💽 (#19)

* fix release?

* file walk

* storage file

* s3 dir uploader working as expected

* dividend s3 upload script complete

* dividend workflows and weird dividend bug

* floating point precision fix

* remove data dir

* move pathfinder to fileops

* s3 integration

* add functions to store class and finish most store tests

* finish store tests

* tests done?

* done?

* linting

* aws configure

* make path before saving

* circular import

* make path

* remove pathlib

* fix rename

* more rename fixes

* does this work

* done

* fixed workflows

* hotfix

* 💾 chore(auto): cache pip dependencies 💾 (#20)

* refactor workflows

* pip cache

* add workflow badges to readme

* spacing

* more workflow and readme changes

* 💎 feat(data): add polygon.io 💎 (#23)

* still need to write get_splits fx

* made iexcloud subdirs

* start polygon class

* fix test

* init function

* switch to polygon api

* auto dep check

* polygon dividend redundancy added

* tests

* update env var in workflows

* secrets

* fixes?

* fix test

* done

* Bump robin-stocks from 1.3.0 to 1.4.2 (#25)

* Feature: init function (#7)

* new files

* create symbol / instrument lookup tables

* init or new?

* abstract a few functions out

* historical data

* update readme to reference roadmap

* update reqs

* newline for function

* Add logo (#8)

* add logo

* adding logo

* update gitignore

* update readme

* Create python-app.yml

* organization and linting

* pytest-check

* flatten tests

* flatten tests

* new tests

* test command update

* just run pytest

* init and copy token bash script

* env vars

* workflow

* secrets

* progress towards automation

* handled non ci case

* update token script almost done

* chmod +x

* make tokens dir

* split prebuild and postbuild scripts

* fix workflow

* script file path in workflow

* housecleaning

* get windows username on wsl

* new workflow

* rename workflows

* yml syntax

* schedule

* try again

* try again

* daily update

* update todo

* code coverage

* import sys

* lower coverage threshold

* test_load_portfolio

* auto versioning

* writing last tests

* tests done

* locked reqs

* wrong dotenv

* try again

* update function name

* fix interval def

* comment out csv creation

* yml descriptions

* 🐞 Bug: Commit New OAuth 🐞 (#12)

* debug workflow

* add email and password env vars

* debug echo

* more debugging

* single line

* see if token exists in project dir

* try quotes

* fixed?

* diff to cmp

* double login

* log cmp result

* use quotes

* fixed issue

* try again

* whyyy

* unsilence gpg

* encrypt in root dir

* pass ref

* force=true

* try different action

* try different action

* [AUTO] OAuth token refresh #patch

* remove comments

* force push option

* [AUTO] OAuth token refresh #patch

* Update update.yml

* Bug: Fix Auth Flow / Add 2FA (#16)

* change env var names

* new auth flow

* fix 2fa

* add 2fa env var to workflow

* readme and auto release

* Feature: Get and Store Dividends (#14)

* script updates

* stricter branch cov & csv utils

* csv tests

* data dir

* typo

* fix test

* fix test data object

* uncomment tests

* yfinance import

* add local token update script

* symbols.csv

* script changes

* flake

* update cron schedule

* fix hist test?

* [AUTO] OAuth token refresh #patch

* comment out print statement

* only save hist purposefully

* only auto commit token

* yaml

* token from master

* readme

* readme again

* create datareader and datawriter classes

* update login script

* broker superclass tangent

* reorganization

* flake8

* dividend function done

* try auto symbol

* try again

* [AUTO] Symbol update #patch

* #minor commit

* auto dividend

* fixed robinhood dot to yahoo dash bug

* add multiprocessing

* fixed filtering for dividend and splits tables

* fixed csv update bug + more robust dividend saving

* div update

* div update iex

* constants

* debug

* working on tests

* working broker tests

* remove yfinance dep

* broker tests done

* fix naming issue

* import

* working on marketdata tests

* finished marketdata tests

* fileops tests almost done

* done

* [AUTO] Symbol update #patch

* [AUTO] Symbol update #patch

* [AUTO] Symbol update #patch

* [AUTO] Symbol update #patch

* Feature: S3 Integration (#19)

* fix release?

* file walk

* storage file

* s3 dir uploader working as expected

* dividend s3 upload script complete

* dividend workflows and weird dividend bug

* floating point precision fix

* remove data dir

* move pathfinder to fileops

* s3 integration

* add functions to store class and finish most store tests

* finish store tests

* tests done?

* done?

* linting

* aws configure

* make path before saving

* circular import

* make path

* remove pathlib

* fix rename

* more rename fixes

* does this work

* done

* fixed workflows

* hotfix

* 💽 Feature: Cache pip dependencies 💽 (#20)

* refactor workflows

* pip cache

* add workflow badges to readme

* spacing

* more workflow and readme changes

* 💎 Feature: Add Polygon.io (Double Redundancy) 💎 (#23)

* still need to write get_splits fx

* made iexcloud subdirs

* start polygon class

* fix test

* init function

* switch to polygon api

* auto dep check

* polygon dividend redundancy added

* tests

* update env var in workflows

* secrets

* fixes?

* fix test

* done

* Bump robin-stocks from 1.3.0 to 1.4.2

Bumps [robin-stocks](https://github.com/jmfernandes/robin_stocks) from 1.3.0 to 1.4.2.
- [Release notes](https://github.com/jmfernandes/robin_stocks/releases)
- [Commits](https://github.com/jmfernandes/robin_stocks/commits)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: Krish Suchak <42231639+suchak1@users.noreply.github.com>
Co-authored-by: Krish Suchak <suchak.krish@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump python-dotenv from 0.13.0 to 0.14.0 (#26)

* Feature: init function (#7)

* new files

* create symbol / instrument lookup tables

* init or new?

* abstract a few functions out

* historical data

* update readme to reference roadmap

* update reqs

* newline for function

* Add logo (#8)

* add logo

* adding logo

* update gitignore

* update readme

* Create python-app.yml

* organization and linting

* pytest-check

* flatten tests

* flatten tests

* new tests

* test command update

* just run pytest

* init and copy token bash script

* env vars

* workflow

* secrets

* progress towards automation

* handled non ci case

* update token script almost done

* chmod +x

* make tokens dir

* split prebuild and postbuild scripts

* fix workflow

* script file path in workflow

* housecleaning

* get windows username on wsl

* new workflow

* rename workflows

* yml syntax

* schedule

* try again

* try again

* daily update

* update todo

* code coverage

* import sys

* lower coverage threshold

* test_load_portfolio

* auto versioning

* writing last tests

* tests done

* locked reqs

* wrong dotenv

* try again

* update function name

* fix interval def

* comment out csv creation

* yml descriptions

* 🐞 Bug: Commit New OAuth 🐞 (#12)

* debug workflow

* add email and password env vars

* debug echo

* more debugging

* single line

* see if token exists in project dir

* try quotes

* fixed?

* diff to cmp

* double login

* log cmp result

* use quotes

* fixed issue

* try again

* whyyy

* unsilence gpg

* encrypt in root dir

* pass ref

* force=true

* try different action

* try different action

* [AUTO] OAuth token refresh #patch

* remove comments

* force push option

* [AUTO] OAuth token refresh #patch

* Update update.yml

* Bug: Fix Auth Flow / Add 2FA (#16)

* change env var names

* new auth flow

* fix 2fa

* add 2fa env var to workflow

* readme and auto release

* Feature: Get and Store Dividends (#14)

* script updates

* stricter branch cov & csv utils

* csv tests

* data dir

* typo

* fix test

* fix test data object

* uncomment tests

* yfinance import

* add local token update script

* symbols.csv

* script changes

* flake

* update cron schedule

* fix hist test?

* [AUTO] OAuth token refresh #patch

* comment out print statement

* only save hist purposefully

* only auto commit token

* yaml

* token from master

* readme

* readme again

* create datareader and datawriter classes

* update login script

* broker superclass tangent

* reorganization

* flake8

* dividend function done

* try auto symbol

* try again

* [AUTO] Symbol update #patch

* #minor commit

* auto dividend

* fixed robinhood dot to yahoo dash bug

* add multiprocessing

* fixed filtering for dividend and splits tables

* fixed csv update bug + more robust dividend saving

* div update

* div update iex

* constants

* debug

* working on tests

* working broker tests

* remove yfinance dep

* broker tests done

* fix naming issue

* import

* working on marketdata tests

* finished marketdata tests

* fileops tests almost done

* done

* [AUTO] Symbol update #patch

* [AUTO] Symbol update #patch

* [AUTO] Symbol update #patch

* [AUTO] Symbol update #patch

* Feature: S3 Integration (#19)

* fix release?

* file walk

* storage file

* s3 dir uploader working as expected

* dividend s3 upload script complete

* dividend workflows and weird dividend bug

* floating point precision fix

* remove data dir

* move pathfinder to fileops

* s3 integration

* add functions to store class and finish most store tests

* finish store tests

* tests done?

* done?

* linting

* aws configure

* make path before saving

* circular import

* make path

* remove pathlib

* fix rename

* more rename fixes

* does this work

* done

* fixed workflows

* hotfix

* 💽 Feature: Cache pip dependencies 💽 (#20)

* refactor workflows

* pip cache

* add workflow badges to readme

* spacing

* more workflow and readme changes

* 💎 Feature: Add Polygon.io (Double Redundancy) 💎 (#23)

* still need to write get_splits fx

* made iexcloud subdirs

* start polygon class

* fix test

* init function

* switch to polygon api

* auto dep check

* polygon dividend redundancy added

* tests

* update env var in workflows

* secrets

* fixes?

* fix test

* done

* Bump python-dotenv from 0.13.0 to 0.14.0

Bumps [python-dotenv](https://github.com/theskumar/python-dotenv) from 0.13.0 to 0.14.0.
- [Release notes](https://github.com/theskumar/python-dotenv/releases)
- [Changelog](https://github.com/theskumar/python-dotenv/blob/master/CHANGELOG.md)
- [Commits](theskumar/python-dotenv@v0.13.0...v0.14.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: Krish Suchak <42231639+suchak1@users.noreply.github.com>
Co-authored-by: Krish Suchak <suchak.krish@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump boto3 from 1.14.35 to 1.14.46 (#27)

* Feature: init function (#7)

* new files

* create symbol / instrument lookup tables

* init or new?

* abstract a few functions out

* historical data

* update readme to reference roadmap

* update reqs

* newline for function

* Add logo (#8)

* add logo

* adding logo

* update gitignore

* update readme

* Create python-app.yml

* organization and linting

* pytest-check

* flatten tests

* flatten tests

* new tests

* test command update

* just run pytest

* init and copy token bash script

* env vars

* workflow

* secrets

* progress towards automation

* handled non ci case

* update token script almost done

* chmod +x

* make tokens dir

* split prebuild and postbuild scripts

* fix workflow

* script file path in workflow

* housecleaning

* get windows username on wsl

* new workflow

* rename workflows

* yml syntax

* schedule

* try again

* try again

* daily update

* update todo

* code coverage

* import sys

* lower coverage threshold

* test_load_portfolio

* auto versioning

* writing last tests

* tests done

* locked reqs

* wrong dotenv

* try again

* update function name

* fix interval def

* comment out csv creation

* yml descriptions

* 🐞 Bug: Commit New OAuth 🐞 (#12)

* debug workflow

* add email and password env vars

* debug echo

* more debugging

* single line

* see if token exists in project dir

* try quotes

* fixed?

* diff to cmp

* double login

* log cmp result

* use quotes

* fixed issue

* try again

* whyyy

* unsilence gpg

* encrypt in root dir

* pass ref

* force=true

* try different action

* try different action

* [AUTO] OAuth token refresh #patch

* remove comments

* force push option

* [AUTO] OAuth token refresh #patch

* Update update.yml

* Bug: Fix Auth Flow / Add 2FA (#16)

* change env var names

* new auth flow

* fix 2fa

* add 2fa env var to workflow

* readme and auto release

* Feature: Get and Store Dividends (#14)

* script updates

* stricter branch cov & csv utils

* csv tests

* data dir

* typo

* fix test

* fix test data object

* uncomment tests

* yfinance import

* add local token update script

* symbols.csv

* script changes

* flake

* update cron schedule

* fix hist test?

* [AUTO] OAuth token refresh #patch

* comment out print statement

* only save hist purposefully

* only auto commit token

* yaml

* token from master

* readme

* readme again

* create datareader and datawriter classes

* update login script

* broker superclass tangent

* reorganization

* flake8

* dividend function done

* try auto symbol

* try again

* [AUTO] Symbol update #patch

* #minor commit

* auto dividend

* fixed robinhood dot to yahoo dash bug

* add multiprocessing

* fixed filtering for dividend and splits tables

* fixed csv update bug + more robust dividend saving

* div update

* div update iex

* constants

* debug

* working on tests

* working broker tests

* remove yfinance dep

* broker tests done

* fix naming issue

* import

* working on marketdata tests

* finished marketdata tests

* fileops tests almost done

* done

* [AUTO] Symbol update #patch

* [AUTO] Symbol update #patch

* [AUTO] Symbol update #patch

* [AUTO] Symbol update #patch

* Feature: S3 Integration (#19)

* fix release?

* file walk

* storage file

* s3 dir uploader working as expected

* dividend s3 upload script complete

* dividend workflows and weird dividend bug

* floating point precision fix

* remove data dir

* move pathfinder to fileops

* s3 integration

* add functions to store class and finish most store tests

* finish store tests

* tests done?

* done?

* linting

* aws configure

* make path before saving

* circular import

* make path

* remove pathlib

* fix rename

* more rename fixes

* does this work

* done

* fixed workflows

* hotfix

* 💽 Feature: Cache pip dependencies 💽 (#20)

* refactor workflows

* pip cache

* add workflow badges to readme

* spacing

* more workflow and readme changes

* 💎 Feature: Add Polygon.io (Double Redundancy) 💎 (#23)

* still need to write get_splits fx

* made iexcloud subdirs

* start polygon class

* fix test

* init function

* switch to polygon api

* auto dep check

* polygon dividend redundancy added

* tests

* update env var in workflows

* secrets

* fixes?

* fix test

* done

* Bump boto3 from 1.14.35 to 1.14.46

Bumps [boto3](https://github.com/boto/boto3) from 1.14.35 to 1.14.46.
- [Release notes](https://github.com/boto/boto3/releases)
- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)
- [Commits](boto/boto3@1.14.35...1.14.46)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: Krish Suchak <42231639+suchak1@users.noreply.github.com>
Co-authored-by: Krish Suchak <suchak.krish@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* 🤖 fix(auto): dependabot race condition 🤖 (#30)

* Feature: init function (#7)

* new files

* create symbol / instrument lookup tables

* init or new?

* abstract a few functions out

* historical data

* update readme to reference roadmap

* update reqs

* newline for function

* Add logo (#8)

* add logo

* adding logo

* update gitignore

* update readme

* Create python-app.yml

* organization and linting

* pytest-check

* flatten tests

* flatten tests

* new tests

* test command update

* just run pytest

* init and copy token bash script

* env vars

* workflow

* secrets

* progress towards automation

* handled non ci case

* update token script almost done

* chmod +x

* make tokens dir

* split prebuild and postbuild scripts

* fix workflow

* script file path in workflow

* housecleaning

* get windows username on wsl

* new workflow

* rename workflows

* yml syntax

* schedule

* try again

* try again

* daily update

* update todo

* code coverage

* import sys

* lower coverage threshold

* test_load_portfolio

* auto versioning

* writing last tests

* tests done

* locked reqs

* wrong dotenv

* try again

* update function name

* fix interval def

* comment out csv creation

* yml descriptions

* 🐞 Bug: Commit New OAuth 🐞 (#12)

* debug workflow

* add email and password env vars

* debug echo

* more debugging

* single line

* see if token exists in project dir

* try quotes

* fixed?

* diff to cmp

* double login

* log cmp result

* use quotes

* fixed issue

* try again

* whyyy

* unsilence gpg

* encrypt in root dir

* pass ref

* force=true

* try different action

* try different action

* [AUTO] OAuth token refresh #patch

* remove comments

* force push option

* [AUTO] OAuth token refresh #patch

* Update update.yml

* Bug: Fix Auth Flow / Add 2FA (#16)

* change env var names

* new auth flow

* fix 2fa

* add 2fa env var to workflow

* readme and auto release

* Feature: Get and Store Dividends (#14)

* script updates

* stricter branch cov & csv utils

* csv tests

* data dir

* typo

* fix test

* fix test data object

* uncomment tests

* yfinance import

* add local token update script

* symbols.csv

* script changes

* flake

* update cron schedule

* fix hist test?

* [AUTO] OAuth token refresh #patch

* comment out print statement

* only save hist purposefully

* only auto commit token

* yaml

* token from master

* readme

* readme again

* create datareader and datawriter classes

* update login script

* broker superclass tangent

* reorganization

* flake8

* dividend function done

* try auto symbol

* try again

* [AUTO] Symbol update #patch

* #minor commit

* auto dividend

* fixed robinhood dot to yahoo dash bug

* add multiprocessing

* fixed filtering for dividend and splits tables

* fixed csv update bug + more robust dividend saving

* div update

* div update iex

* constants

* debug

* working on tests

* working broker tests

* remove yfinance dep

* broker tests done

* fix naming issue

* import

* working on marketdata tests

* finished marketdata tests

* fileops tests almost done

* done

* [AUTO] Symbol update #patch

* [AUTO] Symbol update #patch

* [AUTO] Symbol update #patch

* [AUTO] Symbol update #patch

* Feature: S3 Integration (#19)

* fix release?

* file walk

* storage file

* s3 dir uploader working as expected

* dividend s3 upload script complete

* dividend workflows and weird dividend bug

* floating point precision fix

* remove data dir

* move pathfinder to fileops

* s3 integration

* add functions to store class and finish most store tests

* finish store tests

* tests done?

* done?

* linting

* aws configure

* make path before saving

* circular import

* make path

* remove pathlib

* fix rename

* more rename fixes

* does this work

* done

* fixed workflows

* hotfix

* 💽 Feature: Cache pip dependencies 💽 (#20)

* refactor workflows

* pip cache

* add workflow badges to readme

* spacing

* more workflow and readme changes

* still need to write get_splits fx

* fix save_symbols test

* fixing tests

* tests fixed?

* run nonce

* kwargs

* save_symbols fix

* run id

* again

* 🎊 feat(data): splits 🎊 (#32)

* Feature: init function (#7)

* new files

* create symbol / instrument lookup tables

* init or new?

* abstract a few functions out

* historical data

* update readme to reference roadmap

* update reqs

* newline for function

* Add logo (#8)

* add logo

* adding logo

* update gitignore

* update readme

* Create python-app.yml

* organization and linting

* pytest-check

* flatten tests

* flatten tests

* new tests

* test command update

* just run pytest

* init and copy token bash script

* env vars

* workflow

* secrets

* progress towards automation

* handled non ci case

* update token script almost done

* chmod +x

* make tokens dir

* split prebuild and postbuild scripts

* fix workflow

* script file path in workflow

* housecleaning

* get windows username on wsl

* new workflow

* rename workflows

* yml syntax

* schedule

* try again

* try again

* daily update

* update todo

* code coverage

* import sys

* lower coverage threshold

* test_load_portfolio

* auto versioning

* writing last tests

* tests done

* locked reqs

* wrong dotenv

* try again

* update function name

* fix interval def

* comment out csv creation

* yml descriptions

* 🐞 Bug: Commit New OAuth 🐞 (#12)

* debug workflow

* add email and password env vars

* debug echo

* more debugging

* single line

* see if token exists in project dir

* try quotes

* fixed?

* diff to cmp

* double login

* log cmp result

* use quotes

* fixed issue

* try again

* whyyy

* unsilence gpg

* encrypt in root dir

* pass ref

* force=true

* try different action

* try different action

* [AUTO] OAuth token refresh #patch

* remove comments

* force push option

* [AUTO] OAuth token refresh #patch

* Update update.yml

* Bug: Fix Auth Flow / Add 2FA (#16)

* change env var names

* new auth flow

* fix 2fa

* add 2fa env var to workflow

* readme and auto release

* Feature: Get and Store Dividends (#14)

* script updates

* stricter branch cov & csv utils

* csv tests

* data dir

* typo

* fix test

* fix test data object

* uncomment tests

* yfinance import

* add local token update script

* symbols.csv

* script changes

* flake

* update cron schedule

* fix hist test?

* [AUTO] OAuth token refresh #patch

* comment out print statement

* only save hist purposefully

* only auto commit token

* yaml

* token from master

* readme

* readme again

* create datareader and datawriter classes

* update login script

* broker superclass tangent

* reorganization

* flake8

* dividend function done

* try auto symbol

* try again

* [AUTO] Symbol update #patch

* #minor commit

* auto dividend

* fixed robinhood dot to yahoo dash bug

* add multiprocessing

* fixed filtering for dividend and splits tables

* fixed csv update bug + more robust dividend saving

* div update

* div update iex

* constants

* debug

* working on tests

* working broker tests

* remove yfinance dep

* broker tests done

* fix naming issue

* import

* working on marketdata tests

* finished marketdata tests

* fileops tests almost done

* done

* [AUTO] Symbol update #patch

* [AUTO] Symbol update #patch

* [AUTO] Symbol update #patch

* [AUTO] Symbol update #patch

* Feature: S3 Integration (#19)

* fix release?

* file walk

* storage file

* s3 dir uploader working as expected

* dividend s3 upload script complete

* dividend workflows and weird dividend bug

* floating point precision fix

* remove data dir

* move pathfinder to fileops

* s3 integration

* add functions to store class and finish most store tests

* finish store tests

* tests done?

* done?

* linting

* aws configure

* make path before saving

* circular import

* make path

* remove pathlib

* fix rename

* more rename fixes

* does this work

* done

* fixed workflows

* hotfix

* 💽 Feature: Cache pip dependencies 💽 (#20)

* refactor workflows

* pip cache

* add workflow badges to readme

* spacing

* more workflow and readme changes

* still need to write get_splits fx

* fix save_symbols test

* fixing tests

* tests fixed?

* iex get splits done

* splits functions done

* adding splits update to the pipeline

* writing tests

* higher order fx refactor and almost done w tests

* tests done?

* fix test_get_splits_path

* fix columns naming

* refactor

* try again

* Bump robin-stocks from 1.4.2 to 1.5.0 (#31)

Bumps [robin-stocks](https://github.com/jmfernandes/robin_stocks) from 1.4.2 to 1.5.0.
- [Release notes](https://github.com/jmfernandes/robin_stocks/releases)
- [Commits](https://github.com/jmfernandes/robin_stocks/commits)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump anothrNick/github-tag-action from 1.22.0 to 1.26.0 (#33)

Bumps [anothrNick/github-tag-action](https://github.com/anothrNick/github-tag-action) from 1.22.0 to 1.26.0.
- [Release notes](https://github.com/anothrNick/github-tag-action/releases)
- [Commits](anothrNick/github-tag-action@1.22.0...9aaabdb)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump boto3 from 1.14.46 to 1.15.5 (#38)

Bumps [boto3](https://github.com/boto/boto3) from 1.14.46 to 1.15.5.
- [Release notes](https://github.com/boto/boto3/releases)
- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)
- [Commits](boto/boto3@1.14.46...1.15.5)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump pandas from 1.1.0 to 1.1.2 (#39)

Bumps [pandas](https://github.com/pandas-dev/pandas) from 1.1.0 to 1.1.2.
- [Release notes](https://github.com/pandas-dev/pandas/releases)
- [Changelog](https://github.com/pandas-dev/pandas/blob/master/RELEASE.md)
- [Commits](pandas-dev/pandas@v1.1.0...v1.1.2)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump polygon-api-client from 0.1.6 to 0.1.8 (#36)

* Bump polygon-api-client from 0.1.6 to 0.1.8

Bumps [polygon-api-client](https://github.com/polygon-io/client-python) from 0.1.6 to 0.1.8.
- [Release notes](https://github.com/polygon-io/client-python/releases)
- [Commits](polygon-io/client-python@v0.1.6...v0.1.8)

Signed-off-by: dependabot[bot] <support@github.com>

* Update requirements.txt

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Krish Suchak <42231639+suchak1@users.noreply.github.com>

* 🐦 feat(data): social sentiment 🐦 (#40)

* resolved

* get_ohlc iexcloud

* dep updates

* get_ohlc polygon

* refactor ohlc

* stocktwits

* comment

* progress

* scripts

* finishing up implementation

* workflows

* fixed

* update_sentiment mods

* done

* pipeline env vars

* changes

* fixed

* pls

* 🐛 fix(auto): sentiment schedule 🐛 (#42)

* resolved

* fixed

* workflow name

* fix readme

* 💾 fix(data): save functions 💾 (#45)

* resolved

* deps

* fix

* temp fix

* try again

* 🏖️  fix(test): temp IEX sandbox fix 🏖️ (#46)

* resolved

* deps

* fix

* temp fix

* try again

* notlikethis

* 📈 feat(data): ohlc 📈 (#47)

* resolved

* get_ohlc iexcloud

* dep updates

* get_ohlc polygon

* refactor ohlc

* stocktwits

* comment

* progress

* scripts

* finishing up implementation

* workflows

* semantics

* undo sandbox changes

* upgrade rh

* delete extra files

* add get_ohlc tests

* add standardize_ohlc test

* add save_ohlc test

* add get_ohlc for iex cloud

* add get_ohlc for polygon

* fix get_ohlc for polygon

* add get_ohlc_path test

* add save_ohlc to marketdata

* add get_ohlc to marketdata

* comment out unused code

* lower test coverage threshold from 95 to 90

* 📱 feat(data): crypto social sentiment 📱 (#48)

* resolved

* add mergers to planned data collection in readme

* add crypto tickers to sentiment update

* readme

* ✋ fix(test): stop uploading test data during repo upload ✋ (#49)

* resolved

* add mergers to planned data collection in readme

* done

* 🧵 Ffeat(data): multiprocessing update 🧵 (#50)

* resolved

* add mergers to planned data collection in readme

* add get_intraday_path

* comments

* add institutional sentiment to readme data checklista

* to compile

* limit polygon save timeframe to prevent retroactive weirdness

* add multiprocessing to speed up data updates

* delete local data files after each file update to preserve disk space

* linting

* if

* ❌ chore: rename repo ❌ (#51)

* resolved

* add mergers to planned data collection in readme

* add get_intraday_path

* comments

* add institutional sentiment to readme data checklista

* to compile

* limit polygon save timeframe to prevent retroactive weirdness

* add multiprocessing to speed up data updates

* delete local data files after each file update to preserve disk space

* constants tests done

* conflict

* conflict

* polygon test_intraday

* iex test_intraday

* test_save_intraday + repo rename

* readme

* fix tests

* 💱 feat(data): crypto ohlc + 3 retries 💱 (#52)

* making progress

* try again function

* iex div and splits retry

* poly div and splits retry

* iex ohlc retry

* tests updated

* fix try again

* bugs

* fix tests

* fix retries for tests

* add polygon key

* timezone fix

* 🌲 fix: env var issue 🌲 (#53)

* 📦 chore(auto): publish package on pypi 📦 (#58)

* done?

* upgrade packages

* 🥡 fix(auto): pypi package upload 🥡 (#60)

* done?

* upgrade packages

* fix?

* python

* ⚖️ feat(data): volume weighted average ⚖️ (#61)

* resolved

* add mergers to planned data collection in readme

* add get_intraday_path

* comments

* add institutional sentiment to readme data checklista

* to compile

* limit polygon save timeframe to prevent retroactive weirdness

* add multiprocessing to speed up data updates

* delete local data files after each file update to preserve disk space

* constants tests done

* conflict

* conflict

* polygon test_intraday

* iex test_intraday

* test_save_intraday + repo rename

* readme

* git cleanup local branches

* package name change

* remove package upload in dev pipeline

* fix bugs and add volume weighted avg price to ohlc

* tests

* lint

* lint again

* pandas warning

* fix date polygon

* 🔢 feat(data): number of trades 🔢 (#66)

* update deps

* change symbol?

* remove error log

* debug

* update hist scripts to get s3 truth

* bug fix

* 📅 feat(data): intraday 📅 (#71)

* resolved

* add mergers to planned data collection in readme

* add get_intraday_path

* comments

* add institutional sentiment to readme data checklista

* to compile

* limit polygon save timeframe to prevent retroactive weirdness

* add multiprocessing to speed up data updates

* delete local data files after each file update to preserve disk space

* constants tests done

* conflict

* conflict

* polygon test_intraday

* iex test_intraday

* test_save_intraday + repo rename

* readme

* git cleanup local branches

* package name change

* remove package upload in dev pipeline

* fix bugs and add volume weighted avg price to ohlc

* start generator

* polygon intraday done?

* more progrress

* fix polygon, still need to do iex

* finish iex changes

* add intraday step to dev pipeline and fix tests

* fixed?

* last fix?

* get date from first row of intra to save properly

* more inclusive time range and lower threshold

* 🕰️ fix(data): stock datetime formatting 🕰️ (#72)

* fix formatting

* try again

* lint

* 💱 fix(data): crypto datetime 💱 (#76)

* stop using eod ohlc path for standardize intraday

* missing comma

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
  • Loading branch information
suchak1 and dependabot[bot] authored Jan 13, 2021
1 parent bcb9af7 commit ccdba52
Show file tree
Hide file tree
Showing 3 changed files with 24 additions and 19 deletions.
6 changes: 3 additions & 3 deletions scripts/update_hist_intraday.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
stock_symbols = poly_stocks.get_symbols()
stock_symbols = stock_symbols[stock_symbols.index('DIS')+1:]
# [250:]
crypto_symbols = POLY_CRYPTO_SYMBOLS
crypto_symbols = [POLY_CRYPTO_SYMBOLS[0]]


def update_poly_stocks_intraday():
Expand All @@ -38,8 +38,8 @@ def update_poly_crypto_intraday():
filenames = []
try:
filenames = poly_crypto.save_intraday(
symbol=symbol, timeframe='2y',
delay=POLY_CRYPTO_DELAY, retries=1)
symbol=symbol, timeframe='674d',
delay=POLY_CRYPTO_DELAY, retries=2)
except Exception as e:
print(f'Polygon.io intraday update failed for {symbol}.')
print(e)
Expand Down
1 change: 0 additions & 1 deletion scripts/update_hist_ohlc.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,6 @@ def update_poly_stocks_ohlc():


def update_poly_crypto_ohlc():

for idx, symbol in enumerate(crypto_symbols):
filename = PathFinder().get_ohlc_path(
symbol=symbol, provider=poly_crypto.provider)
Expand Down
36 changes: 21 additions & 15 deletions src/DataSource.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,11 +48,11 @@ def get_dividends(self, symbol, timeframe='max'):
filtered = self.reader.data_in_timeframe(df, C.EX, timeframe)
return filtered

def standardize(self, symbol, df, full_mapping, fx, columns, default):
def standardize(self, df, full_mapping,
filename, columns, default):
mapping = {k: v for k, v in full_mapping.items() if k in df}

df = df[list(mapping)].rename(columns=mapping)
filename = fx(symbol, self.provider)
time_col, val_cols = columns[0], columns[1:]

if time_col in df and set(val_cols).issubset(df.columns):
Expand All @@ -73,11 +73,11 @@ def standardize_dividends(self, symbol, df):
[C.EX, C.PAY, C.DEC, C.DIV]
)
)
filename = self.finder.get_dividends_path(symbol, self.provider)
return self.standardize(
symbol,
df,
full_mapping,
self.finder.get_dividends_path,
filename,
[C.EX, C.DIV],
0
)
Expand Down Expand Up @@ -108,11 +108,11 @@ def standardize_splits(self, symbol, df):
[C.EX, C.PAY, C.DEC, C.RATIO]
)
)
filename = self.finder.get_splits_path(symbol, self.provider)
return self.standardize(
symbol,
df,
full_mapping,
self.finder.get_splits_path,
filename,
[C.EX, C.RATIO],
1
)
Expand All @@ -129,7 +129,7 @@ def save_splits(self, **kwargs):
if os.path.exists(filename):
return filename

def standardize_ohlc(self, symbol, df):
def standardize_ohlc(self, symbol, df, filename=None):
full_mapping = dict(
zip(
['date', 'open', 'high', 'low', 'close',
Expand All @@ -138,11 +138,13 @@ def standardize_ohlc(self, symbol, df):
C.VOL, C.AVG, C.TRADES]
)
)

filename = filename or self.finder.get_ohlc_path(symbol, self.provider)

df = self.standardize(
symbol,
df,
full_mapping,
self.finder.get_ohlc_path,
filename,
[C.TIME, C.OPEN, C.HIGH, C.LOW, C.CLOSE],
0
)
Expand Down Expand Up @@ -222,11 +224,11 @@ def standardize_sentiment(self, symbol, df):
[C.TIME, C.POS, C.NEG]
)
)
filename = self.finder.get_sentiment_path(symbol, self.provider)
df = self.standardize(
symbol,
df,
full_mapping,
self.finder.get_sentiment_path,
filename,
[C.TIME, C.POS, C.NEG],
0
)
Expand All @@ -239,11 +241,11 @@ def standardize_volume(self, symbol, df):
[C.TIME, C.VOL, C.DELTA]
)
)
filename = self.finder.get_sentiment_path(symbol, self.provider)
df = self.standardize(
symbol,
df,
full_mapping,
self.finder.get_sentiment_path,
filename,
[C.TIME, C.VOL, C.DELTA],
0
)
Expand Down Expand Up @@ -469,7 +471,9 @@ def _get_intraday(symbol, min=1, timeframe='max', extra_hrs=True):
df = pd.DataFrame(data)[res_cols].rename(columns=columns)
df['date'] = pd.to_datetime(df['date'] + ' ' + df['minute'])
df.drop(columns='minute', inplace=True)
df = self.standardize_ohlc(symbol, df)
filename = self.finder.get_intraday_path(
symbol, date, self.provider)
df = self.standardize_ohlc(symbol, df, filename)
yield self.reader.data_in_timeframe(
df, C.TIME, timeframe
)
Expand Down Expand Up @@ -562,7 +566,9 @@ def _get_intraday(symbol, min=1, timeframe='max', extra_hrs=True):
df['date'], unit='ms').dt.tz_localize(
'UTC').dt.tz_convert(
C.TZ).dt.tz_localize(None)
df = self.standardize_ohlc(symbol, df)
filename = self.finder.get_intraday_path(
symbol, date, self.provider)
df = self.standardize_ohlc(symbol, df, filename)
df = df[df[C.TIME].dt.strftime(C.DATE_FMT) == date]
yield self.reader.data_in_timeframe(
df, C.TIME, timeframe
Expand Down

0 comments on commit ccdba52

Please sign in to comment.