Rekt pps? problem? => piker.accounting
#489
Conversation
goodboy
force-pushed
the
rekt_pps
branch
3 times, most recently
from
3fa212f
to
fa3cdbe
Compare
|
Lul, gotta drop that |
goodboy
force-pushed
the
rekt_pps
branch
2 times, most recently
from
2e2f49e
to
c913adc
Compare
To be compat with the `Symbol` (for now) and generally allow for reading the (derivative) contract specific part of the fqme. Adjust `contract_info: list[str]` and make `src: str = ''` by default.
Initial attempt at getting the sampling and shm layer to use the new mkt info meta-data type. Draft out a potential `BackendInitMsg: msgspec.Struct` for validating the init msg returned from the `stream_quotes()` start value; obvs don't actually use it yet.
To make nested `msgspec.Struct`s work we need to tell the codec that the `.symbol` is some struct def, since we don't really need to enforce that (yet) we're just going to enc/dec as `str` until we further formalize and/or need something more complex.
| # TODO: instead see if we can hack tomli and tomli-w to do the same: | ||
| # - https://github.com/hukkin/tomli | ||
| # - https://github.com/hukkin/tomli-w | ||
| class PpsEncoder(toml.TomlEncoder): |
There was a problem hiding this comment.
entirely replaced by tomlkit
| conf_path: Path | ||
| conf, conf_path = config.load_account(brokername, acctid) | ||
|
|
||
| if brokername in conf: |
There was a problem hiding this comment.
this was originally for converting legacy account files: pps.toml but we can probably remove it in next patch set for this subsys.
| @@ -439,6 +519,35 @@ async def get_ohlc( | |||
| yield get_ohlc, {'erlangs': 3, 'rate': 3} | |||
|
|
|||
|
|
|||
| @async_lifo_cache() | |||
| async def get_mkt_info( | |||
There was a problem hiding this comment.
this is the new required endpoint for all backends which delivers the MktPair (ideally with embedded .src/.dst attrs which are Assets) translation for any searched market/symbol/ticker pattern.
| @@ -713,6 +771,98 @@ def normalize( | |||
| return data | |||
|
|
|||
|
|
|||
| @async_lifo_cache() | |||
| async def get_mkt_info( | |||
There was a problem hiding this comment.
I'm the least happy with this implementation (in terms of existing backends) just because as a legacy market broker ib tends to assume a source asset for every exchange..
Ideally we can work toward getting this more refined going forward particularly if we're going to be re-doing our tsdb subsys backend choice(s) and don't have to rely on the table keys in the existing replicated data sets..
longer run i would like to see mnq/usd.cme.ib over no usd in the fqme.
| # but likely we want all backends to eventually handle | ||
| # ``dst/src.venue.`` style !? | ||
| src: str | Asset = '' | ||
| if atype == 'forex': |
There was a problem hiding this comment.
bleh, yeah we need some backend specific tests i think to be able to get this to the new proper fqme nomenclature i'm thinking...
goodboy
force-pushed
the
rekt_pps
branch
3 times, most recently
from
c72b998
to
7011187
Compare
|
Ok so as per discussion in dev chat I'm going to cowboy land this massive change set without extensive review bc,
Hopefully this won't happen again 😹 |
Even moar fixes after #470, #475, #484 which mega broke the old
pps.tomlmanagement for live backend 💥 😶🌫️ 🤕Turns out this might be the reason to start assaulting the revamp ofprolly positioning: the accounting of "what/when (is) owned"
our accounting tooling as per our now long list of desirables:
accounting
Testing enhancements (some tasks deferred)
rework test set to use
pytest.mark.parametrizeso we can moreeasily use common harness logic for darks and alert testing.
reorg the ahab API docker supervisor endpoints to be decoupled
from
open_pikerd()and able to accept arbitrary config overrides forboth testing and distributed setups.
dark order and alert clearinglines
add a basic accounting of position only test set?tasks and bugs getting deferred..
pre-existing live order loading is borked on first chart edit from
order mode: Editting initial live order load fails? #502
final removal of
Symboland.symbolsemantics -> summarized in Final removal of.accounting._mktinfo.Symbol#517order_mode.symbolfinal usage..?make a follow up issue to add a
Flume.Cascadetype for fspswhich describes stream IO between flumes:
deribit still need full
get_mkt_info()andFeedInitsync up the new
data.validatestuff with old brokercheck idea inBackend test suite #387, Brokercheck an all backends #393 -> add an actual test?
dependency changes needed for this to land
move to new
TOMLlibs (tomlkitfor styling andtomli-wforfast writing) B)
tomlwriting hacks we have into a new submod..tomldep and go withtomli,tomli-wand
tomlkitB)tomlkitwhich does support writing inlinetables (as well as lotsa other great stuff) as requested in Better
TOMLlibs ->tomlkitfor style,tomli(-w)for speed #496 B)(https://github.com/pikers/tomlkit/tree/piker_pin)
DecimalandMutableMappingcodecs, add a multiline-Arrayindent control python-poetry/tomlkit#294Contextsemantics for cross-actor-task cancellation and overruns goodboy/tractor#357Unrelated to
.accountingchanges.data._web_bsimplementation to avoid the useof backend code using async generators within async context managers.
too many devs are too busy to review it rn so there's not a lot of
point 😂
NEW SUBSYSTEM: the
piker.accounting(subpkg) 🏄move all
.data._sourceandpiker.ppmod contents into a muchbetter organized subsystem API ->
piker.accounting🏄._mktinfocontainsMktPairandAssetprimitive as well as new.fqmeacronym (fully qualified market endpoint) processing.._poscontainsPositionandPpTableloading and processing._ledgeraddsTransactionLedgerandTransactionloading andsync.
.cliadds aledger <broker>.<accountname>syncing prototype cliwhich can be used to dump position summaries to console.
try to see how many bullets we can nail from Positioning system refinements: paper engine, file writing, toml style and perf #345
offer
.write_config()to (the new)TransactionLedgerandPpTablefor immediate writes.further follow ups: Change
Position.clears->.events#510, Offline *executable* storage format #511, Move totyperfor all clis #513, Position accounting refinements, bug fixes, src-asset-pnls,ledgerddaemon #515drop the singleton
pps.tomlinstead breaking into multiple account filesper backend,
$HOME/.config/piker/accounting/account.<brokername>.<(sub)accountname>.tomlaccount.binance.paper.toml,account.kraken.spot.tomlwhich are named from the accounts listed in a user's
brokers.tomlB)table schema: i.e. no longer any
kraken.paperin any section keys.Prototype
ledgercli usingtyperwith follow up to come with Ledger (syncing) CLI and offline toolset #512typerandrichas deps!Move our data feed layer to new
MktPairmarket info struct aswell as support a new "backend validation system" using
the new
.data.validate.FeedInitwe now expect
stream_quotes()to deliver a pedanticlist[FeedInit]onstartup instead of a random
dictof some schema 🏄all backends should provide a
get_mkt_info()endpoint now whichdoes the work of constructing a
MktPairfrom an input searchpattern (
str) and is also expected to construct theMktPair.src/
.dstfields which in the ideal case are.accounting._mktinfo.Assetrefs.move all internal usage of the old
Symboltype over to this aswell as rename as much as we can of
fqsnto the newfqme(fullyqualified market endpoint)
Flume.symbol->.mkt: MktPairPosition.symbol->.mkt_chart,_display_fsp.. etc.ui._chart.LinkedSplits.symboloutright and finally!port all backends as much as is possible to this new interface
.data.validatesince it should bedefault now B)
implementation, but i don't have a live feed..
questrade: use ws api for live rt quotes #509bring in backend checks from Brokercheck an all backends #393 and add to a test set for
superficial audit of all broker/data providers.
data.validatestuff and not sure we really need explicit test set?
Outstanding/Related issues/bugs that should be resolved here:
kraken: balance <-> position disparities with crypto/crypto pairs #373 closed for Position accounting refinements, bug fixes, src-asset-pnls,ledgerddaemon #515Pps toml header leaking after zeroing positions #472 fixed here and prior
Better
TOMLlibs ->tomlkitfor style,tomli(-w)for speed #496ib forex is currently borked since the move to
MktPair(b44b091)behave as before!
burning a shm allocation...
type matching.
kucoinborked on search result feed selection-load. (ff74d47)discovered BUGs tofix, issues likely to re-issued and defer, new feature ideas:
krakenposition tracking gotchas #263 -> moved to Position accounting refinements, bug fixes, src-asset-pnls,ledgerddaemon #515extra follow ToDos (birds with stones):
follow up
.accountingsubsys summary refinements and bug fixesin Position accounting refinements, bug fixes, src-asset-pnls,
ledgerddaemon #515EMS layer: unify the messaging layer to a single msg set: Unify EMS messaging set #514
CLI
piker ledger (sync)development and nice-to-havestyperfor all CLIs: Move totyperfor all clis #513