QueryLanguageParser regression in the handling of inequalities inside of `update_view` #5998

chipkent · 2024-08-28T18:04:18Z

I had a community user report that deephaven-ib example scripts were no longer working. After digging in, Deephaven seems to have introduced a regression in one of the recent releases. The following example works in 0.34 but fails in 0.35.

# Run this example in the Deephaven IDE Console

from ibapi.contract import Contract
from ibapi.order import Order

import deephaven_ib as dhib
from deephaven.updateby import ema_time, emstd_time
from deephaven import time_table
from deephaven.plot import Figure
from deephaven.plot.selectable_dataset import one_click
from deephaven.plot import PlotStyle


###########################################################################
# WARNING: THIS SCRIPT EXECUTES TRADES!! ONLY USE ON PAPER TRADING ACCOUNTS
###########################################################################

print("==============================================================================================================")
print("==== Create a client and connect.")
print("==== ** Accept the connection in TWS **")
print("==============================================================================================================")

client = dhib.IbSessionTws(host="localhost", port=7497, client_id=0, download_short_rates=False, read_only=False)
print(f"IsConnected: {client.is_connected()}")

client.connect()
print(f"IsConnected: {client.is_connected()}")

## Setup

account = "DU4943848"
max_position_dollars = 10000.0
em_time = "PT00:02:00"

errors = client.tables["errors"]
requests = client.tables["requests"]
positions = client.tables["accounts_positions"].where("Account = account")
ticks_bid_ask = client.tables["ticks_bid_ask"]
orders_submitted = client.tables["orders_submitted"].where("Account = account")
orders_status = client.tables["orders_status"]
orders_exec_details = client.tables["orders_exec_details"].where("Account = account")

print("==============================================================================================================")
print("==== Request data.")
print("==============================================================================================================")

registered_contracts_data = {}
registred_contracts_orders = {}

def add_contract(symbol: str, exchange: str="SMART") -> None:
    """
    Configure a contract for trading.
    :param symbol: Symbol to trade.
    :param exchange: exchange where orders get routed.
    :return: None
    """

    contract = Contract()
    contract.symbol = symbol
    contract.secType = "STK"
    contract.currency = "USD"
    contract.exchange = "SMART"

    rc = client.get_registered_contract(contract)
    id = rc.contract_details[0].contract.conId
    registered_contracts_data[id] = rc
    client.request_tick_data_realtime(rc, dhib.TickDataType.BID_ASK)
    print(f"Registered contract: id={id} rc={rc}")

    if exchange != "SMART":
        contract.exchange = "NYSE"
        rc = client.get_registered_contract(contract)

    registred_contracts_orders[id] = rc
    print(f"Registered contract: id={id} rc={rc}")


add_contract("GOOG")
add_contract("BAC")
add_contract("AAPL", exchange="NYSE")

print("==============================================================================================================")
print("==== Compute predictions.")
print("==============================================================================================================")

preds = ticks_bid_ask \
    .update_view(["MidPrice=0.5*(BidPrice+AskPrice)"]) \
    .update_by([
        ema_time("Timestamp", em_time, ["PredPrice=MidPrice"]),
        emstd_time("Timestamp", em_time, ["PredSD=MidPrice"]),
    ], by="Symbol") \
    .view([
        "ReceiveTime",
        "Timestamp",
        "ContractId",
        "Symbol",
        "BidPrice",
        "AskPrice",
        "MidPrice",
        "PredPrice",
        "PredSD",
        "PredLow=PredPrice-PredSD",
        "PredHigh=PredPrice+PredSD",
    ])

preds_start = preds.first_by("Symbol").view(["Symbol", "Timestamp"])
preds = preds.natural_join(preds_start, on="Symbol", joins="TimestampFirst=Timestamp")

preds_one_click = one_click(preds, by=["Symbol"], require_all_filters=True)

preds_plot = Figure() \
    .plot_xy("BidPrice", t=preds_one_click, x="Timestamp", y="BidPrice") \
    .plot_xy("AskPrice", t=preds_one_click, x="Timestamp", y="AskPrice") \
    .plot_xy("MidPrice", t=preds_one_click, x="Timestamp", y="MidPrice") \
    .plot_xy("PredPrice", t=preds_one_click, x="Timestamp", y="PredPrice") \
    .plot_xy("PredLow", t=preds_one_click, x="Timestamp", y="PredLow") \
    .plot_xy("PredHigh", t=preds_one_click, x="Timestamp", y="PredHigh") \
    .show()

print("==============================================================================================================")
print("==== Generate orders.")
print("==============================================================================================================")

open_orders = {}

def update_orders(contract_id: int, pred_low: float, pred_high: float, buy_order: bool, sell_order:bool) -> int:
    """
    Update orders on a contract.  First existing orders are canceled.  Then new buy/sell limit orders are placed.

    :param contract_id: Contract id.
    :param pred_low: Price for buy limit orders.
    :param pred_high: Price for sell limit orders.
    :param buy_order: True to post a buy order; False to not post a buy order.
    :param sell_order: True to post a sell order; False to not post a sell order.
    :return: Number of orders submitted.
    """

    if contract_id in open_orders:
        for order in open_orders[contract_id]:
            # print(f"Canceling order: contract_id={contract_id} order_id={order.request_id}")
            order.cancel()

    new_orders = []
    rc = registred_contracts_orders[contract_id]

    if sell_order:
        order_sell = Order()
        order_sell.account = account
        order_sell.action = "SELL"
        order_sell.orderType = "LIMIT"
        order_sell.totalQuantity = 100
        order_sell.lmtPrice = round( pred_high, 2)
        order_sell.transmit = True

        order = client.order_place(rc, order_sell)
        new_orders.append(order)

    if buy_order:
        order_buy = Order()
        order_buy.account = account
        order_buy.action = "BUY"
        order_buy.orderType = "LIMIT"
        order_buy.totalQuantity = 100
        order_buy.lmtPrice = round( pred_low, 2)
        order_buy.transmit = True

        order = client.order_place(rc, order_buy)
        new_orders.append(order)

    open_orders[contract_id] = new_orders
    return len(new_orders)

orders = preds.last_by(["Symbol"]) \
    .snapshot_when(time_table("PT00:01:00"), stamp_cols="SnapTime=Timestamp") \
    .where(f"Timestamp > TimestampFirst + '{em_time}'") \
    .natural_join(positions, on="ContractId", joins="Position") \
    .update_view([
        "Position = replaceIfNull(Position, 0.0)",
        "PositionDollars = Position * MidPrice",
        "MaxPositionDollars = max_position_dollars",
        "BuyOrder = PositionDollars < MaxPositionDollars",
        "SellOrder = PositionDollars > -MaxPositionDollars",
    ]) \
    .update("NumNewOrders = (long)update_orders(ContractId, PredLow, PredHigh, BuyOrder, SellOrder)")

Exception:

r-Scheduler-Serial-1 | .c.ConsoleServiceGrpcImpl | Error running script: java.lang.RuntimeException: Error in Python interpreter:
Type: <class 'deephaven.dherror.DHError'>
Value: table update_view operation failed. : Assertion failed
Traceback (most recent call last):
  File "/Users/chip/dev/deephaven-ib/venv-release-dhib=0.34.1/lib/python3.12/site-packages/deephaven/table.py", line 821, in update_view
    return Table(j_table=self.j_table.updateView(*formulas))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: io.deephaven.engine.table.impl.select.FormulaCompilationException: Formula compilation error for: PositionDollars > -MaxPositionDollars
	at io.deephaven.engine.table.impl.select.DhFormulaColumn.initDef(DhFormulaColumn.java:216)
	at io.deephaven.engine.table.impl.select.SwitchColumn.initDef(SwitchColumn.java:64)
	at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer.create(SelectAndViewAnalyzer.java:124)
	at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer.create(SelectAndViewAnalyzer.java:73)
	at io.deephaven.engine.table.impl.QueryTable.lambda$viewOrUpdateView$40(QueryTable.java:1753)
	at io.deephaven.engine.table.impl.remote.ConstructSnapshot.callDataSnapshotFunction(ConstructSnapshot.java:1368)
	at io.deephaven.engine.table.impl.remote.ConstructSnapshot.callDataSnapshotFunction(ConstructSnapshot.java:1168)
	at io.deephaven.engine.table.impl.BaseTable.initializeWithSnapshot(BaseTable.java:1296)
	at io.deephaven.engine.table.impl.QueryTable.lambda$viewOrUpdateView$41(QueryTable.java:1751)
	at io.deephaven.engine.table.impl.perf.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:369)
	at io.deephaven.engine.table.impl.QueryTable.lambda$viewOrUpdateView$42(QueryTable.java:1745)
	at io.deephaven.engine.table.impl.QueryTable.memoizeResult(QueryTable.java:3639)
	at io.deephaven.engine.table.impl.QueryTable.viewOrUpdateView(QueryTable.java:1744)
	at io.deephaven.engine.table.impl.QueryTable.updateView(QueryTable.java:1730)
	at io.deephaven.engine.table.impl.QueryTable.updateView(QueryTable.java:100)
	at io.deephaven.api.TableOperationsDefaults.updateView(TableOperationsDefaults.java:87)
	at org.jpy.PyLib.executeCode(Native Method)
	at org.jpy.PyObject.executeCode(PyObject.java:138)
	at io.deephaven.engine.util.PythonEvaluatorJpy.evalScript(PythonEvaluatorJpy.java:73)
	at io.deephaven.integrations.python.PythonDeephavenSession.lambda$evaluate$1(PythonDeephavenSession.java:205)
	at io.deephaven.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:51)
	at io.deephaven.integrations.python.PythonDeephavenSession.evaluate(PythonDeephavenSession.java:205)
	at io.deephaven.engine.util.AbstractScriptSession.lambda$evaluateScript$0(AbstractScriptSession.java:163)
	at io.deephaven.engine.context.ExecutionContext.lambda$apply$0(ExecutionContext.java:196)
	at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:207)
	at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:195)
	at io.deephaven.engine.util.AbstractScriptSession.evaluateScript(AbstractScriptSession.java:163)
	at io.deephaven.engine.util.DelegatingScriptSession.evaluateScript(DelegatingScriptSession.java:72)
	at io.deephaven.engine.util.ScriptSession.evaluateScript(ScriptSession.java:75)
	at io.deephaven.server.console.ConsoleServiceGrpcImpl.lambda$executeCommand$4(ConsoleServiceGrpcImpl.java:191)
	at io.deephaven.server.session.SessionState$ExportBuilder.lambda$submit$2(SessionState.java:1537)
	at io.deephaven.server.session.SessionState$ExportObject.doExport(SessionState.java:995)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.deephaven.server.runner.scheduler.SchedulerModule$ThreadFactory.lambda$newThread$0(SchedulerModule.java:100)
	at java.base/java.lang.Thread.run(Thread.java:829)
caused by io.deephaven.engine.table.impl.lang.QueryLanguageParser$QueryLanguageParseException: 

Having trouble with the following expression:
Full expression           : PositionDollars > -MaxPositionDollars
Expression having trouble : 
Exception type            : io.deephaven.base.verify.AssertionFailure
Exception message         : Assertion failed: asserted ret.equals(result), instead ret == class java.lang.Double, result == double.

	at io.deephaven.base.verify.Assert.fail(Assert.java:100)
	at io.deephaven.base.verify.Assert.equals(Assert.java:1454)
	at io.deephaven.engine.table.impl.lang.QueryLanguageParser.visit(QueryLanguageParser.java:1696)
	at io.deephaven.engine.table.impl.lang.QueryLanguageParser.visit(QueryLanguageParser.java:133)
	at com.github.javaparser.ast.expr.UnaryExpr.accept(UnaryExpr.java:112)
	at io.deephaven.engine.table.impl.lang.QueryLanguageParser.getTypeWithCaching(QueryLanguageParser.java:1146)
	at io.deephaven.engine.table.impl.lang.QueryLanguageParser.visit(QueryLanguageParser.java:1536)
	at io.deephaven.engine.table.impl.lang.QueryLanguageParser.visit(QueryLanguageParser.java:133)
	at com.github.javaparser.ast.expr.BinaryExpr.accept(BinaryExpr.java:140)
	at io.deephaven.engine.table.impl.lang.QueryLanguageParser.<init>(QueryLanguageParser.java:293)
	at io.deephaven.engine.table.impl.lang.QueryLanguageParser.<init>(QueryLanguageParser.java:209)
	at io.deephaven.engine.table.impl.select.codegen.FormulaAnalyzer.parseFormula(FormulaAnalyzer.java:209)
	at io.deephaven.engine.table.impl.select.codegen.FormulaAnalyzer.parseFormula(FormulaAnalyzer.java:88)
	at io.deephaven.engine.table.impl.select.DhFormulaColumn.initDef(DhFormulaColumn.java:196)
	... 37 more


Line: 823
Namespace: update_view
File: /Users/chip/dev/deephaven-ib/venv-release-dhib=0.34.1/lib/python3.12/site-packages/deephaven/table.py
Traceback (most recent call last):
  File "<string>", line 5, in <module>
  File "/Users/chip/dev/deephaven-ib/venv-release-dhib=0.34.1/lib/python3.12/site-packages/deephaven/table.py", line 823, in update_view

	at org.jpy.PyLib.executeCode(Native Method)
	at org.jpy.PyObject.executeCode(PyObject.java:138)
	at io.deephaven.engine.util.PythonEvaluatorJpy.evalScript(PythonEvaluatorJpy.java:73)
	at io.deephaven.integrations.python.PythonDeephavenSession.lambda$evaluate$1(PythonDeephavenSession.java:205)
	at io.deephaven.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:51)
	at io.deephaven.integrations.python.PythonDeephavenSession.evaluate(PythonDeephavenSession.java:205)
	at io.deephaven.engine.util.AbstractScriptSession.lambda$evaluateScript$0(AbstractScriptSession.java:163)
	at io.deephaven.engine.context.ExecutionContext.lambda$apply$0(ExecutionContext.java:196)
	at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:207)
	at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:195)
	at io.deephaven.engine.util.AbstractScriptSession.evaluateScript(AbstractScriptSession.java:163)
	at io.deephaven.engine.util.DelegatingScriptSession.evaluateScript(DelegatingScriptSession.java:72)
	at io.deephaven.engine.util.ScriptSession.evaluateScript(ScriptSession.java:75)
	at io.deephaven.server.console.ConsoleServiceGrpcImpl.lambda$executeCommand$4(ConsoleServiceGrpcImpl.java:191)
	at io.deephaven.server.session.SessionState$ExportBuilder.lambda$submit$2(SessionState.java:1537)
	at io.deephaven.server.session.SessionState$ExportObject.doExport(SessionState.java:995)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.deephaven.server.runner.scheduler.SchedulerModule$ThreadFactory.lambda$newThread$0(SchedulerModule.java:100)
	at java.base/java.lang.Thread.run(Thread.java:829)

The code can be made to pass by changing:

orders = preds.last_by(["Symbol"]) \
    .snapshot_when(time_table("PT00:01:00"), stamp_cols="SnapTime=Timestamp") \
    .where(f"Timestamp > TimestampFirst + '{em_time}'") \
    .natural_join(positions, on="ContractId", joins="Position") \
    .update_view([
        "Position = replaceIfNull(Position, 0.0)",
        "PositionDollars = Position * MidPrice",
        "MaxPositionDollars = max_position_dollars",
        "BuyOrder = PositionDollars < MaxPositionDollars",
        "SellOrder = PositionDollars > -MaxPositionDollars",
    ]) \
    .update("NumNewOrders = (long)update_orders(ContractId, PredLow, PredHigh, BuyOrder, SellOrder)")

to

orders = preds.last_by(["Symbol"]) \
    .snapshot_when(time_table("PT00:01:00"), stamp_cols="SnapTime=Timestamp") \
    .where(f"Timestamp > TimestampFirst + '{em_time}'") \
    .natural_join(positions, on="ContractId", joins="Position") \
    .update_view([
        "Position = replaceIfNull(Position, 0.0)",
        "PositionDollars = Position * MidPrice",
        "MaxPositionDollars = max_position_dollars",
    ]) \
    .update_view([
        "BuyOrder = PositionDollars < MaxPositionDollars",
        "SellOrder = PositionDollars > -MaxPositionDollars",
    ]) \
    .update("NumNewOrders = (long)update_orders(ContractId, PredLow, PredHigh, BuyOrder, SellOrder)")

The text was updated successfully, but these errors were encountered:

…ore#5998

Modify examples to avoid update_view regression : deephaven/deephaven-core#5998

chipkent · 2024-08-28T18:28:34Z

The community user seemed to make this change to make it work:

        "SellOrder = PositionDollars > (-1 * MaxPositionDollars)",

* Modify examples to avoid a known bug (deephaven/deephaven-core#5998) * Make the examples easier to change the account number

chipkent · 2024-09-10T18:08:30Z

I broke the failing part of the query up:

o1 = preds.last_by(["Symbol"]) \
    .snapshot_when(time_table("PT00:01:00"), stamp_cols="SnapTime=Timestamp") \
    .where(f"Timestamp > TimestampFirst + '{em_time}'") \
    .natural_join(positions, on="ContractId", joins="Position")

o1m = o1.meta_table

o2 = o1.update_view([
        "Position = replaceIfNull(Position, 0.0)",
        "PositionDollars = Position * MidPrice",
        "MaxPositionDollars = max_position_dollars",
        "BuyOrder = PositionDollars < MaxPositionDollars",
        "SellOrder = PositionDollars > -MaxPositionDollars",
    ])

It still fails in computing o2.

Here is a simple reproducer that seems to illustrate the problem. I have confirmed that it fails in the demo system.

from deephaven import time_table

d1 = (time_table("PT1s")
    .update([
        "Symbol=`AAPL`",
        "ReceiveTime=Timestamp",
        "ContractId=1L",
        "BidPrice=2.1",
        "AskPrice=1.3",
        "MidPrice=4.5",
        "PredPrice=6.7",
        "PredSd=1.3",
        "PredLow=1.2",
        "PredHigh=1.5",
        "TimestampFirst=Timestamp",
        "SnapTime=Timestamp",
        "Position=11.3",
    ])
)

d1m = d1.meta_table

max_position_dollars=123.45

d2 = d1.update_view([
        "Position = replaceIfNull(Position, 0.0)",
        "PositionDollars = Position * MidPrice",
        "MaxPositionDollars = max_position_dollars",
        "BuyOrder = PositionDollars < MaxPositionDollars",
        "SellOrder = PositionDollars > -MaxPositionDollars",
    ])

chipkent added bug Something isn't working triage devrel-watch DevRel team is watching labels Aug 28, 2024

chipkent added a commit to deephaven-examples/deephaven-ib that referenced this issue Aug 28, 2024

Modify examples to avoid update_view regression deephaven/deephaven-c…

89a7e6e

…ore#5998

chipkent mentioned this issue Aug 28, 2024

fix: Modify examples to avoid update_view regression deephaven-examples/deephaven-ib#128

Merged

chipkent added a commit to deephaven-examples/deephaven-ib that referenced this issue Aug 28, 2024

fix: Modify examples to avoid update_view regression (#128)

e5f0164

Modify examples to avoid update_view regression : deephaven/deephaven-core#5998

chipkent mentioned this issue Aug 28, 2024

fix: Modify examples to avoid known bugs deephaven-examples/deephaven-ib#129

Merged

chipkent added a commit to deephaven-examples/deephaven-ib that referenced this issue Aug 28, 2024

fix: Modify examples to avoid known bugs (#129)

b42069c

* Modify examples to avoid a known bug (deephaven/deephaven-core#5998) * Make the examples easier to change the account number

nbauernfeind linked a pull request Sep 12, 2024 that will close this issue

fix: SelectColumn Should Unbox Result Type #6052

Open

nbauernfeind self-assigned this Sep 12, 2024

rcaudy added query engine core Core development tasks and removed triage labels Sep 25, 2024

rcaudy added this to the 0.37.0 milestone Sep 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

QueryLanguageParser regression in the handling of inequalities inside of `update_view` #5998

QueryLanguageParser regression in the handling of inequalities inside of `update_view` #5998

chipkent commented Aug 28, 2024

chipkent commented Aug 28, 2024

chipkent commented Sep 10, 2024

QueryLanguageParser regression in the handling of inequalities inside of update_view #5998

QueryLanguageParser regression in the handling of inequalities inside of update_view #5998

Comments

chipkent commented Aug 28, 2024

chipkent commented Aug 28, 2024

chipkent commented Sep 10, 2024

QueryLanguageParser regression in the handling of inequalities inside of `update_view` #5998

QueryLanguageParser regression in the handling of inequalities inside of `update_view` #5998