Updated for monitoring multiple coins simultaneously and other improvements #77

chrisleekr · 2021-04-03T13:10:27Z

Description

The change is focused to reduce the Binance API call weight.

A new cronjob trailingTradeIndicator has been added.
- This job will retrieve candles and calculate the lowest price within the configured time.
- Steps
  - Get account info from Binance API
  - Get symbol's candles from Binance API
  - Get symbol's open orders from Binance API
- Note that getting open orders for all symbols does available. However, it is a very expensive weight to call and it definitely will over the API limit.
- If there were orders opened manually from Binance, then the changes will be showing in the frontend a few seconds later.
A new server component binance has been added.
- This component will set WebSocket between the bot and Binance for candles.
- To reconnect WebSocket when there are symbols changed from the frontend, it uses PubSub to monitor changes in the global configuration.
The existing cronjob trailingTrade has been updated
- Previously, the job was monitoring each coin per second. However, now it monitors all the coins every second.
- The main change is this job will not call Binance API unless there is an update required from the API. All necessary information should be existing in the Redis saved from trailingTradeIndicator job and binance WebSocket.
- Steps
  - Get account info and open orders from the Redis
  - Run existing steps for all coins

With this change, the bot can monitor many coins every second.

This change does not expect to lose existing data.

If anyone would like to test it, please checkout the branch feature-monitoring-simultaneously and build the docker image, then relaunch the bot with docker-compose.

$ git pull
$ git checkout feature-monitoring-simultaneously
$ npm run docker:build
$ docker-compose -f docker-compose.server.yml up -d
$ docker ps -a
$ docker logs binance-bot -f --tail=1000 | bunyan

Related Issue

#52 #59 #90 #104 #93 #85 #99

Motivation and Context

Currently, the bot is monitoring coins one at each second. If you have many coins to monitoring, there will be many gaps between each tick per coin. To avoid that kind of issue, it was necessary to monitor the coin's current value simultaneously.

The suggestion was from @ienthach

How Has This Been Tested?

The test coverage is 100% and I have been running for a few days now.

Screenshots (if appropriate):

chrisleekr · 2021-04-03T13:12:23Z

Let me continue to test in my local and merge when it is confident this is working.

…elling an order

codecov-io · 2021-04-04T03:23:32Z

Codecov Report

Merging #77 (2f00b85) into master (01c14a9) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##            master       #77    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files           40        47     +7     
  Lines          819       994   +175     
  Branches        86       115    +29     
==========================================
+ Hits           819       994   +175

Impacted Files	Coverage Δ
app/cronjob/alive.js	`100.00% <ø> (ø)`
app/cronjob/trailingTrade/step/determine-action.js	`100.00% <ø> (ø)`
app/frontend/websocket/configure.js	`100.00% <ø> (ø)`
app/frontend/websocket/handlers/index.js	`100.00% <ø> (ø)`
app/cronjob/alive/helper.js	`100.00% <100.00%> (ø)`
app/cronjob/index.js	`100.00% <100.00%> (ø)`
app/cronjob/trailingTrade.js	`100.00% <100.00%> (ø)`
app/cronjob/trailingTrade/step/get-balances.js	`100.00% <100.00%> (ø)`
app/cronjob/trailingTrade/step/get-indicators.js	`100.00% <100.00%> (ø)`
...job/trailingTrade/step/get-symbol-configuration.js	`100.00% <100.00%> (ø)`
... and 45 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 01c14a9...2f00b85. Read the comment docs.

chopeta · 2021-04-04T21:34:14Z

Testing it now.... will let you know if I find something wrong...

chopeta · 2021-04-04T22:06:48Z

Testing with 24 symbols... the only thing I've found so far is that when I add a new symbol in Global Settings it takes a while for it to show up in the GUI... I guess that's OK and works that way since new binance server component was added.

It seems to be working fine... will wait until a couple of orders get executed...

I am a little concerned with reaching API limits when using many symbols... can you please clarify how many API calls is the bot doing in case you have for instance, 10 different symbols configured?

chopeta · 2021-04-04T22:16:50Z

I see this is also implementing #59
The only think I would change in there would be set latest version link to the releases pages:
https://github.com/chrisleekr/binance-trading-bot/releases

chopeta · 2021-04-05T00:07:58Z

OK, I've added more symbols.
Now I've noticed that SELL ORDER is not getting created.
I get a Slack notification for the sell order but no order is getting created on Binance and neither shows up in the GUI.
I can't see any errors in the logs :-(
Any idea?

chopeta · 2021-04-05T00:15:09Z

Ok, the orders are getting created on Binance...
Just now showing up in the GUI...
The weird part is that I got lots of slack msgs, likely more than the orders created and canceled.
Here's a screenshot of the orders.
At the end the coin was sold properly and GUI now reflect that.

chopeta · 2021-04-05T00:52:53Z

OK, there's definitely something wrong (likely some racing condition)... I got 3 active buy orders for the same symbol. See screenshot for NPXS.

chopeta · 2021-04-05T01:14:42Z

Buy orders are a no go... they likely have some bug related to the fact that updates are now made every second for all symbols... something is getting really messed with that.

chrisleekr · 2021-04-05T02:11:02Z

I see this is also implementing #59
The only think I would change in there would be set latest version link to the releases pages:
https://github.com/chrisleekr/binance-trading-bot/releases

Good point. I will update it.

chrisleekr · 2021-04-05T02:14:30Z

@chopeta

OK, there's definitely something wrong (likely some racing condition)... I got 3 active buy orders for the same symbol. See screenshot for NPXS.

Hmmmm, ok I think I know what to do. It could be placing an order taking more than 1 second, and the next tick process the same action.

Besides this, I also currently have a strange issue that the last buy price is not saved, which should be. :(

Buy orders are a no go... they likely have some bug related to the fact that updates are now made every second for all symbols... something is getting really messed with that.

I agree, since monitoring multiple coins, it is so unpredictable.

Thank you for testing, let me update some code and get back to you.

chrisleekr · 2021-04-05T04:57:04Z

@chopeta

I have implemented a locking symbol mechanism to prevent duplicated processes.

If the symbol is locked, then the process will not perform any action.
When the symbol is locked, you can see the lock icon in the frontend.

I hope this can fix the issue with making multiple transactions.

To update, run following commands

$ git pull
$ docker build . --build-arg NODE_ENV=development --target dev-stage -t chrisleekr/binance-trading-bot:latest
$ docker-compose -f docker-compose.server.yml up -d
$ docker ps -a
$ docker logs binance-bot -f --tail=1000 | bunyan

chopeta · 2021-04-05T07:13:13Z

Nice, already updated here... testing to see how it works...
Will keep you posted.

chrisleekr · 2021-04-05T12:43:31Z

@chopeta

I am a little concerned with reaching API limits when using many symbols... can you please clarify how many API calls is the bot doing in case you have, for instance, 10 different symbols configured?

With this change, it will never be over API limits unless 24 symbols suddenly buy/sell all together.

Here are Binance API limits:

1200 requests per minute
10 orders per second
100,000 orders per 24hrs

These are detailed API calls:

TrailingTradeIndicator - run every second
- This job is in charge of getting latest data and cache it for the job TrailingTrade.
- To avoid API limits, candles/open orders will retrieve specific
- Steps:
  - Get account info from Binance API: GET /v3/account - 5 weights
  - Get symbol's candles from Binance API: GET /v3/klines - 1 weight
  - Get symbol's open orders from Binance API: GET /v3/openOrders - 1 weight

Hence the TrailingTradeIndicator job will use 7 weights * 60 seconds = 420 weights

TrailingTrade - run every second
- With this job, if there is nothing to be actioned, then no API call will be made.
- However, if there are some coins to be actions, then it will call API
- Steps:
  - Handle open orders
    - If needs to cancel, then call the following API requests
      - Cancel existing order: DELETE /v3/order - 1 weight
        
        If cancelling order is failed, which can happen
        
        Get symbol's open orders: GET /v3/openOrders - 1 weight
        
        Get account info from Binance API: GET /v3/account - 5 weights
    - If needs to buy/sell, then call the following API request
      - Get account info from Binance API: GET /v3/account - 5 weights
  - Place buy orders
    - If need to buy, then call the following API requests
    - Place new order: POST /v3/order - 1 weight
    - Get symbol's open orders: GET /v3/openOrders - 1 weight
    - Get account info from Binance API: GET /v3/account - 5 weights
  - Place sell orders
    - If need to sell, then call the following API requests
    - Place new order: POST /v3/order - 1 weight
    - Get symbol's open orders: GET /v3/openOrders - 1 weight
    - Get account info from Binance API: GET /v3/account - 5 weights

Minimum is 420 weights, plus the number of orders with cancellation * 13 weights (or if cancel failed 7 weights, but let's assume cancelling an order is always working)

So summing up, 420 base weights + 60 orders * 13 weights = 1200 weights.

So basically, unless the bot is making 60 orders within a minute, it won't be exceeding the 1200 request limit.

Hope I made it clear to you.

BramFr · 2021-04-09T10:59:51Z

@chopeta NPXS transferd to PUNDIX
https://www.binance.com/en/trade/PUNDIX_USDT?layout=pro

garyng · 2021-05-14T10:47:02Z

To run npm run docker:build on Windows, you can set npm's script-shell to use powershell or pwsh (powershell core).
To use powershell core, you can either

npm config set script-shell "pwsh", this configure npm to always use pwsh; or,
$env:npm_config_script_shell="pwsh", this configure npm to use pwsh for current powershell session only

chrisleekr · 2021-05-14T10:49:04Z

@garyng Wow thanks for the tip. Could you add to wiki please? https://github.com/chrisleekr/binance-trading-bot/wiki/Troubleshooting

garyng · 2021-05-14T11:22:41Z

@chrisleekr: I created a powershell script so that others would not need to tinker with npm config: #175. Can you take a look and maybe merge it if it's okay?

chrisleekr · 2021-05-14T11:25:17Z

@garyng So good. I will merge in.

garyng · 2021-05-14T11:25:53Z

@chrisleekr: I updated the wiki as well: https://github.com/chrisleekr/binance-trading-bot/wiki/Troubleshooting#building-docker-image-under-windows-with-powershell

* Add powershell scripts for docker build on windows * Use switch instead

chrisleekr · 2021-05-14T11:26:19Z

@garyng Merged!

radumalica · 2021-05-14T12:15:38Z

@radumalica

It seems that the issue was copying the ".env" file from master branch and i was missing these 2 lines:

You don't need those lines in the .env file. I really don't think this is the code issue or bug because it's not happening to others.
Could you create a new issue with this detail? Let's figure it out from there. I will leave some more feedback to debug your issue in the new issue.

It seems that was not the issue, it was just a coincidence, after restarting containers, even with these 2 config lines or not, the bot doesn't display anything on frontend and has the same errors as before

[2021-05-14T11:47:47.289Z]  INFO: binance-api/19 on f5536e1e40ac: Existing opened socket for candles found, clean first (version=0.0.65, gitHash=a9e4c36, server=binance)
[2021-05-14T11:47:48.291Z]  WARN: binance-api/19 on f5536e1e40ac: Binance candle is not received in last mintues. Reconfigure websocket (version=0.0.65, gitHash=a9e4c36, server=binance, debug=true)
[2021-05-14T11:47:48.291Z]  INFO: binance-api/19 on f5536e1e40ac: Set websocket for candles (version=0.0.65, gitHash=a9e4c36, server=binance)
[2021-05-14T11:47:48.291Z]  INFO: binance-api/19 on f5536e1e40ac: Finding document from MongoDB (version=0.0.65, gitHash=a9e4c36, server=binance, helper=mongo, funcName=findOne, collectionName=trailing-trade-common)
    query: {
      "key": "configuration"
    }
[2021-05-14T11:47:48.294Z]  INFO: binance-api/19 on f5536e1e40ac: Found document from MongoDB (version=0.0.65, gitHash=a9e4c36, server=binance, helper=mongo, funcName=findOne)
    result: {
      "_id": "609e627f17efb3a018af44b8",
      "key": "configuration",
      "buy": {
        "enabled": true,
        "maxPurchaseAmount": -1,
        "maxPurchaseAmounts": {
          "USDT": 100,
          "BTC": 0.001
        },
        "triggerPercentage": 1,
        "stopPercentage": 1.02,
        "limitPercentage": 1.021
      },
      "candles": {
        "interval": "1h",
        "limit": 100
      },
      "cronTime": "* * * * * *",
      "enabled": true,
      "sell": {
        "enabled": true,
        "triggerPercentage": 1.06,
        "stopPercentage": 0.98,
        "limitPercentage": 0.979,
        "stopLoss": {
          "enabled": false,
          "maxLossPercentage": 0.8,
          "disableBuyMinutes": 360,
          "orderType": "market"
        }
      },
      "symbols": [
        "ETHUSDT",
        "ADAUSDT",
        "BNBUSDT",
        "XRPUSDT"
      ]
    }
[2021-05-14T11:47:48.294Z]  INFO: binance-api/19 on f5536e1e40ac: Retrieved symbols (version=0.0.65, gitHash=a9e4c36, server=binance)
    symbols: [
      "ETHUSDT",
      "ADAUSDT",
      "BNBUSDT",
      "XRPUSDT"
    ]

As I said , bot was deployed on brand new VM with Ubuntu 20.04.02 LTS 64bit, with NO other software installed, just docker, docker-compose and npm for building the image.

I even deleted ALL volumes and images, redeployed again and the same issue is happening.

Any thoughts on how to debug what the bot receives from Binance ? The only difference i see from the master branch is that the jobs are handled differently, on this branch you have and extra "server-binance.js" which handles the candles and websocket, and master branch doesn't have this file and handles the operations in a different manner

Also, WS logs from Chrome:

chrisleekr · 2021-05-14T12:17:06Z

@radumalica

Let's open a new issue and start from there.

chrisleekr · 2021-05-15T01:07:54Z

With the latest commit, the bot will disable action for 20 seconds after confirming an order per coin.

It will reduce unexpected behaviour due to Binance API overload. It will also stable the bot processing every second.
You can see the action is disabled in the frontend. Refer screenshot.

https://github.com/chrisleekr/binance-trading-bot/blob/feature-monitoring-simultaneously/app/cronjob/trailingTrade/step/ensure-order-placed.js#L174

It also fixed the issue of not selling the coin like ALHPABTC. ALPHABTC's step size was 1.00000000 which was incorrectly calculated for the precision.

https://github.com/chrisleekr/binance-trading-bot/blob/feature-monitoring-simultaneously/app/cronjob/trailingTrade/step/__tests__/place-sell-order.test.js#L1348

Also, you will be able to see current API usage in the slack message.

You know what to do for updating.

$ git checkout feature-monitoring-simultaneously
$ git pull
$ npm run docker:build
$ docker-compose -f docker-compose.server.yml up -d
$ docker ps -a
$ docker logs binance-bot -f --tail=1000 | bunyan

chrisleekr · 2021-05-16T06:18:11Z

Ok, looks like the latest change made the bot more stable.

I tested with an aggressive strategy and lost money, but I happily confirmed there was no missing last buy price and the order was placed correctly. I even experienced the failure in cancelling the order, which is handled well with the bot.

I will run for few days and see how it goes. I think it's time to merge in.

vzwick · 2021-05-18T16:11:02Z

Failing for me with cannot find module 'redlock' in app/helpers/cache.js

Naramsim · 2021-05-18T21:06:45Z

@chrisleekr: With the latest commit, the bot will disable action for 20 seconds after confirming an order per coin.

Are these 20 seconds hardcoded in the code? It could be cool to allow users to tune them. Sometimes 20 seconds are too many to wait for.

FanaticNinja · 2021-05-19T00:26:59Z

Works great on test, but can't get it to work with Binance.US.

chrisleekr · 2021-05-21T00:56:51Z

@Naramsim Oh hmm, interesting. Customising that value can cause the failure of placing the order.

Hmm, I may add to the environment parameter to customise it instead of in the UI since that is a configuration for running the system.

chrisleekr · 2021-05-21T02:33:44Z

@Naramsim

I have added a new environment parameter to control disable action seconds after confirming the order.

https://github.com/chrisleekr/binance-trading-bot/blob/feature-monitoring-simultaneously/config/custom-environment-variables.json#L138

You can open your .env and add the following variable.

BINANCE_JOBS_TRAILING_TRADE_SYSTEM_TEMPORARY_DISABLE_ACTION_AFTER_CONFIRMING_ORDER=10

And then you know what to do

$ git pull
$ npm run docker:build
$ docker-compose -f docker-compose.server.yml up -d
$ docker ps -a
$ docker logs binance-bot -f --tail=1000 | bunyan

Naramsim · 2021-05-21T10:25:44Z

@chrisleekr

I have added a new environment parameter to control disable action seconds after confirming the order.

Cool, thanks for the addition. Why do you think setting a timeout of 5 (or even 1) seconds would break the bot?

I'm asking this question since the market is often very volatile and stop-limit orders can be filled pretty quickly. So I was expecting that the frequency of API calls when there is a stop-limit order in place should be the highest.

Well, anyways, thanks a lot for addressing my request.

PS:

I'd change the description from -> to

-Set the second for disabling the action after confirming the order. Note that decreasing this value can cause the issue with the bot acting wrong.
+Set the seconds for disabling any actions after confirming an order. Note that decreasing this value can cause issues with the bot acting in a wrong way (maybe specify the risks and what are the wrong ways).

chrisleekr · 2021-05-21T12:50:37Z

@Naramsim

Oh thanks, man. I just updated the config description with why it may not work well if reduces the second (and typo).

Cool, thanks for the addition. Why do you think setting a timeout of 5 (or even 1) seconds would break the bot?

When the market is volatile, Binance API became busy and slow response. Sometimes it does not return the response of the open orders immediately (or sometimes more than 5-30 secs). It's because of too many requests to process.

The bot is checking the order in the open order API request every second to make sure the order is showing in Binance after placing the order. However, the bot is also trying to figure out what to do whether to buy/sell/remove the last buy price as well. Unexpectedly the bot seems too fast to check before confirming the order.

The solution was to give the bot some time to disable any actions until the order is definitely confirmed in the open order.

Hope it answered your question.

chrisleekr · 2021-05-21T12:51:50Z

Ok, guys. I am merging in.

Updated for monitoring mutliple coins simultaneously

41c426b

chrisleekr added the enhancement New feature or request label Apr 3, 2021

chrisleekr linked an issue Apr 3, 2021 that may be closed by this pull request

Change multi symbol trade setting. #52

Closed

chrisleekr self-assigned this Apr 3, 2021

chrisleekr added 4 commits April 4, 2021 13:00

Updated code to make sure syncing open orders/account info after canc…

1567bbf

…elling an order

Fixed lint

0cebdc3

Updated REAMDE.md

afc91f7

Updated REAMDE.md

dba3093

chrisleekr added 4 commits April 4, 2021 22:35

Updated frontend to display version

f1429ee

Updated test

eeac0ec

Fixed frontend state

c0a941b

Updated footer for frontend

2f00b85

chopeta mentioned this pull request Apr 4, 2021

Display release version on the web GUI #59

Closed

chrisleekr added 2 commits April 5, 2021 13:37

Implement lock symbol to avoid unexpected behaviours

3c87beb

Updated tests

b5dcad1

garyng mentioned this pull request May 14, 2021

Add npm task for building docker image on Windows #175

Merged

Add npm task for building docker image on Windows (#175)

a29744d

* Add powershell scripts for docker build on windows * Use switch instead

radumalica mentioned this pull request May 14, 2021

#77 branch doesn't show symbols on frontend #176

Closed

chrisleekr mentioned this pull request May 14, 2021

Indicator data is not retrieved, wait for cache #177

Closed

chrisleekr added 2 commits May 15, 2021 00:56

Now disabling action after placing an order and bug fix

2cdb4c6

Added more tests

ea7605d

chrisleekr mentioned this pull request May 14, 2021

Not enough precision when using BTC as base coin? #173

Closed

Updated README.md

3a0fda6

Added new env for temporaryDisableActionAfterConfirmingOrder

a616aa9

Updated config description

bbc8ba4

chrisleekr merged commit f535b18 into master May 21, 2021

chrisleekr deleted the feature-monitoring-simultaneously branch May 30, 2021 03:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updated for monitoring multiple coins simultaneously and other improvements #77

Updated for monitoring multiple coins simultaneously and other improvements #77

chrisleekr commented Apr 3, 2021 •

edited

Loading

chrisleekr commented Apr 3, 2021

codecov-io commented Apr 4, 2021 •

edited

Loading

chopeta commented Apr 4, 2021 •

edited

Loading

chopeta commented Apr 4, 2021

chopeta commented Apr 4, 2021

chopeta commented Apr 5, 2021

chopeta commented Apr 5, 2021

chopeta commented Apr 5, 2021

chopeta commented Apr 5, 2021

chrisleekr commented Apr 5, 2021

chrisleekr commented Apr 5, 2021

chrisleekr commented Apr 5, 2021

chopeta commented Apr 5, 2021

chrisleekr commented Apr 5, 2021

BramFr commented Apr 9, 2021

garyng commented May 14, 2021

chrisleekr commented May 14, 2021

garyng commented May 14, 2021

chrisleekr commented May 14, 2021

garyng commented May 14, 2021

chrisleekr commented May 14, 2021

radumalica commented May 14, 2021 •

edited

Loading

chrisleekr commented May 14, 2021

chrisleekr commented May 15, 2021 •

edited

Loading

chrisleekr commented May 16, 2021 •

edited

Loading

vzwick commented May 18, 2021

Naramsim commented May 18, 2021 •

edited

Loading

FanaticNinja commented May 19, 2021

chrisleekr commented May 21, 2021

chrisleekr commented May 21, 2021

Naramsim commented May 21, 2021 •

edited

Loading

chrisleekr commented May 21, 2021 •

edited

Loading

chrisleekr commented May 21, 2021

Updated for monitoring multiple coins simultaneously and other improvements #77

Updated for monitoring multiple coins simultaneously and other improvements #77

Conversation

chrisleekr commented Apr 3, 2021 • edited Loading

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

chrisleekr commented Apr 3, 2021

codecov-io commented Apr 4, 2021 • edited Loading

Codecov Report

chopeta commented Apr 4, 2021 • edited Loading

chopeta commented Apr 4, 2021

chopeta commented Apr 4, 2021

chopeta commented Apr 5, 2021

chopeta commented Apr 5, 2021

chopeta commented Apr 5, 2021

chopeta commented Apr 5, 2021

chrisleekr commented Apr 5, 2021

chrisleekr commented Apr 5, 2021

chrisleekr commented Apr 5, 2021

chopeta commented Apr 5, 2021

chrisleekr commented Apr 5, 2021

BramFr commented Apr 9, 2021

garyng commented May 14, 2021

chrisleekr commented May 14, 2021

garyng commented May 14, 2021

chrisleekr commented May 14, 2021

garyng commented May 14, 2021

chrisleekr commented May 14, 2021

radumalica commented May 14, 2021 • edited Loading

chrisleekr commented May 14, 2021

chrisleekr commented May 15, 2021 • edited Loading

chrisleekr commented May 16, 2021 • edited Loading

vzwick commented May 18, 2021

Naramsim commented May 18, 2021 • edited Loading

FanaticNinja commented May 19, 2021

chrisleekr commented May 21, 2021

chrisleekr commented May 21, 2021

Naramsim commented May 21, 2021 • edited Loading

chrisleekr commented May 21, 2021 • edited Loading

chrisleekr commented May 21, 2021

chrisleekr commented Apr 3, 2021 •

edited

Loading

codecov-io commented Apr 4, 2021 •

edited

Loading

chopeta commented Apr 4, 2021 •

edited

Loading

radumalica commented May 14, 2021 •

edited

Loading

chrisleekr commented May 15, 2021 •

edited

Loading

chrisleekr commented May 16, 2021 •

edited

Loading

Naramsim commented May 18, 2021 •

edited

Loading

Naramsim commented May 21, 2021 •

edited

Loading

chrisleekr commented May 21, 2021 •

edited

Loading