gitcoinbot did not comment on github when work_done event fired

[owocki]

https://gitcoin.co/funding/details?url=https://github.com/gitcoinco/web/issues/439
https://gitcoincommunity.slack.com/archives/C8DCUQU91/p1519617040000057

expected behaviour

when this bounty was marked as done, gitcoin bot should have updated the thread to tell everyone about it.

actual behaviour

an email was received, but no github comment appeared

[owocki]

this just happened again on https://gitcoin.co/funding/details?url=https://github.com/MetaMask/metamask-extension/issues/2562

stacktrace i found in the logs

Internal Server Error: /sync/web3/
Traceback (most recent call last):
  File "/home/ubuntu/gitcoin/gitcoin-3/lib/python3.6/site-packages/web3/contract.py", line 1043, in call_contract_function
    output_data = decode_abi(output_types, return_data)
  File "/home/ubuntu/gitcoin/gitcoin-3/lib/python3.6/site-packages/eth_abi/abi.py", line 63, in decode_abi
    return decoder(stream)
  File "/home/ubuntu/gitcoin/gitcoin-3/lib/python3.6/site-packages/eth_abi/decoding.py", line 99, in __call__
    return self.decode(stream)
  File "/home/ubuntu/gitcoin/gitcoin-3/lib/python3.6/site-packages/eth_utils/functional.py", line 22, in inner
    return callback(fn(*args, **kwargs))
  File "/home/ubuntu/gitcoin/gitcoin-3/lib/python3.6/site-packages/eth_abi/decoding.py", line 137, in decode
    yield decoder(stream)
  File "/home/ubuntu/gitcoin/gitcoin-3/lib/python3.6/site-packages/eth_abi/decoding.py", line 99, in __call__
    return self.decode(stream)
  File "/home/ubuntu/gitcoin/gitcoin-3/lib/python3.6/site-packages/eth_abi/decoding.py", line 162, in decode
    raw_data = cls.read_data_from_stream(stream)
  File "/home/ubuntu/gitcoin/gitcoin-3/lib/python3.6/site-packages/eth_abi/decoding.py", line 244, in read_data_from_stream
    len(data),
eth_abi.exceptions.InsufficientDataBytes: Tried to read 32 bytes.  Only got 0 bytes

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/gitcoin/gitcoin-3/lib/python3.6/site-packages/django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "/home/ubuntu/gitcoin/gitcoin-3/lib/python3.6/site-packages/django/core/handlers/base.py", line 128, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/ubuntu/gitcoin/gitcoin-3/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/ubuntu/gitcoin/gitcoin-3/lib/python3.6/site-packages/django/views/decorators/http.py", line 40, in inner
    return func(request, *args, **kwargs)
  File "/home/ubuntu/gitcoin/gitcoin-3/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/ubuntu/gitcoin/gitcoin-3/lib/python3.6/site-packages/ratelimit/decorators.py", line 30, in _wrapped
    return fn(*args, **kw)
  File "/home/ubuntu/gitcoin/coin/app/dashboard/views.py", line 649, in sync_web3
    bounty_id = get_bounty_id(issue_url, network)
  File "/home/ubuntu/gitcoin/coin/app/dashboard/utils.py", line 225, in get_bounty_id
    bounty_id = get_bounty_id_from_web3(issue_url, network, highest_known_bounty_id, direction='down')
  File "/home/ubuntu/gitcoin/coin/app/dashboard/utils.py", line 264, in get_bounty_id_from_web3
    bounty = get_bounty(bounty_enum, network)
  File "/home/ubuntu/gitcoin/coin/app/dashboard/utils.py", line 151, in get_bounty
    num_fulfillments = int(standard_bounties.functions.getNumFulfillments(bounty_enum).call())
  File "/home/ubuntu/gitcoin/gitcoin-3/lib/python3.6/site-packages/web3/contract.py", line 852, in call
    **self.kwargs)
  File "/home/ubuntu/gitcoin/gitcoin-3/lib/python3.6/site-packages/web3/contract.py", line 1065, in call_contract_function
    raise BadFunctionCallOutput(msg) from e
web3.exceptions.BadFunctionCallOutput: Could not decode contract function call getNumFulfillments return data b'' for output_types ['uint256']
could not delete issue comment because JSON response could not be decoded: 371361001, gitcoinco, web.  204,
could not delete issue comment because JSON response could not be decoded: 371361012, gitcoinco, web.  204,
* Exception in sync_geth => BountyFulfillment matching query does not exist.
* Exception in sync_geth => Could not decode contract function call getBountyArbiter return data b'' for output_types ['address']
could not delete issue comment because JSON response could not be decoded: 371557306, gitcoinco, skunkworks.  204,
could not delete issue comment because JSON response could not be decoded: 371559902, gitcoinco, web.  204,
* Exception in sync_geth => Expecting value: line 1 column 1 (char 0)
* Exception in sync_geth => Expecting value: line 1 column 1 (char 0)

[mbeacom]

@owocki Do you think it's worth it to raise an issue on eth-abi or web3.py ?

[owocki]

@mbeacom Looks like there is an ongoing issue being discussed at ethereum/web3.py#185 that looks similar

Some workaround thoughts / ideas:

1. Is this an Infura RPC issue? I dont see any issues on status.infura.io
2. Piper states "This is what happens when you try to read data from an empty address (an address that has no code associated with it).". Is it possible that we're using the wrong contract address or the contract address is being cleared somewhere?
3. Would it be worth just adding some code to rote-retry (or retry in a celery queue) these reads? We might want to add an @transaction.atomic() around some of this code too.

i am open to other suggestions here.. seems we will need to figure out a way to come up with reproduction steps before we can really submit a bug report upstream or fix the root cause. But I am on calls most of today and won't be able to get to that today.

[owocki]

one other data point: i seem to be able to sync this bounty just fine on my local:

(gitcoin) kevinowocki@local /Users/kevinowocki/Sites/gitcoin/web/app~ % ./manage.py sync_geth mainnet 42 42           (git)-[master] :
[3/8 12:00] Getting bounty 42
[3/8 12:00] Processing bounty 42
* Bounty did_change: False

[owocki]

@mbeacom

weird. it looks like the system is now editing old gitcoinbot comments with the payout comment, instead of submitting a new comment ?

MetaMask/metamask-extension#2562 (comment)
#503 (comment)

i was not aware of this!

looks like this was your work https://github.com/gitcoinco/web/blame/master/app/dashboard/notifications.py#L325 -- my only concern about this is that it might not trigger an email back to the submitter letting them know they've been paid.

[mbeacom]

@owocki I'll split off work_done and work_submitted

[owocki]

@mbeacom i'm not saying anything has to change.... just thinking out loud here... i was not aware of this.. maybe its better? im not sure

[owocki]

looks like there is indeed no email when the comment is updated http://bits.owocki.com/3J3x1J0K2K0r/Screen%20Shot%202018-03-08%20at%204.22.24%20PM.png

idk, im kind of leaning towards making approved/issue comment a new comment. what does everyone else think? @mbeacom @PixelantDesign ?

[owocki]

idk, im kind of leaning towards making approved/issue comment a new comment. what does everyone else think? @mbeacom @PixelantDesign ?

ended up doing this