Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

InvalidRequest reproducer #962

Closed
binocarlos opened this issue Oct 27, 2022 · 2 comments
Closed

InvalidRequest reproducer #962

binocarlos opened this issue Oct 27, 2022 · 2 comments
Labels
type/bug Type: Something is not working as expected

Comments

@binocarlos
Copy link
Contributor

We have added a mechanism to deal with requests that arrive at the requester node after a job has completed (by issuing an InvalidRequest) message

There is an easy reproducer that will emit these messages so we can start to dig into it and understand why they are emitted...

reproducer

In one terminal:

export PREDICTABLE_API_PORT=1
make devstack

In another terminal:

export BACALHAU_API_HOST=localhost
export BACALHAU_API_PORT=20000
go run . docker run --concurrency 3 ubuntu echo hello

You will see the devstack logs light up with a bunch of these InvalidRequest like this:

image

Running something as simply as above (i.e. devstack with 3 nodes) shouldn't really be issuing these InvalidRequest messages so let's try to understand why it is doing that and fix it

@binocarlos binocarlos added the type/bug Type: Something is not working as expected label Oct 27, 2022
@wjam
Copy link
Contributor

wjam commented Oct 27, 2022

Hadn't realised this was a problem. I managed to reproduce a similar issue locally when running the Lotus stack test when this line was left out:
https://github.com/filecoin-project/bacalhau/blob/90289561845e8d0e5a9074d162097533166d3415/pkg/publisher/filecoin_lotus/publisher.go#L383

    publisher.go:213: 16:46:25.529 | ERR filecoin_lotus/publisher.go:213 > Couldn't find a miner error="0 errors occurred:\n\t\n\n" [NodeID:QmVx8sT1]
    tracer.go:242: 16:46:25.529 | TRC system/tracer.go:242 > {"Attributes":null,"ChildSpanCount":5,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.529406732+01:00","Events":null,"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"pkg/publisher/filecoin_lotus/createDeal","Parent":{"Remote":false,"SpanID":"8418229d8189c3f4","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"6c1c5f7fd1a3f7d9","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.370875+01:00","Status":{"Code":"Unset","Description":""}}
    tracer.go:242: 16:46:25.529 | TRC system/tracer.go:242 > {"Attributes":null,"ChildSpanCount":3,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.529521001+01:00","Events":null,"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"pkg/publisher/filecoin_lotus/PublishShardResult","Parent":{"Remote":false,"SpanID":"4414d1a659882606","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"8418229d8189c3f4","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.310878+01:00","Status":{"Code":"Unset","Description":""}}
    shard_fsm.go:334: 16:46:25.53 | DBG computenode/shard_fsm.go:334 > [QmVx8sT1] shard: ce33a849-e433-42c5-a1bc-1e30ea5f2c53:0 at state: PublishingToRequester transitioning from PublishingToRequester -> Error [NodeID:QmVx8sT1]
    shard_fsm.go:496: 16:46:25.53 | ERR computenode/shard_fsm.go:496 > errorState: error completing job due to: unable to find a miner [NodeID:QmVx8sT1]
    tracer.go:301: 16:46:25.53 | TRC system/tracer.go:301 > adding jobid from baggage to span as attribute: nodeid=QmVx8sT11fzkW7CmVPYyfiqGjxYEszfxWJ7mxqYVRdtKvs,jobid=ce33a849-e433-42c5-a1bc-1e30ea5f2c53
    tracer.go:242: 16:46:25.53 | TRC system/tracer.go:242 > {"Attributes":[{"Key":"jobid","Value":{"Type":"STRING","Value":"ce33a849-e433-42c5-a1bc-1e30ea5f2c53"}}],"ChildSpanCount":3,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.529738688+01:00","Events":null,"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"pkg/computenode/ShardFSM.publishingToRequesterState","Parent":{"Remote":false,"SpanID":"cb8edae537a64795","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"4414d1a659882606","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.039129+01:00","Status":{"Code":"Unset","Description":""}}
    libp2p.go:335: 16:46:25.53 | TRC libp2p/libp2p.go:335 > Sending event ComputeError: {"sent_time":"2022-10-27T16:46:25.530354+01:00","job_event":{"JobID":"ce33a849-e433-42c5-a1bc-1e30ea5f2c53","SourceNodeID":"QmVx8sT11fzkW7CmVPYyfiqGjxYEszfxWJ7mxqYVRdtKvs","EventName":"ComputeError","Spec":{"Docker":{},"Language":{"JobContext":{}},"Wasm":{},"Resources":{"GPU":""},"Sharding":{}},"JobExecutionPlan":{},"Deal":{},"Status":"errorState: error completing job due to: unable to find a miner","VerificationResult":{},"PublishedResult":{},"EventTime":"2022-10-27T16:46:25.530327+01:00","RunOutput":{"stdout":"hello world","stdouttruncated":false,"stderr":"","stderrtruncated":false,"exitCode":0,"runnerError":""}},"trace_data":{"baggage":"nodeid=QmVx8sT11fzkW7CmVPYyfiqGjxYEszfxWJ7mxqYVRdtKvs,jobid=ce33a849-e433-42c5-a1bc-1e30ea5f2c53","traceparent":"00-eca2bf878a2f2106e141960c0e7d8bc8-5388494728e8488e-01"}} [NodeID:QmVx8sT1]
    tracer.go:242: 16:46:25.531 | TRC system/tracer.go:242 > {"Attributes":null,"ChildSpanCount":0,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.531125846+01:00","Events":null,"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"pkg/publisher/filecoin_lotus/api/StateMinerInfo","Parent":{"Remote":false,"SpanID":"6c1c5f7fd1a3f7d9","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"f98f70f2154054b8","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.529306+01:00","Status":{"Code":"Unset","Description":""}}
    tracer.go:242: 16:46:25.533 | TRC system/tracer.go:242 > {"Attributes":null,"ChildSpanCount":0,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.532771379+01:00","Events":null,"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"pkg/publisher/filecoin_lotus/api/StateMinerPower","Parent":{"Remote":false,"SpanID":"6c1c5f7fd1a3f7d9","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"24d827efeba8ad7a","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.531234+01:00","Status":{"Code":"Unset","Description":""}}
    libp2p.go:369: 16:46:25.533 | TRC libp2p/libp2p.go:369 > [QmVx8sT1=>QmVx8sT1] Message latency: 3 ms (ComputeError) [NodeID:QmVx8sT1]
    tracer.go:242: 16:46:25.533 | TRC system/tracer.go:242 > {"Attributes":null,"ChildSpanCount":0,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.533490068+01:00","Events":null,"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"pkg/transport/libp2p.writeJobEvent","Parent":{"Remote":false,"SpanID":"9dfae7dd222b0b87","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"5388494728e8488e","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.530341+01:00","Status":{"Code":"Unset","Description":""}}
    shard_fsm.go:334: 16:46:25.533 | DBG computenode/shard_fsm.go:334 > [QmVx8sT1] shard: ce33a849-e433-42c5-a1bc-1e30ea5f2c53:0 at state: Error transitioning from Error -> Completed [NodeID:QmVx8sT1]
    libp2p.go:377: 16:46:25.533 | TRC libp2p/libp2p.go:377 > Received event ComputeError: {SentTime:2022-10-27 16:46:25.530354 +0100 BST JobEvent:{APIVersion: JobID:ce33a849-e433-42c5-a1bc-1e30ea5f2c53 ShardIndex:0 ClientID: SourceNodeID:QmVx8sT11fzkW7CmVPYyfiqGjxYEszfxWJ7mxqYVRdtKvs TargetNodeID: EventName:ComputeError Spec:{Engine:engineUnknown Verifier:verifierUnknown Publisher:publisherUnknown Docker:{Image: Entrypoint:[] EnvironmentVariables:[] WorkingDirectory:} Language:{Language: LanguageVersion: Deterministic:false Context:{StorageSource:storageSourceUnknown Name: CID: URL: Path: Metadata:map[]} Command: ProgramPath: RequirementsPath:} Wasm:{EntryPoint: Parameters:[]} Resources:{CPU: Memory: Disk: GPU:} Inputs:[] Contexts:[] Outputs:[] Annotations:[] Sharding:{GlobPattern: BatchSize:0 BasePath:} DoNotTrack:false} JobExecutionPlan:{TotalShards:0} Deal:{Concurrency:0 Confidence:0 MinBids:0} Status:errorState: error completing job due to: unable to find a miner VerificationProposal:[] VerificationResult:{Complete:false Result:false} PublishedResult:{StorageSource:storageSourceUnknown Name: CID: URL: Path: Metadata:map[]} EventTime:2022-10-27 16:46:25.530327 +0100 BST SenderPublicKey:[] RunOutput:0xc00073ec80} TraceData:map[baggage:nodeid=QmVx8sT11fzkW7CmVPYyfiqGjxYEszfxWJ7mxqYVRdtKvs,jobid=ce33a849-e433-42c5-a1bc-1e30ea5f2c53 traceparent:00-eca2bf878a2f2106e141960c0e7d8bc8-5388494728e8488e-01]} [NodeID:QmVx8sT1]
    tracer.go:242: 16:46:25.534 | TRC system/tracer.go:242 > {"Attributes":[{"Key":"jobid","Value":{"Type":"STRING","Value":"ce33a849-e433-42c5-a1bc-1e30ea5f2c53"}}],"ChildSpanCount":1,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.533665963+01:00","Events":null,"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"pkg/computenode/ShardFSM.errorState","Parent":{"Remote":false,"SpanID":"cb8edae537a64795","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"c92eeeedc21e09c8","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.530221+01:00","Status":{"Code":"Unset","Description":""}}
    tracer.go:242: 16:46:25.534 | TRC system/tracer.go:242 > {"Attributes":[{"Key":"nodeid","Value":{"Type":"STRING","Value":"QmVx8sT11fzkW7CmVPYyfiqGjxYEszfxWJ7mxqYVRdtKvs"}},{"Key":"jobid","Value":{"Type":"STRING","Value":"ce33a849-e433-42c5-a1bc-1e30ea5f2c53"}},{"Key":"environment","Value":{"Type":"STRING","Value":"test"}}],"ChildSpanCount":0,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.534112188+01:00","Events":[{"Attributes":[{"Key":"nodeid","Value":{"Type":"STRING","Value":"QmVx8sT11fzkW7CmVPYyfiqGjxYEszfxWJ7mxqYVRdtKvs"}},{"Key":"jobid","Value":{"Type":"STRING","Value":"ce33a849-e433-42c5-a1bc-1e30ea5f2c53"}}],"DroppedAttributeCount":0,"Name":"read_ComputeError","Time":"2022-10-27T16:46:25.534111+01:00"}],"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"service/JobLifecycle-QmVx8sT1","Parent":{"Remote":true,"SpanID":"5388494728e8488e","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"266153e4bcbd01ab","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.534105+01:00","Status":{"Code":"Unset","Description":""}}
    tracer.go:242: 16:46:25.534 | TRC system/tracer.go:242 > {"Attributes":null,"ChildSpanCount":0,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.534211734+01:00","Events":null,"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"pkg/localdb/inmemory/InMemoryDatastore.AddEvent","Parent":{"Remote":false,"SpanID":"266153e4bcbd01ab","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"beaf1f54aa1e0251","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.534208+01:00","Status":{"Code":"Unset","Description":""}}
    tracer.go:242: 16:46:25.534 | TRC system/tracer.go:242 > {"Attributes":null,"ChildSpanCount":0,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.534561994+01:00","Events":null,"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"pkg/localdb/inmemory/InMemoryDatastore.UpdateShardState","Parent":{"Remote":false,"SpanID":"266153e4bcbd01ab","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"9ddb9f192a45b1a4","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.534558+01:00","Status":{"Code":"Unset","Description":""}}
    shard_fsm.go:199: 16:46:25.535 | WRN requesternode/shard_fsm.go:199 > [QmVx8sT1] shard: ce33a849-e433-42c5-a1bc-1e30ea5f2c53:0 at state: WaitingToPublishResults ignoring request due to `invalid action ActionResultReceived in state WaitingToPublishResults`: {action:1 sourceNodeID:QmVx8sT11fzkW7CmVPYyfiqGjxYEszfxWJ7mxqYVRdtKvs} [NodeID:QmVx8sT1]
    tracer.go:242: 16:46:25.535 | TRC system/tracer.go:242 > {"Attributes":null,"ChildSpanCount":0,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.534813776+01:00","Events":null,"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"pkg/localdb/inmemory/InMemoryDatastore.GetJob","Parent":{"Remote":false,"SpanID":"266153e4bcbd01ab","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"714201a468ae2e1a","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.534811+01:00","Status":{"Code":"Unset","Description":""}}
    libp2p.go:335: 16:46:25.535 | TRC libp2p/libp2p.go:335 > Sending event InvalidRequest: {"sent_time":"2022-10-27T16:46:25.535222+01:00","job_event":{"JobID":"ce33a849-e433-42c5-a1bc-1e30ea5f2c53","SourceNodeID":"QmVx8sT11fzkW7CmVPYyfiqGjxYEszfxWJ7mxqYVRdtKvs","TargetNodeID":"QmVx8sT11fzkW7CmVPYyfiqGjxYEszfxWJ7mxqYVRdtKvs","EventName":"InvalidRequest","Spec":{"Docker":{},"Language":{"JobContext":{}},"Wasm":{},"Resources":{"GPU":""},"Sharding":{}},"JobExecutionPlan":{},"Deal":{},"Status":"invalid action ActionResultReceived in state WaitingToPublishResults","VerificationResult":{},"PublishedResult":{},"EventTime":"2022-10-27T16:46:25.535203+01:00"},"trace_data":{"traceparent":"00-973d6d1d6e0bb78f9ae744321dbdd9c4-3fa43864c72b5fde-01"}} [NodeID:QmVx8sT1]
    tracer.go:242: 16:46:25.535 | TRC system/tracer.go:242 > {"Attributes":[{"Key":"environment","Value":{"Type":"STRING","Value":"test"}}],"ChildSpanCount":0,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.535059698+01:00","Events":null,"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"service/ComputeError","Parent":{"Remote":false,"SpanID":"266153e4bcbd01ab","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"0f5ae55df8e55e07","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.535055+01:00","Status":{"Code":"Unset","Description":""}}
    tracer.go:242: 16:46:25.535 | TRC system/tracer.go:242 > {"Attributes":null,"ChildSpanCount":0,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.535315721+01:00","Events":null,"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"pkg/localdb/inmemory/InMemoryDatastore.GetJob","Parent":{"Remote":false,"SpanID":"ba7ff51f59584ae2","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"74a4435ec3b497a0","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.535314+01:00","Status":{"Code":"Unset","Description":""}}
    tracer.go:242: 16:46:25.535 | TRC system/tracer.go:242 > {"Attributes":[{"Key":"environment","Value":{"Type":"STRING","Value":"test"}}],"ChildSpanCount":1,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.535559165+01:00","Events":null,"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"service/HandleJobEvent","Parent":{"Remote":false,"SpanID":"266153e4bcbd01ab","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"ba7ff51f59584ae2","TraceFlags":"01","TraceID":"eca2bf878a2f2106e141960c0e7d8bc8","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.535312+01:00","Status":{"Code":"Unset","Description":""}}
    libp2p.go:369: 16:46:25.538 | TRC libp2p/libp2p.go:369 > [QmVx8sT1=>QmVx8sT1] Message latency: 2 ms (InvalidRequest) [NodeID:QmVx8sT1]
    tracer.go:242: 16:46:25.538 | TRC system/tracer.go:242 > {"Attributes":null,"ChildSpanCount":0,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.538023684+01:00","Events":null,"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"pkg/transport/libp2p.writeJobEvent","Parent":{"Remote":false,"SpanID":"34ec14e2aa8aa647","TraceFlags":"01","TraceID":"973d6d1d6e0bb78f9ae744321dbdd9c4","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"3fa43864c72b5fde","TraceFlags":"01","TraceID":"973d6d1d6e0bb78f9ae744321dbdd9c4","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.535215+01:00","Status":{"Code":"Unset","Description":""}}
    libp2p.go:377: 16:46:25.538 | TRC libp2p/libp2p.go:377 > Received event InvalidRequest: {SentTime:2022-10-27 16:46:25.535222 +0100 BST JobEvent:{APIVersion: JobID:ce33a849-e433-42c5-a1bc-1e30ea5f2c53 ShardIndex:0 ClientID: SourceNodeID:QmVx8sT11fzkW7CmVPYyfiqGjxYEszfxWJ7mxqYVRdtKvs TargetNodeID:QmVx8sT11fzkW7CmVPYyfiqGjxYEszfxWJ7mxqYVRdtKvs EventName:InvalidRequest Spec:{Engine:engineUnknown Verifier:verifierUnknown Publisher:publisherUnknown Docker:{Image: Entrypoint:[] EnvironmentVariables:[] WorkingDirectory:} Language:{Language: LanguageVersion: Deterministic:false Context:{StorageSource:storageSourceUnknown Name: CID: URL: Path: Metadata:map[]} Command: ProgramPath: RequirementsPath:} Wasm:{EntryPoint: Parameters:[]} Resources:{CPU: Memory: Disk: GPU:} Inputs:[] Contexts:[] Outputs:[] Annotations:[] Sharding:{GlobPattern: BatchSize:0 BasePath:} DoNotTrack:false} JobExecutionPlan:{TotalShards:0} Deal:{Concurrency:0 Confidence:0 MinBids:0} Status:invalid action ActionResultReceived in state WaitingToPublishResults VerificationProposal:[] VerificationResult:{Complete:false Result:false} PublishedResult:{StorageSource:storageSourceUnknown Name: CID: URL: Path: Metadata:map[]} EventTime:2022-10-27 16:46:25.535203 +0100 BST SenderPublicKey:[] RunOutput:<nil>} TraceData:map[traceparent:00-973d6d1d6e0bb78f9ae744321dbdd9c4-3fa43864c72b5fde-01]} [NodeID:QmVx8sT1]
    tracer.go:242: 16:46:25.538 | TRC system/tracer.go:242 > {"Attributes":[{"Key":"nodeid","Value":{"Type":"STRING","Value":"QmVx8sT11fzkW7CmVPYyfiqGjxYEszfxWJ7mxqYVRdtKvs"}},{"Key":"jobid","Value":{"Type":"STRING","Value":"ce33a849-e433-42c5-a1bc-1e30ea5f2c53"}},{"Key":"environment","Value":{"Type":"STRING","Value":"test"}}],"ChildSpanCount":0,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.538600402+01:00","Events":[{"Attributes":[{"Key":"nodeid","Value":{"Type":"STRING","Value":"QmVx8sT11fzkW7CmVPYyfiqGjxYEszfxWJ7mxqYVRdtKvs"}},{"Key":"jobid","Value":{"Type":"STRING","Value":"ce33a849-e433-42c5-a1bc-1e30ea5f2c53"}}],"DroppedAttributeCount":0,"Name":"read_InvalidRequest","Time":"2022-10-27T16:46:25.5386+01:00"}],"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"service/JobLifecycle-QmVx8sT1","Parent":{"Remote":true,"SpanID":"3fa43864c72b5fde","TraceFlags":"01","TraceID":"973d6d1d6e0bb78f9ae744321dbdd9c4","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"f0a35288cd20ad97","TraceFlags":"01","TraceID":"973d6d1d6e0bb78f9ae744321dbdd9c4","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.538593+01:00","Status":{"Code":"Unset","Description":""}}
    tracer.go:242: 16:46:25.539 | TRC system/tracer.go:242 > {"Attributes":null,"ChildSpanCount":0,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.538817575+01:00","Events":null,"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"pkg/localdb/inmemory/InMemoryDatastore.AddEvent","Parent":{"Remote":false,"SpanID":"f0a35288cd20ad97","TraceFlags":"01","TraceID":"973d6d1d6e0bb78f9ae744321dbdd9c4","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"da63d243b6fd9d5d","TraceFlags":"01","TraceID":"973d6d1d6e0bb78f9ae744321dbdd9c4","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.538813+01:00","Status":{"Code":"Unset","Description":""}}
    tracer.go:242: 16:46:25.539 | TRC system/tracer.go:242 > {"Attributes":null,"ChildSpanCount":0,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.539076357+01:00","Events":null,"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"pkg/localdb/inmemory/InMemoryDatastore.UpdateShardState","Parent":{"Remote":false,"SpanID":"f0a35288cd20ad97","TraceFlags":"01","TraceID":"973d6d1d6e0bb78f9ae744321dbdd9c4","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"62374a79db824c09","TraceFlags":"01","TraceID":"973d6d1d6e0bb78f9ae744321dbdd9c4","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.539072+01:00","Status":{"Code":"Unset","Description":""}}
    tracer.go:242: 16:46:25.539 | TRC system/tracer.go:242 > {"Attributes":null,"ChildSpanCount":0,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.539484491+01:00","Events":null,"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"pkg/localdb/inmemory/InMemoryDatastore.GetJob","Parent":{"Remote":false,"SpanID":"f0a35288cd20ad97","TraceFlags":"01","TraceID":"973d6d1d6e0bb78f9ae744321dbdd9c4","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"833c48fd337dfa6e","TraceFlags":"01","TraceID":"973d6d1d6e0bb78f9ae744321dbdd9c4","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.539482+01:00","Status":{"Code":"Unset","Description":""}}
    tracer.go:242: 16:46:25.54 | TRC system/tracer.go:242 > {"Attributes":null,"ChildSpanCount":0,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.539776883+01:00","Events":null,"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"pkg/localdb/inmemory/InMemoryDatastore.GetJob","Parent":{"Remote":false,"SpanID":"bfe9c0db83c5ee45","TraceFlags":"01","TraceID":"973d6d1d6e0bb78f9ae744321dbdd9c4","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"0d7c05fba0b3a4c9","TraceFlags":"01","TraceID":"973d6d1d6e0bb78f9ae744321dbdd9c4","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.539775+01:00","Status":{"Code":"Unset","Description":""}}
    tracer.go:242: 16:46:25.54 | TRC system/tracer.go:242 > {"Attributes":[{"Key":"environment","Value":{"Type":"STRING","Value":"test"}}],"ChildSpanCount":1,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.540095466+01:00","Events":null,"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"service/HandleJobEvent","Parent":{"Remote":false,"SpanID":"f0a35288cd20ad97","TraceFlags":"01","TraceID":"973d6d1d6e0bb78f9ae744321dbdd9c4","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"bfe9c0db83c5ee45","TraceFlags":"01","TraceID":"973d6d1d6e0bb78f9ae744321dbdd9c4","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.539772+01:00","Status":{"Code":"Unset","Description":""}}
    tracer.go:301: 16:46:25.568 | TRC system/tracer.go:301 > adding jobid from baggage to span as attribute: jobid=ce33a849-e433-42c5-a1bc-1e30ea5f2c53
    tracer.go:242: 16:46:25.568 | TRC system/tracer.go:242 > {"Attributes":[{"Key":"jobid","Value":{"Type":"STRING","Value":"ce33a849-e433-42c5-a1bc-1e30ea5f2c53"}}],"ChildSpanCount":0,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.56866317+01:00","Events":null,"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"pkg/localdb/inmemory/InMemoryDatastore.GetJobState","Parent":{"Remote":false,"SpanID":"f06e4f30123188f7","TraceFlags":"01","TraceID":"61cf27622e717cb12c2fb040f2487d11","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"a2fe16b9042678fa","TraceFlags":"01","TraceID":"61cf27622e717cb12c2fb040f2487d11","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.568535+01:00","Status":{"Code":"Unset","Description":""}}
    tracer.go:242: 16:46:25.569 | TRC system/tracer.go:242 > {"Attributes":null,"ChildSpanCount":1,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.568805932+01:00","Events":null,"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"pkg/publicapi/getJobStateFromRequest","Parent":{"Remote":false,"SpanID":"f902384e9bc29976","TraceFlags":"01","TraceID":"61cf27622e717cb12c2fb040f2487d11","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"f06e4f30123188f7","TraceFlags":"01","TraceID":"61cf27622e717cb12c2fb040f2487d11","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.568534+01:00","Status":{"Code":"Unset","Description":""}}
    tracer.go:242: 16:46:25.569 | TRC system/tracer.go:242 > {"Attributes":null,"ChildSpanCount":0,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.569035501+01:00","Events":null,"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"pkg/publicapi/states","Parent":{"Remote":false,"SpanID":"f902384e9bc29976","TraceFlags":"01","TraceID":"61cf27622e717cb12c2fb040f2487d11","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"c6187a60c4b93210","TraceFlags":"01","TraceID":"61cf27622e717cb12c2fb040f2487d11","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.568499+01:00","Status":{"Code":"Unset","Description":""}}
    tracer.go:242: 16:46:25.569 | TRC system/tracer.go:242 > {"Attributes":[{"Key":"net.transport","Value":{"Type":"STRING","Value":"ip_tcp"}},{"Key":"net.peer.ip","Value":{"Type":"STRING","Value":"127.0.0.1"}},{"Key":"net.peer.port","Value":{"Type":"INT64","Value":54929}},{"Key":"net.host.ip","Value":{"Type":"STRING","Value":"0.0.0.0"}},{"Key":"net.host.port","Value":{"Type":"INT64","Value":54894}},{"Key":"http.method","Value":{"Type":"STRING","Value":"POST"}},{"Key":"http.target","Value":{"Type":"STRING","Value":"/states"}},{"Key":"http.server_name","Value":{"Type":"STRING","Value":"pkg/publicapi/states"}},{"Key":"http.user_agent","Value":{"Type":"STRING","Value":"Go-http-client/1.1"}},{"Key":"http.request_content_length","Value":{"Type":"INT64","Value":129}},{"Key":"http.scheme","Value":{"Type":"STRING","Value":"http"}},{"Key":"http.host","Value":{"Type":"STRING","Value":"0.0.0.0:54894"}},{"Key":"http.flavor","Value":{"Type":"STRING","Value":"1.1"}},{"Key":"http.read_bytes","Value":{"Type":"INT64","Value":129}},{"Key":"http.wrote_bytes","Value":{"Type":"INT64","Value":446}},{"Key":"http.status_code","Value":{"Type":"INT64","Value":200}}],"ChildSpanCount":2,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.569300222+01:00","Events":null,"InstrumentationLibrary":{"Name":"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp","SchemaURL":"","Version":"semver:0.32.0"},"Links":null,"Name":"pkg/publicapi/states","Parent":{"Remote":true,"SpanID":"c4f9959c8f063ac5","TraceFlags":"01","TraceID":"61cf27622e717cb12c2fb040f2487d11","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"f902384e9bc29976","TraceFlags":"01","TraceID":"61cf27622e717cb12c2fb040f2487d11","TraceState":""},"SpanKind":2,"StartTime":"2022-10-27T16:46:25.568485+01:00","Status":{"Code":"Unset","Description":""}}
    tracer.go:242: 16:46:25.57 | TRC system/tracer.go:242 > {"Attributes":[{"Key":"clientID","Value":{"Type":"STRING","Value":"ac8250fe6ff6460e66dc7cce9e7199d0d292829186354c87c8d0571d34c20914"}},{"Key":"http.method","Value":{"Type":"STRING","Value":"POST"}},{"Key":"http.url","Value":{"Type":"STRING","Value":"http://0.0.0.0:54894/states"}},{"Key":"http.request_content_length","Value":{"Type":"INT64","Value":129}},{"Key":"http.scheme","Value":{"Type":"STRING","Value":"http"}},{"Key":"http.host","Value":{"Type":"STRING","Value":"0.0.0.0:54894"}},{"Key":"http.flavor","Value":{"Type":"STRING","Value":"1.1"}},{"Key":"http.status_code","Value":{"Type":"INT64","Value":200}}],"ChildSpanCount":0,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.569703082+01:00","Events":null,"InstrumentationLibrary":{"Name":"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp","SchemaURL":"","Version":"semver:0.32.0"},"Links":null,"Name":"HTTP POST","Parent":{"Remote":false,"SpanID":"db1f890e072c3669","TraceFlags":"01","TraceID":"61cf27622e717cb12c2fb040f2487d11","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"c4f9959c8f063ac5","TraceFlags":"01","TraceID":"61cf27622e717cb12c2fb040f2487d11","TraceState":""},"SpanKind":3,"StartTime":"2022-10-27T16:46:25.567825+01:00","Status":{"Code":"Unset","Description":""}}
    tracer.go:242: 16:46:25.57 | TRC system/tracer.go:242 > {"Attributes":null,"ChildSpanCount":1,"DroppedAttributes":0,"DroppedEvents":0,"DroppedLinks":0,"EndTime":"2022-10-27T16:46:25.570036215+01:00","Events":null,"InstrumentationLibrary":{"Name":"bacalhau-v0.0.0-xxxxxxx","SchemaURL":"","Version":""},"Links":null,"Name":"pkg/publicapi.post","Parent":{"Remote":false,"SpanID":"1ef737011aa86db7","TraceFlags":"01","TraceID":"61cf27622e717cb12c2fb040f2487d11","TraceState":""},"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:___10TestLotusNodeSuite_in_github_com_filecoin_project_bacalhau_pkg_test_devstack.test"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.7.0"}}],"SpanContext":{"Remote":false,"SpanID":"db1f890e072c3669","TraceFlags":"01","TraceID":"61cf27622e717cb12c2fb040f2487d11","TraceState":""},"SpanKind":1,"StartTime":"2022-10-27T16:46:25.567763+01:00","Status":{"Code":"Unset","Description":""}}
    lotus_test.go:84: 
        	Error Trace:	/Users/wjam/dev/winder-ai/bacalhau/pkg/test/devstack/lotus_test.go:84
        	Error:      	Received unexpected error:
        	            	job has error state Error on node QmVx8sT11fzkW7CmVPYyfiqGjxYEszfxWJ7mxqYVRdtKvs (invalid action ActionResultReceived in state WaitingToPublishResults)
        	Test:       	TestLotusNodeSuite/TestLotusNode

@aronchick
Copy link
Collaborator

Stale?

@aronchick aronchick reopened this Feb 16, 2023
github-actions bot pushed a commit that referenced this issue Oct 14, 2023
Bumps [watchdog](https://github.com/gorakhargosh/watchdog) from 0.9.0 to
3.0.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/gorakhargosh/watchdog/releases">watchdog's
releases</a>.</em></p>
<blockquote>
<h2>3.0.0</h2>
<ul>
<li>Drop support for Python 3.6.</li>
<li><code>watchdog</code> is now PEP 561 compatible, and tested with
<code>mypy</code></li>
<li>Fix missing <code>&gt;</code> in
<code>FileSystemEvent.__repr__()</code> (<a
href="https://github.com/gorakhargosh/watchdog/issues/980">#980</a>)</li>
<li>[ci] Lots of improvements</li>
<li>[inotify] Return from <code>InotifyEmitter.queue_events()</code> if
not launched when thread is inactive (<a
href="https://github.com/gorakhargosh/watchdog/issues/963">#963</a>)</li>
<li>[tests] Stability improvements</li>
<li>[utils] Remove handling of <code>threading.Event.isSet</code>
spelling (<a
href="https://github.com/gorakhargosh/watchdog/issues/962">#962</a>)</li>
<li>[watchmedo] Fixed tricks YAML generation (<a
href="https://github.com/gorakhargosh/watchdog/issues/965">#965</a>)</li>
</ul>
<p>:heart_decoration: Thanks to our beloved contributors: <a
href="https://github.com/kurtmckee"><code>@​kurtmckee</code></a>, <a
href="https://github.com/altendky"><code>@​altendky</code></a>, <a
href="https://github.com/agroszer"><code>@​agroszer</code></a>, <a
href="https://github.com/BoboTiG"><code>@​BoboTiG</code></a></p>
<h2>2.3.1</h2>
<ul>
<li>Run <code>black</code> on the entire source code</li>
<li>Bundle the <code>requirements-tests.txt</code> file in the source
distribution (<a
href="https://github.com/gorakhargosh/watchdog/issues/939">#939</a>)</li>
<li>[watchmedo] Exclude <code>FileOpenedEvent</code> events from
<code>AutoRestartTrick</code>, and <code>ShellCommandTrick</code>, to
restore watchdog &lt; 2.3.0 behavior. A better solution should be found
in the future. (<a
href="https://github.com/gorakhargosh/watchdog/issues/949">#949</a>)</li>
<li>[watchmedo] Log <code>FileOpenedEvent</code>, and
<code>FileClosedEvent</code>, events in <code>LoggerTrick</code></li>
</ul>
<p>:heart_decoration: Thanks to our beloved contributors: <a
href="https://github.com/BoboTiG"><code>@​BoboTiG</code></a></p>
<h2>2.3.0</h2>
<ul>
<li>[inotify] Add support for <code>IN_OPEN</code> events: a
<code>FileOpenedEvent</code> event will be fired (<a
href="https://github.com/gorakhargosh/watchdog/issues/941">#941</a>)</li>
<li>[watchmedo] Add optional event debouncing for
<code>auto-restart</code>, only restarting once if many events happen in
quick succession (<code>--debounce-interval</code>) (<a
href="https://github.com/gorakhargosh/watchdog/issues/940">#940</a>)</li>
<li>[watchmedo] Add option to not auto-restart the command after it
exits (<code>--no-restart-on-command-exit</code>) (<a
href="https://github.com/gorakhargosh/watchdog/issues/946">#946</a>)</li>
<li>[watchmedo] Exit gracefully on <code>KeyboardInterrupt</code>
exception (Ctrl+C) (<a
href="https://github.com/gorakhargosh/watchdog/issues/945">#945</a>)</li>
</ul>
<p>:heart_decoration: Thanks to our beloved contributors: <a
href="https://github.com/BoboTiG"><code>@​BoboTiG</code></a>, <a
href="https://github.com/dstaple"><code>@​dstaple</code></a>, <a
href="https://github.com/taleinat"><code>@​taleinat</code></a>, <a
href="https://github.com/cernekj"><code>@​cernekj</code></a></p>
<h2>2.2.1</h2>
<ul>
<li>Enable mypy to discover type hints as specified in PEP 561 (<a
href="https://github.com/gorakhargosh/watchdog/issues/933">#933</a>)</li>
<li>[ci] Set the expected Python version when building release
files</li>
<li>[ci] Update actions versions in use</li>
<li>[watchmedo] [regression] Fix usage of missing
<code>signal.SIGHUP</code> attribute on non-Unix OSes (<a
href="https://github.com/gorakhargosh/watchdog/issues/935">#935</a>)</li>
</ul>
<p>:heart_decoration: Thanks to our beloved contributors: <a
href="https://github.com/BoboTiG"><code>@​BoboTiG</code></a>, <a
href="https://github.com/simon04"><code>@​simon04</code></a>, <a
href="https://github.com/piotrpdev"><code>@​piotrpdev</code></a></p>
<h2>2.2.0</h2>
<ul>
<li>[build] Wheels are now available for Python 3.11 (<a
href="https://github.com/gorakhargosh/watchdog/issues/932">#932</a>)</li>
<li>[documentation] HTML documentation builds are now tested for errors
(<a
href="https://github.com/gorakhargosh/watchdog/issues/902">#902</a>)</li>
<li>[documentation] Fix typos here, and there (<a
href="https://github.com/gorakhargosh/watchdog/issues/910">#910</a>)</li>
<li>[fsevents2] The <code>fsevents2</code> observer is now deprecated
(<a
href="https://github.com/gorakhargosh/watchdog/issues/909">#909</a>)</li>
<li>[tests] The error message returned by musl libc for error code
<code>-1</code> is now allowed (<a
href="https://github.com/gorakhargosh/watchdog/issues/923">#923</a>)</li>
<li>[utils] Remove unnecessary code in <code>dirsnapshot.py</code> (<a
href="https://github.com/gorakhargosh/watchdog/issues/930">#930</a>)</li>
<li>[watchmedo] Handle shutdown events from <code>SIGHUP</code> (<a
href="https://github.com/gorakhargosh/watchdog/issues/912">#912</a>)</li>
</ul>
<p>:heart_decoration: Thanks to our beloved contributors: <a
href="https://github.com/kurtmckee"><code>@​kurtmckee</code></a>, <a
href="https://github.com/babymastodon"><code>@​babymastodon</code></a>,
<a
href="https://github.com/QuantumEnergyE"><code>@​QuantumEnergyE</code></a>,
<a href="https://github.com/timgates42"><code>@​timgates42</code></a>,
<a href="https://github.com/BoboTiG"><code>@​BoboTiG</code></a></p>
<h2>2.1.9</h2>
<ul>
<li>[fsevents] Fix flakey test to assert that there are no errors when
stopping the emitter.</li>
<li>[inotify] Suppress occasional <code>OSError: [Errno 9] Bad file
descriptor</code> at shutdown. <a
href="https://github.com/gorakhargosh/watchdog/issues/805">#805</a></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/gorakhargosh/watchdog/blob/master/changelog.rst">watchdog's
changelog</a>.</em></p>
<blockquote>
<p>3.0.0</p>
<pre><code>
2023-03-20 • `full history
&lt;https://github.com/gorakhargosh/watchdog/compare/v2.3.1...v3.0.0&gt;`__
<ul>
<li>Drop support for Python 3.6.</li>
<li><code>watchdog</code> is now PEP 561 compatible, and tested with
<code>mypy</code></li>
<li>Fix missing <code>&amp;gt;</code> in
<code>FileSystemEvent.__repr__()</code>
(<code>[#980](gorakhargosh/watchdog#980)
&amp;lt;https://github.com/gorakhargosh/watchdog/pull/980&amp;gt;</code>__)</li>
<li>[ci] Lots of improvements</li>
<li>[inotify] Return from <code>InotifyEmitter.queue_events()</code> if
not launched when thread is inactive
(<code>[#963](gorakhargosh/watchdog#963)
&amp;lt;https://github.com/gorakhargosh/watchdog/pull/963&amp;gt;</code>__)</li>
<li>[tests] Stability improvements</li>
<li>[utils] Remove handling of <code>threading.Event.isSet</code>
spelling
(<code>[#962](gorakhargosh/watchdog#962)
&amp;lt;https://github.com/gorakhargosh/watchdog/pull/962&amp;gt;</code>__)</li>
<li>[watchmedo] Fixed tricks YAML generation
(<code>[#965](gorakhargosh/watchdog#965)
&amp;lt;https://github.com/gorakhargosh/watchdog/pull/965&amp;gt;</code>__)</li>
<li>Thanks to our beloved contributors: <a
href="https://github.com/kurtmckee"><code>@​kurtmckee</code></a>, <a
href="https://github.com/altendky"><code>@​altendky</code></a>, <a
href="https://github.com/agroszer"><code>@​agroszer</code></a>, <a
href="https://github.com/BoboTiG"><code>@​BoboTiG</code></a></li>
</ul>
<p>2.3.1
</code></pre></p>
<p>2023-02-28 • <code>full history
&lt;https://github.com/gorakhargosh/watchdog/compare/v2.3.0...v2.3.1&gt;</code>__</p>
<ul>
<li>Run <code>black</code> on the entire source code</li>
<li>Bundle the <code>requirements-tests.txt</code> file in the source
distribution
(<code>[#939](gorakhargosh/watchdog#939)
&lt;https://github.com/gorakhargosh/watchdog/pull/939&gt;</code>__)</li>
<li>[watchmedo] Exclude <code>FileOpenedEvent</code> events from
<code>AutoRestartTrick</code>, and <code>ShellCommandTrick</code>, to
restore watchdog <!-- raw HTML omitted -->`__)</li>
<li>[watchmedo] Log <code>FileOpenedEvent</code>, and
<code>FileClosedEvent</code>, events in <code>LoggerTrick</code></li>
<li>Thanks to our beloved contributors: <a
href="https://github.com/BoboTiG"><code>@​BoboTiG</code></a></li>
</ul>
<p>2.3.0</p>
<pre><code>
2023-02-23 • `full history
&lt;https://github.com/gorakhargosh/watchdog/compare/v2.2.1...v2.3.0&gt;`__
<ul>
<li>[inotify] Add support for <code>IN_OPEN</code> events: a
<code>FileOpenedEvent</code> event will be fired
(<code>[#941](gorakhargosh/watchdog#941)
&amp;lt;https://github.com/gorakhargosh/watchdog/pull/941&amp;gt;</code>__)</li>
<li>[watchmedo] Add optional event debouncing for
<code>auto-restart</code>, only restarting once if many events happen in
quick succession (<code>--debounce-interval</code>)
(<code>[#940](gorakhargosh/watchdog#940)
&amp;lt;https://github.com/gorakhargosh/watchdog/pull/940&amp;gt;</code>__)</li>
<li>[watchmedo] Exit gracefully on <code>KeyboardInterrupt</code>
exception (Ctrl+C)
(<code>[#945](gorakhargosh/watchdog#945)
&amp;lt;https://github.com/gorakhargosh/watchdog/pull/945&amp;gt;</code>__)</li>
<li>[watchmedo] Add option to not auto-restart the command after it
exits (<code>--no-restart-on-command-exit</code>)
(<code>[#946](gorakhargosh/watchdog#946)
&amp;lt;https://github.com/gorakhargosh/watchdog/pull/946&amp;gt;</code>__)</li>
<li>Thanks to our beloved contributors: <a
href="https://github.com/BoboTiG"><code>@​BoboTiG</code></a>, <a
href="https://github.com/dstaple"><code>@​dstaple</code></a>, <a
href="https://github.com/taleinat"><code>@​taleinat</code></a>, <a
href="https://github.com/cernekj"><code>@​cernekj</code></a></li>
</ul>
<p>2.2.1
</code></pre></p>
<p>2023-01-01 • <code>full history
&lt;https://github.com/gorakhargosh/watchdog/compare/v2.2.0...v2.2.1&gt;</code>__</p>
<ul>
<li>Enable <code>mypy</code> to discover type hints as specified in PEP
561 (<code>[#933](gorakhargosh/watchdog#933)
&lt;https://github.com/gorakhargosh/watchdog/pull/933&gt;</code>__)</li>
<li>[ci] Set the expected Python version when building release
files</li>
<li>[ci] Update actions versions in use</li>
<li>[watchmedo] [regression] Fix usage of missing
<code>signal.SIGHUP</code> attribute on non-Unix OSes
(<code>[#935](gorakhargosh/watchdog#935)
&lt;https://github.com/gorakhargosh/watchdog/pull/935&gt;</code>__)</li>
<li>Thanks to our beloved contributors: <a
href="https://github.com/BoboTiG"><code>@​BoboTiG</code></a>, <a
href="https://github.com/simon04"><code>@​simon04</code></a>, <a
href="https://github.com/piotrpdev"><code>@​piotrpdev</code></a></li>
</ul>
<p>2.2.0</p>
<pre><code>&lt;/tr&gt;&lt;/table&gt; 
</code></pre>
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/gorakhargosh/watchdog/commit/da09c060a007fe7fddde27592e4f63ae1e8697bc"><code>da09c06</code></a>
Release 3.0.0</li>
<li><a
href="https://github.com/gorakhargosh/watchdog/commit/9fc1ce2eb04cbe9ca9321949c3962015c2699a06"><code>9fc1ce2</code></a>
fix: missing <code>&gt;</code> in
<code>FileSystemEvent.__repr__()</code> (<a
href="https://github.com/gorakhargosh/watchdog/issues/980">#980</a>)</li>
<li><a
href="https://github.com/gorakhargosh/watchdog/commit/1838e0b19df98c1ec82acd1d681b7092403e8848"><code>1838e0b</code></a>
doc: clean-up</li>
<li><a
href="https://github.com/gorakhargosh/watchdog/commit/ce6218cebbb17c39adfe35e69124dc3390368196"><code>ce6218c</code></a>
Update global.rst.inc</li>
<li><a
href="https://github.com/gorakhargosh/watchdog/commit/989fddcb3963781e702e7c688e3871b416e2b53e"><code>989fddc</code></a>
Update installation.rst</li>
<li><a
href="https://github.com/gorakhargosh/watchdog/commit/71f2df38b4092ba6674f59be10a91c51585d4aa7"><code>71f2df3</code></a>
Update README.rst</li>
<li><a
href="https://github.com/gorakhargosh/watchdog/commit/9c28c61d001a6c05d47e4c0bd852bbb1dac78e2e"><code>9c28c61</code></a>
mypy check_untyped_defs (<a
href="https://github.com/gorakhargosh/watchdog/issues/966">#966</a>)</li>
<li><a
href="https://github.com/gorakhargosh/watchdog/commit/764a23494b82e163b92617377d0347cf72e304d2"><code>764a234</code></a>
tests: refactor test setups towards fixtures and hinting (<a
href="https://github.com/gorakhargosh/watchdog/issues/968">#968</a>)</li>
<li><a
href="https://github.com/gorakhargosh/watchdog/commit/ddb9bd1ba10c1ecfc79e67730ea55bcc5c870809"><code>ddb9bd1</code></a>
tests: xfail tests until we work on them (<a
href="https://github.com/gorakhargosh/watchdog/issues/975">#975</a>)</li>
<li><a
href="https://github.com/gorakhargosh/watchdog/commit/344f342123e46aaac06a398b8b7b8592a1c5251d"><code>344f342</code></a>
tests: skip pypy on windows (<a
href="https://github.com/gorakhargosh/watchdog/issues/976">#976</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/gorakhargosh/watchdog/compare/v0.9.0...v3.0.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=watchdog&package-manager=pip&previous-version=0.9.0&new-version=3.0.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
wdbaruni pushed a commit that referenced this issue Oct 16, 2023
Bumps [watchdog](https://github.com/gorakhargosh/watchdog) from 0.9.0 to
3.0.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/gorakhargosh/watchdog/releases">watchdog's
releases</a>.</em></p>
<blockquote>
<h2>3.0.0</h2>
<ul>
<li>Drop support for Python 3.6.</li>
<li><code>watchdog</code> is now PEP 561 compatible, and tested with
<code>mypy</code></li>
<li>Fix missing <code>&gt;</code> in
<code>FileSystemEvent.__repr__()</code> (<a
href="https://github.com/gorakhargosh/watchdog/issues/980">#980</a>)</li>
<li>[ci] Lots of improvements</li>
<li>[inotify] Return from <code>InotifyEmitter.queue_events()</code> if
not launched when thread is inactive (<a
href="https://github.com/gorakhargosh/watchdog/issues/963">#963</a>)</li>
<li>[tests] Stability improvements</li>
<li>[utils] Remove handling of <code>threading.Event.isSet</code>
spelling (<a
href="https://github.com/gorakhargosh/watchdog/issues/962">#962</a>)</li>
<li>[watchmedo] Fixed tricks YAML generation (<a
href="https://github.com/gorakhargosh/watchdog/issues/965">#965</a>)</li>
</ul>
<p>:heart_decoration: Thanks to our beloved contributors: <a
href="https://github.com/kurtmckee"><code>@​kurtmckee</code></a>, <a
href="https://github.com/altendky"><code>@​altendky</code></a>, <a
href="https://github.com/agroszer"><code>@​agroszer</code></a>, <a
href="https://github.com/BoboTiG"><code>@​BoboTiG</code></a></p>
<h2>2.3.1</h2>
<ul>
<li>Run <code>black</code> on the entire source code</li>
<li>Bundle the <code>requirements-tests.txt</code> file in the source
distribution (<a
href="https://github.com/gorakhargosh/watchdog/issues/939">#939</a>)</li>
<li>[watchmedo] Exclude <code>FileOpenedEvent</code> events from
<code>AutoRestartTrick</code>, and <code>ShellCommandTrick</code>, to
restore watchdog &lt; 2.3.0 behavior. A better solution should be found
in the future. (<a
href="https://github.com/gorakhargosh/watchdog/issues/949">#949</a>)</li>
<li>[watchmedo] Log <code>FileOpenedEvent</code>, and
<code>FileClosedEvent</code>, events in <code>LoggerTrick</code></li>
</ul>
<p>:heart_decoration: Thanks to our beloved contributors: <a
href="https://github.com/BoboTiG"><code>@​BoboTiG</code></a></p>
<h2>2.3.0</h2>
<ul>
<li>[inotify] Add support for <code>IN_OPEN</code> events: a
<code>FileOpenedEvent</code> event will be fired (<a
href="https://github.com/gorakhargosh/watchdog/issues/941">#941</a>)</li>
<li>[watchmedo] Add optional event debouncing for
<code>auto-restart</code>, only restarting once if many events happen in
quick succession (<code>--debounce-interval</code>) (<a
href="https://github.com/gorakhargosh/watchdog/issues/940">#940</a>)</li>
<li>[watchmedo] Add option to not auto-restart the command after it
exits (<code>--no-restart-on-command-exit</code>) (<a
href="https://github.com/gorakhargosh/watchdog/issues/946">#946</a>)</li>
<li>[watchmedo] Exit gracefully on <code>KeyboardInterrupt</code>
exception (Ctrl+C) (<a
href="https://github.com/gorakhargosh/watchdog/issues/945">#945</a>)</li>
</ul>
<p>:heart_decoration: Thanks to our beloved contributors: <a
href="https://github.com/BoboTiG"><code>@​BoboTiG</code></a>, <a
href="https://github.com/dstaple"><code>@​dstaple</code></a>, <a
href="https://github.com/taleinat"><code>@​taleinat</code></a>, <a
href="https://github.com/cernekj"><code>@​cernekj</code></a></p>
<h2>2.2.1</h2>
<ul>
<li>Enable mypy to discover type hints as specified in PEP 561 (<a
href="https://github.com/gorakhargosh/watchdog/issues/933">#933</a>)</li>
<li>[ci] Set the expected Python version when building release
files</li>
<li>[ci] Update actions versions in use</li>
<li>[watchmedo] [regression] Fix usage of missing
<code>signal.SIGHUP</code> attribute on non-Unix OSes (<a
href="https://github.com/gorakhargosh/watchdog/issues/935">#935</a>)</li>
</ul>
<p>:heart_decoration: Thanks to our beloved contributors: <a
href="https://github.com/BoboTiG"><code>@​BoboTiG</code></a>, <a
href="https://github.com/simon04"><code>@​simon04</code></a>, <a
href="https://github.com/piotrpdev"><code>@​piotrpdev</code></a></p>
<h2>2.2.0</h2>
<ul>
<li>[build] Wheels are now available for Python 3.11 (<a
href="https://github.com/gorakhargosh/watchdog/issues/932">#932</a>)</li>
<li>[documentation] HTML documentation builds are now tested for errors
(<a
href="https://github.com/gorakhargosh/watchdog/issues/902">#902</a>)</li>
<li>[documentation] Fix typos here, and there (<a
href="https://github.com/gorakhargosh/watchdog/issues/910">#910</a>)</li>
<li>[fsevents2] The <code>fsevents2</code> observer is now deprecated
(<a
href="https://github.com/gorakhargosh/watchdog/issues/909">#909</a>)</li>
<li>[tests] The error message returned by musl libc for error code
<code>-1</code> is now allowed (<a
href="https://github.com/gorakhargosh/watchdog/issues/923">#923</a>)</li>
<li>[utils] Remove unnecessary code in <code>dirsnapshot.py</code> (<a
href="https://github.com/gorakhargosh/watchdog/issues/930">#930</a>)</li>
<li>[watchmedo] Handle shutdown events from <code>SIGHUP</code> (<a
href="https://github.com/gorakhargosh/watchdog/issues/912">#912</a>)</li>
</ul>
<p>:heart_decoration: Thanks to our beloved contributors: <a
href="https://github.com/kurtmckee"><code>@​kurtmckee</code></a>, <a
href="https://github.com/babymastodon"><code>@​babymastodon</code></a>,
<a
href="https://github.com/QuantumEnergyE"><code>@​QuantumEnergyE</code></a>,
<a href="https://github.com/timgates42"><code>@​timgates42</code></a>,
<a href="https://github.com/BoboTiG"><code>@​BoboTiG</code></a></p>
<h2>2.1.9</h2>
<ul>
<li>[fsevents] Fix flakey test to assert that there are no errors when
stopping the emitter.</li>
<li>[inotify] Suppress occasional <code>OSError: [Errno 9] Bad file
descriptor</code> at shutdown. <a
href="https://github.com/gorakhargosh/watchdog/issues/805">#805</a></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/gorakhargosh/watchdog/blob/master/changelog.rst">watchdog's
changelog</a>.</em></p>
<blockquote>
<p>3.0.0</p>
<pre><code>
2023-03-20 • `full history
&lt;https://github.com/gorakhargosh/watchdog/compare/v2.3.1...v3.0.0&gt;`__
<ul>
<li>Drop support for Python 3.6.</li>
<li><code>watchdog</code> is now PEP 561 compatible, and tested with
<code>mypy</code></li>
<li>Fix missing <code>&amp;gt;</code> in
<code>FileSystemEvent.__repr__()</code>
(<code>[#980](gorakhargosh/watchdog#980)
&amp;lt;https://github.com/gorakhargosh/watchdog/pull/980&amp;gt;</code>__)</li>
<li>[ci] Lots of improvements</li>
<li>[inotify] Return from <code>InotifyEmitter.queue_events()</code> if
not launched when thread is inactive
(<code>[#963](gorakhargosh/watchdog#963)
&amp;lt;https://github.com/gorakhargosh/watchdog/pull/963&amp;gt;</code>__)</li>
<li>[tests] Stability improvements</li>
<li>[utils] Remove handling of <code>threading.Event.isSet</code>
spelling
(<code>[#962](gorakhargosh/watchdog#962)
&amp;lt;https://github.com/gorakhargosh/watchdog/pull/962&amp;gt;</code>__)</li>
<li>[watchmedo] Fixed tricks YAML generation
(<code>[#965](gorakhargosh/watchdog#965)
&amp;lt;https://github.com/gorakhargosh/watchdog/pull/965&amp;gt;</code>__)</li>
<li>Thanks to our beloved contributors: <a
href="https://github.com/kurtmckee"><code>@​kurtmckee</code></a>, <a
href="https://github.com/altendky"><code>@​altendky</code></a>, <a
href="https://github.com/agroszer"><code>@​agroszer</code></a>, <a
href="https://github.com/BoboTiG"><code>@​BoboTiG</code></a></li>
</ul>
<p>2.3.1
</code></pre></p>
<p>2023-02-28 • <code>full history
&lt;https://github.com/gorakhargosh/watchdog/compare/v2.3.0...v2.3.1&gt;</code>__</p>
<ul>
<li>Run <code>black</code> on the entire source code</li>
<li>Bundle the <code>requirements-tests.txt</code> file in the source
distribution
(<code>[#939](gorakhargosh/watchdog#939)
&lt;https://github.com/gorakhargosh/watchdog/pull/939&gt;</code>__)</li>
<li>[watchmedo] Exclude <code>FileOpenedEvent</code> events from
<code>AutoRestartTrick</code>, and <code>ShellCommandTrick</code>, to
restore watchdog <!-- raw HTML omitted -->`__)</li>
<li>[watchmedo] Log <code>FileOpenedEvent</code>, and
<code>FileClosedEvent</code>, events in <code>LoggerTrick</code></li>
<li>Thanks to our beloved contributors: <a
href="https://github.com/BoboTiG"><code>@​BoboTiG</code></a></li>
</ul>
<p>2.3.0</p>
<pre><code>
2023-02-23 • `full history
&lt;https://github.com/gorakhargosh/watchdog/compare/v2.2.1...v2.3.0&gt;`__
<ul>
<li>[inotify] Add support for <code>IN_OPEN</code> events: a
<code>FileOpenedEvent</code> event will be fired
(<code>[#941](gorakhargosh/watchdog#941)
&amp;lt;https://github.com/gorakhargosh/watchdog/pull/941&amp;gt;</code>__)</li>
<li>[watchmedo] Add optional event debouncing for
<code>auto-restart</code>, only restarting once if many events happen in
quick succession (<code>--debounce-interval</code>)
(<code>[#940](gorakhargosh/watchdog#940)
&amp;lt;https://github.com/gorakhargosh/watchdog/pull/940&amp;gt;</code>__)</li>
<li>[watchmedo] Exit gracefully on <code>KeyboardInterrupt</code>
exception (Ctrl+C)
(<code>[#945](gorakhargosh/watchdog#945)
&amp;lt;https://github.com/gorakhargosh/watchdog/pull/945&amp;gt;</code>__)</li>
<li>[watchmedo] Add option to not auto-restart the command after it
exits (<code>--no-restart-on-command-exit</code>)
(<code>[#946](gorakhargosh/watchdog#946)
&amp;lt;https://github.com/gorakhargosh/watchdog/pull/946&amp;gt;</code>__)</li>
<li>Thanks to our beloved contributors: <a
href="https://github.com/BoboTiG"><code>@​BoboTiG</code></a>, <a
href="https://github.com/dstaple"><code>@​dstaple</code></a>, <a
href="https://github.com/taleinat"><code>@​taleinat</code></a>, <a
href="https://github.com/cernekj"><code>@​cernekj</code></a></li>
</ul>
<p>2.2.1
</code></pre></p>
<p>2023-01-01 • <code>full history
&lt;https://github.com/gorakhargosh/watchdog/compare/v2.2.0...v2.2.1&gt;</code>__</p>
<ul>
<li>Enable <code>mypy</code> to discover type hints as specified in PEP
561 (<code>[#933](gorakhargosh/watchdog#933)
&lt;https://github.com/gorakhargosh/watchdog/pull/933&gt;</code>__)</li>
<li>[ci] Set the expected Python version when building release
files</li>
<li>[ci] Update actions versions in use</li>
<li>[watchmedo] [regression] Fix usage of missing
<code>signal.SIGHUP</code> attribute on non-Unix OSes
(<code>[#935](gorakhargosh/watchdog#935)
&lt;https://github.com/gorakhargosh/watchdog/pull/935&gt;</code>__)</li>
<li>Thanks to our beloved contributors: <a
href="https://github.com/BoboTiG"><code>@​BoboTiG</code></a>, <a
href="https://github.com/simon04"><code>@​simon04</code></a>, <a
href="https://github.com/piotrpdev"><code>@​piotrpdev</code></a></li>
</ul>
<p>2.2.0</p>
<pre><code>&lt;/tr&gt;&lt;/table&gt; 
</code></pre>
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/gorakhargosh/watchdog/commit/da09c060a007fe7fddde27592e4f63ae1e8697bc"><code>da09c06</code></a>
Release 3.0.0</li>
<li><a
href="https://github.com/gorakhargosh/watchdog/commit/9fc1ce2eb04cbe9ca9321949c3962015c2699a06"><code>9fc1ce2</code></a>
fix: missing <code>&gt;</code> in
<code>FileSystemEvent.__repr__()</code> (<a
href="https://github.com/gorakhargosh/watchdog/issues/980">#980</a>)</li>
<li><a
href="https://github.com/gorakhargosh/watchdog/commit/1838e0b19df98c1ec82acd1d681b7092403e8848"><code>1838e0b</code></a>
doc: clean-up</li>
<li><a
href="https://github.com/gorakhargosh/watchdog/commit/ce6218cebbb17c39adfe35e69124dc3390368196"><code>ce6218c</code></a>
Update global.rst.inc</li>
<li><a
href="https://github.com/gorakhargosh/watchdog/commit/989fddcb3963781e702e7c688e3871b416e2b53e"><code>989fddc</code></a>
Update installation.rst</li>
<li><a
href="https://github.com/gorakhargosh/watchdog/commit/71f2df38b4092ba6674f59be10a91c51585d4aa7"><code>71f2df3</code></a>
Update README.rst</li>
<li><a
href="https://github.com/gorakhargosh/watchdog/commit/9c28c61d001a6c05d47e4c0bd852bbb1dac78e2e"><code>9c28c61</code></a>
mypy check_untyped_defs (<a
href="https://github.com/gorakhargosh/watchdog/issues/966">#966</a>)</li>
<li><a
href="https://github.com/gorakhargosh/watchdog/commit/764a23494b82e163b92617377d0347cf72e304d2"><code>764a234</code></a>
tests: refactor test setups towards fixtures and hinting (<a
href="https://github.com/gorakhargosh/watchdog/issues/968">#968</a>)</li>
<li><a
href="https://github.com/gorakhargosh/watchdog/commit/ddb9bd1ba10c1ecfc79e67730ea55bcc5c870809"><code>ddb9bd1</code></a>
tests: xfail tests until we work on them (<a
href="https://github.com/gorakhargosh/watchdog/issues/975">#975</a>)</li>
<li><a
href="https://github.com/gorakhargosh/watchdog/commit/344f342123e46aaac06a398b8b7b8592a1c5251d"><code>344f342</code></a>
tests: skip pypy on windows (<a
href="https://github.com/gorakhargosh/watchdog/issues/976">#976</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/gorakhargosh/watchdog/compare/v0.9.0...v3.0.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=watchdog&package-manager=pip&previous-version=0.9.0&new-version=3.0.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug Type: Something is not working as expected
Projects
None yet
Development

No branches or pull requests

3 participants