Remove broken "Test unstable" GitHub action #3064
Conversation
**What does this PR do?**:
The "Test unstable" GitHub Action runs the dd-trace-rb CI with
ruby-head and jruby-head.
Because ruby-head and jruby-head aren't always rock solid, we've marked
these tests with "continue-on-error: true" which in practice means you
need to actually open them to know if they failed, because they get
always marked as succeeded.
Today, while looking into our CI config, I noticed these tests have
been broken for what looks like *months* (due to some refactoring
in our gemspec):
```
[!] There was an error parsing `Gemfile`:
[!] There was an error while loading `ddtrace.gemspec`: syntax error, unexpected ']', expecting `end' or dummy end - ...ERSION::MAXIMUM_RUBY_VERSION}"]
... ^
. Bundler cannot continue.
# from /home/runner/work/dd-trace-rb/dd-trace-rb/ddtrace.gemspec:10
# -------------------------------------------
# spec.version = DDTrace::VERSION::STRING
> "< #{DDTrace::VERSION::MAXIMUM_RUBY_VERSION}"]
# spec.required_rubygems_version = '>= 2.0.0'
# -------------------------------------------
. Bundler cannot continue.
# from /home/runner/work/dd-trace-rb/dd-trace-rb/Gemfile:3
# -------------------------------------------
#
> gemspec
#
# -------------------------------------------
Error: The process '/home/runner/.rubies/ruby-head/bin/bundle' failed with exit code 14
```
(https://github.com/DataDog/dd-trace-rb/actions/runs/5890935813/job/15977021676)
Nobody seems to notice when they fail, and thus it really looks like
we're not getting any value at all.
Thus, I decided to open a PR to remove them for now.
**Motivation**:
As our CI gets more complex, we get little value from things that
need to be checked manually -- nowadays we have hundreds of
validations.
**Additional Notes**:
N/A
**How to test the change?**:
Validate that CI is still green, and this action no longer runs.
github-actions
bot
added
the
dev/github
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #3064 +/- ##
=======================================
Coverage 98.10% 98.10%
=======================================
Files 1225 1225
Lines 72846 72846
Branches 3489 3489
=======================================
+ Hits 71463 71465 +2
+ Misses 1383 1381 -2 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Another victim of the lack of support for https://github.com/orgs/community/discussions/15452
|
Oh, this looks interesting! https://github.com/mainmatter/continue-on-error-comment |
|
Interesting... Let me see if I can get that action to work. |
|
Update: I think I'm going to take a bit longer to test out the comment thing. On the other hand, having this action failing is harmless. So I guess for now I'll leave this PR open so I don't forget and will come back to it in a couple of weeks :) |
|
Update: This approach seems to work! I've tested it in libdatadog and works nicely: DataDog/libdatadog#280 . I have it on my todo list to come back to this PR and a) fix the broken thingy, and b) use the continue on error comment to remind us when it broke. |
I think most CI providers are a bit sucky at this... (Or at least, I haven't seen a good solution). |
|
TBH I've been sitting on this PR for a long time as I was kinda planning to do the continue on error but haven't done it. Any thoughts on merging as-is vs waiting for that change? cc @p-datadog @marcotc |
|
I approve this PR as it is. |
|
Ok, let's go ahead with this PR. I've refreshed it to remove the conflict, and out of curiosity I checked a recent run of this job and yeap no surprise there's multiple things broken that are getting marked as ✔️ -- https://github.com/DataDog/dd-trace-rb/actions/runs/9485780142/job/26138547233?pr=3711#step:3:1 . I'll add a separate internal ticket for us to consider reviving this in a better shape, right now we're only really contributing to global warming. |
What does this PR do?:
The "Test unstable" GitHub Action runs the dd-trace-rb CI with ruby-head and jruby-head.
Because ruby-head and jruby-head aren't always rock solid, we've marked these tests with "continue-on-error: true" which in practice means you need to actually open them to know if they failed, because they get always marked as succeeded.
Today, while looking into our CI config, I noticed these tests have been broken for what looks like months (due to some refactoring in our gemspec):
(https://github.com/DataDog/dd-trace-rb/actions/runs/5890935813/job/15977021676)
Nobody seems to notice when they fail, and thus it really looks like we're not getting any value at all.
Thus, I decided to open a PR to remove them for now.
Motivation:
As our CI gets more complex, we get little value from things that need to be checked manually -- nowadays we have hundreds of validations.
Additional Notes:
N/A
How to test the change?:
Validate that CI is still green, and this action no longer runs.