Bump smallrye-open-api from 3.10.0 to 3.12.0, use new builder API

[MikeEdgar]

smallrye-open-api 3.11 introduced a new builder style API that consolidates the process of building an OpenAPI model from the various sources (reader, static file, and annotations). This PR migrates to the new API and picks up several fixes for reported Quarkus issues as noted below.

Fixes #42101
Fixes #42221

[MikeEdgar]

@phillip-kruger , please take a look. Thank you!

[phillip-kruger]

@MikeEdgar I'll have a look on Friday, on the road at the moment. Can you check the ci failure?

[gsmet]

This new version comes with a significant regression in the native executable size.

Not sure what is causing it but I'm not sure we want to go this way given how ubiquitous SmallRye OpenAPI is.

We need to figure out what's causing it and see if we can avoid it.

[MikeEdgar]

I'll check into it.

[MikeEdgar]

This new version comes with a significant regression in the native executable size.

Not sure what is causing it but I'm not sure we want to go this way given how ubiquitous SmallRye OpenAPI is.

We need to figure out what's causing it and see if we can avoid it.

@gsmet is there a simple way to determine the delta in the native image? I suspect the new builder API may be causing additional SmallRye classes to be incorporated.

[MikeEdgar]

@gsmet, @phillip-kruger , I've rebased on main and I noticed that the failing test has been removed since the last run of the CI (#41116). I have not had time to dig in to the issue, but I am curious if this change was flagged as noted in #40076 ?

[MikeEdgar]

@gsmet , I'm not sure how to proceed on this one. The failing native image test is no longer a factor. Do you have concerns that will still be a problem?

[github-actions]

🎊 PR Preview e1577e6 has been successfully built and deployed to https://quarkus-pr-main-41037-preview.surge.sh/version/main/guides/

• Images of blog posts older than 3 months are not available.
• Newsletters older than 3 months are not available.

[quarkus-bot]

Status for workflow Quarkus CI

This is the status report for running Quarkus CI on commit d3798c0.

✅ The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

You can consult the Develocity build scans.

---

Flaky tests - Develocity

⚙️ JVM Tests - JDK 17 Windows

📦 extensions/websockets-next/deployment

✖ io.quarkus.websockets.next.test.EchoWebSocketTest.testEchoMultiConsume - History

• expected: <hello> but was: <null> - org.opentest4j.AssertionFailedError

org.opentest4j.AssertionFailedError: expected: <hello> but was: <null>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
	at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1145)
	at io.quarkus.websockets.next.test.EchoWebSocketTest.assertEcho(EchoWebSocketTest.java:148)

---

⚙️ Native Tests - HTTP

📦 integration-tests/rest-client-reactive

✖ io.quarkus.it.rest.client.BasicTestIT.shouldCreateClientSpans - History

• expected: <1> but was: <2> - org.opentest4j.AssertionFailedError

org.opentest4j.AssertionFailedError: expected: <1> but was: <2>
	at io.quarkus.it.rest.client.BasicTest.shouldCreateClientSpans(BasicTest.java:216)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at io.quarkus.test.junit.QuarkusTestExtension.interceptTestMethod(QuarkusTestExtension.java:810)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

[gsmet]

@MikeEdgar sorry this one slipped. I think we should probably get it in. If the native image metrics are too bad, I suppose it will raise a warning on @zakkak 's side and we can dig deeper from there.

[zakkak]

This PR indeed resulted in significant binary size increase (> 3%) in some of the intergration tests:

Bmark	Timestamp	Total Size (MB)	Diff (%)	Mandrel Version	Quarkus Version	Previous Q version
amazon-lambda-rest-funqy	2024-09-12 19:08:05.966000+00:00	63.8037	3.48%	Mandrel-23.1.4.0-Final	50c2b36	4636861
amazon-lambda-rest-reactive-routes	2024-09-12 19:08:07.183000+00:00	63.8272	4.32%	Mandrel-23.1.4.0-Final	50c2b36	4636861
amazon-lambda-rest-resteasy-reactive	2024-09-12 19:08:06.579000+00:00	65.2647	3.41%	Mandrel-23.1.4.0-Final	50c2b36	4636861
amazon-lambda-rest-servlet	2024-09-12 19:08:05.699000+00:00	65.1123	4.25%	Mandrel-23.1.4.0-Final	50c2b36	4636861
elytron-undertow	2024-09-12 19:07:36.334000+00:00	70.5224	3.14%	Mandrel-23.1.4.0-Final	50c2b36	4636861
resteasy-jackson	2024-09-12 19:07:13.865000+00:00	59.7500	4.01%	Mandrel-23.1.4.0-Final	50c2b36	4636861

[gsmet]

@zakkak does your infra provide a list of the classes that were added?

[zakkak]

@zakkak does your infra provide a list of the classes that were added?

No, I can only get the numbers not specific classes from the collected data. We would need to set quarkus.native.enable-reports in CI to get the list of classes as well, but again we would probably not store the generated reports on our infra, we would probably fetch them from github (as long as they are still available).

[zakkak]

More detailed tables:

amazon-lambda-rest-funqy - 2024-09-12

Quarkus version	4636861	50c2b36	Diff (%)
Total Size (MB)	61.6592	63.8037	3.48%
Reflection stats	---	---	---
Classes (Reflection)	3963	4003	1.01%
Methods (Reflection)	3518	3523	0.14%
Fields (Reflection)	194	194	0.00%
Reachability stats	---	---	---
Classes (Reachability)	13036	13782	5.72%
Methods (Reachability)	68156	70548	3.51%
Fields (Reachability)	18809	19659	4.52%
JNI stats	---	---	---
Classes (JNI)	62	62	0.00%
Methods (JNI)	55	55	0.00%
Fields (JNI)	67	67	0.00%
Classpath stats	---	---	---
Classes (classpath)	14834	15576	5.00%
Methods (classpath)	117460	120831	2.87%
Fields (classpath)	33342	34644	3.90%
Build Perf stats	---	---	---
Total Build Time (s)	124.9208	131.8367	5.54%
GC Time (s)	12.2010	13.4470	10.21%
CPU Load	3.5953	3.5617	-0.93%
Peak RSS (GB)	2.3130	2.3607	2.06%
Available RAM (GB)	15.6065	15.6065	0.00%

amazon-lambda-rest-reactive-routes - 2024-09-12

Quarkus version	4636861	50c2b36	Diff (%)
Total Size (MB)	61.1866	63.8272	4.32%
Reflection stats	---	---	---
Classes (Reflection)	3961	4001	1.01%
Methods (Reflection)	3518	3523	0.14%
Fields (Reflection)	194	194	0.00%
Reachability stats	---	---	---
Classes (Reachability)	13021	13767	5.73%
Methods (Reachability)	68184	70577	3.51%
Fields (Reachability)	18801	19651	4.52%
JNI stats	---	---	---
Classes (JNI)	62	62	0.00%
Methods (JNI)	55	55	0.00%
Fields (JNI)	67	67	0.00%
Classpath stats	---	---	---
Classes (classpath)	14819	15561	5.01%
Methods (classpath)	117428	120801	2.87%
Fields (classpath)	33329	34631	3.91%
Build Perf stats	---	---	---
Total Build Time (s)	127.4898	134.2549	5.31%
GC Time (s)	12.3010	13.8430	12.54%
CPU Load	3.5712	3.5636	-0.21%
Peak RSS (GB)	2.2777	2.2202	-2.52%
Available RAM (GB)	15.6065	15.6065	0.00%

amazon-lambda-rest-resteasy-reactive - 2024-09-12

Quarkus version	4636861	50c2b36	Diff (%)
Total Size (MB)	63.1123	65.2647	3.41%
Reflection stats	---	---	---
Classes (Reflection)	4154	4194	0.96%
Methods (Reflection)	3664	3669	0.14%
Fields (Reflection)	196	196	0.00%
Reachability stats	---	---	---
Classes (Reachability)	13524	14270	5.52%
Methods (Reachability)	70163	72554	3.41%
Fields (Reachability)	19443	20293	4.37%
JNI stats	---	---	---
Classes (JNI)	62	62	0.00%
Methods (JNI)	55	55	0.00%
Fields (JNI)	67	67	0.00%
Classpath stats	---	---	---
Classes (classpath)	15432	16174	4.81%
Methods (classpath)	121145	124517	2.78%
Fields (classpath)	34237	35565	3.88%
Build Perf stats	---	---	---
Total Build Time (s)	130.1454	136.9162	5.20%
GC Time (s)	13.0940	14.6920	12.20%
CPU Load	3.5966	3.5623	-0.95%
Peak RSS (GB)	2.3957	2.2957	-4.18%
Available RAM (GB)	15.6065	15.6065	0.00%

amazon-lambda-rest-servlet - 2024-09-12

Quarkus version	4636861	50c2b36	Diff (%)
Total Size (MB)	62.4600	65.1123	4.25%
Reflection stats	---	---	---
Classes (Reflection)	4079	4119	0.98%
Methods (Reflection)	3568	3573	0.14%
Fields (Reflection)	195	195	0.00%
Reachability stats	---	---	---
Classes (Reachability)	13406	14152	5.56%
Methods (Reachability)	69786	72177	3.43%
Fields (Reachability)	19502	20352	4.36%
JNI stats	---	---	---
Classes (JNI)	62	62	0.00%
Methods (JNI)	55	55	0.00%
Fields (JNI)	67	67	0.00%
Classpath stats	---	---	---
Classes (classpath)	15323	16065	4.84%
Methods (classpath)	121664	125034	2.77%
Fields (classpath)	34346	35648	3.79%
Build Perf stats	---	---	---
Total Build Time (s)	127.7363	135.9078	6.40%
GC Time (s)	12.8600	13.8280	7.53%
CPU Load	3.5655	3.5811	0.44%
Peak RSS (GB)	2.3997	2.2575	-5.92%
Available RAM (GB)	15.6065	15.6065	0.00%

elytron-undertow - 2024-09-12

Quarkus version	4636861	50c2b36	Diff (%)
Total Size (MB)	68.3740	70.5224	3.14%
Reflection stats	---	---	---
Classes (Reflection)	4641	4682	0.88%
Methods (Reflection)	3454	3459	0.14%
Fields (Reflection)	137	137	0.00%
Reachability stats	---	---	---
Classes (Reachability)	15010	15758	4.98%
Methods (Reachability)	76929	79383	3.19%
Fields (Reachability)	23278	24126	3.64%
JNI stats	---	---	---
Classes (JNI)	62	62	0.00%
Methods (JNI)	55	55	0.00%
Fields (JNI)	67	67	0.00%
Classpath stats	---	---	---
Classes (classpath)	17098	17839	4.33%
Methods (classpath)	134608	137966	2.49%
Fields (classpath)	38365	39661	3.38%
Build Perf stats	---	---	---
Total Build Time (s)	139.9706	143.2997	2.38%
GC Time (s)	12.9400	13.2860	2.67%
CPU Load	3.5879	3.5739	-0.39%
Peak RSS (GB)	2.6097	2.6545	1.72%
Available RAM (GB)	15.6065	15.6065	0.00%

resteasy-jackson - 2024-09-12

Quarkus version	4636861	50c2b36	Diff (%)
Total Size (MB)	57.4443	59.7500	4.01%
Reflection stats	---	---	---
Classes (Reflection)	4041	4076	0.87%
Methods (Reflection)	3616	3630	0.39%
Fields (Reflection)	227	225	-0.88%
Reachability stats	---	---	---
Classes (Reachability)	13249	13917	5.04%
Methods (Reachability)	67656	69990	3.45%
Fields (Reachability)	18926	19749	4.35%
JNI stats	---	---	---
Classes (JNI)	62	78	25.81%
Methods (JNI)	55	68	23.64%
Fields (JNI)	67	65	-2.99%
Classpath stats	---	---	---
Classes (classpath)	15134	15759	4.13%
Methods (classpath)	117927	120829	2.46%
Fields (classpath)	32687	33864	3.60%
Build Perf stats	---	---	---
Total Build Time (s)	129.8977	176.3090	35.73%
GC Time (s)	13.1490	15.2210	15.76%
CPU Load	3.5756	3.4591	-3.26%
Peak RSS (GB)	2.3618	2.2277	-5.68%
Available RAM (GB)	15.6065	15.9996	2.52%

The results indicate that the update brings new classes into the classpath most of which also appear to be reachable.

[MikeEdgar]

My guess is that the annotation scanning classes may be pulled in because of the new builder API, even though they do not execute at runtime. Would using a recorder possibly help there?

quarkus/extensions/smallrye-openapi/runtime/src/main/java/io/quarkus/smallrye/openapi/runtime/OpenApiDocumentService.java

Lines 84 to 90 in 3e0f8d6

	SmallRyeOpenAPI.Builder builder = SmallRyeOpenAPI.builder()
	.withConfig(config)
	.enableModelReader(false)
	.enableStandardStaticFiles(false)
	.enableAnnotationScan(false)
	.enableStandardFilter(false)
	.withCustomStaticFile(() -> source);

[MikeEdgar]

I'm doing some testing with an additional recorder step, but it sometimes throws a MethodTooLargeException. Is it the case that a recorder method can only do a limited amount of "work"?

[zakkak]

Yes, since the recorder generates bytecode in a synthetic method it is constraint by the 64k limit imposed by the JVM specification. Does such a big recorded method make sense in this case (did you try looking at the decompiled code)?

[MikeEdgar]

Does such a big recorded method make sense in this case (did you try looking at the decompiled code)?

Yes, it makes sense because it's doing a significant amount of processing. Maybe it can somehow be split up to smaller chunks.

[MikeEdgar]

Are there any options to explicitly exclude classes that are known to not be used at runtime? I suspect that many of the new classes are included from the SmallRyeOpenAPI.Builder class that appear to be reachable, but in practice are not due to how the builder is configured.

Otherwise, I suspect that the SmallRyeOpenAPI.Builder#build method may need to be broken into smaller pieces upstream in smallrye, and overridden in the Quarkus extension to avoid the appearance that certain things are reachable. Would that even make sense? I don't have a great mental model for how "reach-ability" is determined.

[zakkak]

Are there any options to explicitly exclude classes that are known to not be used at runtime?

You can potentially use the @Delete annotation, but you might get build failures if the analysis ends up reaching the deleted class.

Unfortunately there is still no way to explicitly exclude classes (even if they seem reachable) please see oracle/graal#3225

Otherwise, I suspect that the SmallRyeOpenAPI.Builder#build method may need to be broken into smaller pieces upstream in smallrye, and overridden in the Quarkus extension to avoid the appearance that certain things are reachable. Would that even make sense?

Yes, that would be a way to go as well.

I don't have a great mental model for how "reach-ability" is determined.

Judging by your comment I would say your understanding is pretty good already :)

You can think of it as follows:
If there is anyway to reach some code by following the call-graph, generated starting from main and some other root methods called vm-entry points, then that code is dimmed reachable.

[MikeEdgar]

With changes to the smallrye builder to split the build function so that the Quarkus extensions can override it, I have some promising results locally:
main:

[2/8] Performing analysis...  [******]                                                                  (65.2s @ 1.16GB)
   13,622 reachable types   (88.0% of   15,488 total)
   19,153 reachable fields  (57.6% of   33,261 total)
   68,441 reachable methods (58.1% of  117,852 total)
    4,000 types,   121 fields, and 3,261 methods registered for reflection
       62 types,    67 fields, and    55 methods registered for JNI access
        4 native libraries: dl, pthread, rt, z
[3/8] Building universe...                                                                              (10.9s @ 1.12GB)
[4/8] Parsing methods...      [***]                                                                      (6.7s @ 1.28GB)
[5/8] Inlining methods...     [***]                                                                      (4.6s @ 1.41GB)
[6/8] Compiling methods...    [********]                                                                (72.5s @ 1.04GB)
[7/8] Layouting methods...    [***]                                                                      (6.3s @ 1.42GB)
[8/8] Creating image...       [***]                                                                      (8.1s @ 1.64GB)
  27.35MB (47.04%) for code area:    43,781 compilation units
  30.39MB (52.27%) for image heap:  350,352 objects and 60 resources
 407.81kB ( 0.69%) for other data
  58.13MB in total

With updates:

[2/8] Performing analysis...  [******]                                                                  (58.1s @ 1.09GB)
   13,143 reachable types   (87.5% of   15,012 total)
   18,747 reachable fields  (57.7% of   32,498 total)
   66,808 reachable methods (57.8% of  115,529 total)
    3,969 types,   121 fields, and 3,256 methods registered for reflection
       62 types,    67 fields, and    55 methods registered for JNI access
        4 native libraries: dl, pthread, rt, z
[3/8] Building universe...                                                                              (11.0s @ 1.11GB)
[4/8] Parsing methods...      [***]                                                                      (6.7s @ 1.23GB)
[5/8] Inlining methods...     [***]                                                                      (4.4s @ 1.34GB)
[6/8] Compiling methods...    [********]                                                                (60.7s @ 1.02GB)
[7/8] Layouting methods...    [***]                                                                      (5.7s @ 1.38GB)
[8/8] Creating image...       [***]                                                                      (7.3s @ 1.55GB)
  26.84MB (47.57%) for code area:    42,770 compilation units
  29.18MB (51.73%) for image heap:  331,026 objects and 60 resources
 405.38kB ( 0.70%) for other data
  56.41MB in total

[zakkak]

That is pretty promising @MikeEdgar , great work!