-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[fix](hive docker)Reserve host port for hive2 namenode and datanode (#47262) #47354
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…pache#47262) Problem Summary: The [External hive CI](http://43.132.222.7:8111/buildConfiguration/Doris_External_Regression/612304?buildTab=log&linesState=3650&logView=flowAware) failed because of `namenode` error( 50070 port already in used), docker logs: ```txt 2025-01-21T04:22:37.955682469Z java.net.BindException: Port in use: 0.0.0.0:50070 2025-01-21T04:22:37.955686106Z at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:940) 2025-01-21T04:22:37.955689402Z at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:876) 2025-01-21T04:22:37.955692708Z at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142) 2025-01-21T04:22:37.955697828Z at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:760) 2025-01-21T04:22:37.955701444Z at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:639) 2025-01-21T04:22:37.955704831Z at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:819) 2025-01-21T04:22:37.955708237Z at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:803) 2025-01-21T04:22:37.955711674Z at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1500) 2025-01-21T04:22:37.955715090Z at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1566) 2025-01-21T04:22:37.955718446Z Caused by: java.net.BindException: Address already in use 2025-01-21T04:22:37.955722013Z at sun.nio.ch.Net.bind0(Native Method) 2025-01-21T04:22:37.955725460Z at sun.nio.ch.Net.bind(Net.java:433) 2025-01-21T04:22:37.955729227Z at sun.nio.ch.Net.bind(Net.java:425) 2025-01-21T04:22:37.955733074Z at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) 2025-01-21T04:22:37.955736600Z at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) 2025-01-21T04:22:37.955740197Z at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216) 2025-01-21T04:22:37.955743884Z at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:934) 2025-01-21T04:22:37.955747391Z ... 8 more 2025-01-21T04:22:37.961686454Z 25/01/21 04:22:37 INFO util.ExitUtil: Exiting with status 1 ``` The best choice is avoid the services using server port at range `/proc/sys/net/ipv4/ip_local_port_range` (32768-60999). But since the namenode [hardcode exposing port `50070` in docker image](https://hub.docker.com/layers/bde2020/hadoop-datanode/2.0.0-hadoop2.7.4-java8/images/sha256-5623fca5e36d890983cdc6cfd29744d1d65476528117975b3af6a80d99b3c62f), so we add the port to `net.ipv4.ip_local_reserved_ports` and introduce a new flags `--reserve-ports` to control it (default false, because not everyone want to modify system reserved ports). Change-Id: I03a81e9931cb555695199436b6f0517cccf83588
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
run buildall |
run p0 |
run external |
zgxme
approved these changes
Jan 24, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
PR approved by anyone and no changes requested. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem Summary:
The External hive
CI failed because of
namenode
error( 50070 port already in used), docker logs:The best choice is avoid the services using server port at range
/proc/sys/net/ipv4/ip_local_port_range
(32768-60999). But since the namenode hardcode exposing port50070
in docker image, so we add the port tonet.ipv4.ip_local_reserved_ports
and introduce a new flags--reserve-ports
to control it (default false, because not everyone want to modify system reserved ports).Change-Id: I03a81e9931cb555695199436b6f0517cccf83588
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)