HADOOP-18301.Upgrade commons-io to 2.11.0

[ashutoshcipher]

Description of PR

Upgrade commons-io to 2.11.0

Upgrading to new release to keep up for new features and bug fixes.

JIRA: HADOOP-18301

How was this patch tested?

CI/Build Check

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 52s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	40m 7s		trunk passed
+1 💚	compile	0m 29s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	0m 29s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	mvnsite	0m 34s		trunk passed
+1 💚	javadoc	0m 37s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 32s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	shadedclient	65m 33s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 18s		the patch passed
+1 💚	compile	0m 17s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	0m 17s		the patch passed
+1 💚	compile	0m 17s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 17s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 20s		the patch passed
+1 💚	javadoc	0m 18s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 18s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
-1 ❌	shadedclient	9m 11s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 21s		hadoop-project in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		79m 21s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4455/1/artifact/out/Dockerfile
GITHUB PR	#4455
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname	Linux 9f72f7f73d4a 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / fd21b40
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4455/1/testReport/
Max. process+thread count	524 (vs. ulimit of 5500)
modules	C: hadoop-project U: hadoop-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4455/1/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[PrabhuJoseph]

Thanks @ashutoshcipher for the patch. Could you check why shadedclient fails with the patch.

[ayushtkn]

What is the reason for upgrade? some CVE or some feature that we want to use? We can't upgrade any thirdparty library just because we want to do that or there is a latest version for it...
These upgrades if not backward compatible makes the downstream projects adapting the hadoop release a Pain.
@PrabhuJoseph please check on these params before committing if you intend to do so..

[ashutoshcipher]

Thanks @ayushtkn for checking. There isn't any specific CVE related to this as far as I know. As the dependency is old and new version has general new features and bug fixes. Updating it seems to be a good idea to me.

Also the the same is updated in Hive as well - apache/hive#3370

[steveloughran]

Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?

if you haven't done this, can you uncheck the box. if you have, which stores, which endpoints?

[ashutoshcipher]

Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?

if you haven't done this, can you uncheck the box. if you have, which stores, which endpoints?

Thanks for pointing it out. Corrected it.

[aajisaka]

The error message in shadedclient: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4455/1/artifact/out/patch-shadedclient.txt

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.10.1:compile (default-compile) on project hadoop-hdfs: Compilation failure
[ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4455/ubuntu-focal/src/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HostRestrictingAuthorizationFilter.java:[128,9] exception java.io.IOException is never thrown in body of corresponding try statement

Since commons-io 2.11.0 throws IOException has been removed from FilenameUtils#directoryContains and that's why the compile failed: apache/commons-io@083c584

Given there is an incompatibility between 2.11.0 and 2.8.0, I recommend to upgrade to 2.11.0 only for trunk if there is no security issue in the current version.

[ayushtkn]

Given there is an incompatibility between 2.11.0 and 2.8.0, I recommend to upgrade to 2.11.0 only for trunk if there is no security issue in the current version.

@aajisaka If there is no security issue and if this is incompatible and if we don't have a present use case for this, then why should we even upgrade on trunk and have an incompatible change on our name?
Out of curiosity do you folks have some internal requirement/use case with the higher version?

[aajisaka]

Thank you @ayushtkn for your comment.

do you folks have some internal requirement/use case with the higher version?

Yes. My use case is to match the libraries versions with all the Hadoop-related applications to avoid conflicts as possible, and I think someone other than us has similar use cases. Given Apache Spark has already upgraded to 2.11.0 in Spark 3.3.0 https://issues.apache.org/jira/browse/SPARK-36737, so I want to follow.

[ashutoshcipher]

Thank you @ayushtkn for your comment.

do you folks have some internal requirement/use case with the higher version?

Yes. My use case is to match the libraries versions with all the Hadoop-related applications to avoid conflicts as possible, and I think someone other than us has similar use cases. Given Apache Spark has already upgraded to 2.11.0 in Spark 3.3.0 https://issues.apache.org/jira/browse/SPARK-36737, so I want to follow.

Thanks @aajisaka and @ayushtkn - Adding further to Akira's comment; The same has been done in Hive as well as I mentioned earlier https://issues.apache.org/jira/browse/HIVE-26321

[aajisaka]

+1 for trunk

[aajisaka]

Cancelling my +1.

@ashutoshcipher could you fix the error in #4455 (comment) ?

[ashutoshcipher]

@aajisaka - Sorry I missed addressing you comment earlier. In my latest commit, I have addressed you comment. Thanks.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 58s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 51s		Maven dependency ordering for branch
+1 💚	mvninstall	34m 8s		trunk passed
+1 💚	compile	27m 53s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	23m 52s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 24s		trunk passed
+1 💚	mvnsite	2m 45s		trunk passed
+1 💚	javadoc	2m 19s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 42s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+0 🆗	spotbugs	1m 4s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	29m 23s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 54s		the patch passed
+1 💚	compile	27m 33s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	27m 33s		the patch passed
+1 💚	compile	25m 7s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	25m 7s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	4m 57s		the patch passed
+1 💚	mvnsite	3m 22s		the patch passed
+1 💚	javadoc	2m 52s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	3m 22s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+0 🆗	spotbugs	1m 3s		hadoop-project has no data from spotbugs
+1 💚	shadedclient	30m 33s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 2s		hadoop-project in the patch passed.
+1 💚	unit	360m 50s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	2m 16s		The patch does not generate ASF License warnings.
		620m 39s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4455/2/artifact/out/Dockerfile
GITHUB PR	#4455
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 8e077ce3c8ed 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 7192b8f
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4455/2/testReport/
Max. process+thread count	2161 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4455/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[aajisaka]

+1 for trunk. Thanks @ashutoshcipher