HDFS-17655. Cannot run HDFS balancer with BlockPlacementPolicyWithNodeGroup #7141

eubnara · 2024-10-31T22:37:46Z

Description of PR

When I use BlockPlacementPolicyWithNodeGroup, I cannot run balancer.
It is a side effect of HDFS-15363.

How was this patch tested?

Manual test.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?

hadoop-yetus · 2024-11-01T04:51:58Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 34s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	43m 47s		trunk passed
+1 💚	compile	1m 16s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	1m 9s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	1m 1s		trunk passed
+1 💚	mvnsite	1m 17s		trunk passed
+1 💚	javadoc	1m 3s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	1m 40s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	3m 14s		trunk passed
+1 💚	shadedclient	41m 30s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 20s		the patch passed
+1 💚	compile	1m 21s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	1m 21s		the patch passed
+1 💚	compile	1m 16s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	1m 16s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 0s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 122 unchanged - 0 fixed = 123 total (was 122)
+1 💚	mvnsite	1m 23s		the patch passed
+1 💚	javadoc	1m 1s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	1m 42s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	3m 39s		the patch passed
+1 💚	shadedclient	44m 12s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	221m 13s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 43s		The patch does not generate ASF License warnings.
		372m 59s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/1/artifact/out/Dockerfile
GITHUB PR	#7141
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux b16d8c6d3140 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `cf7d519`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/1/testReport/
Max. process+thread count	4005 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Hexiaoqiao · 2024-12-08T10:24:13Z

Try to trigger CI manually, let's wait to what it will say.

hadoop-yetus · 2024-12-08T16:47:03Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	32m 14s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	43m 38s		trunk passed
+1 💚	compile	1m 11s		trunk passed
+1 💚	checkstyle	1m 10s		trunk passed
+1 💚	mvnsite	1m 19s		trunk passed
+1 💚	javadoc	1m 48s		trunk passed
+1 💚	spotbugs	3m 14s		trunk passed
+1 💚	shadedclient	34m 10s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 6s		the patch passed
+1 💚	compile	1m 2s		the patch passed
+1 💚	javac	1m 2s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 58s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 122 unchanged - 0 fixed = 123 total (was 122)
+1 💚	mvnsite	1m 7s		the patch passed
+1 💚	javadoc	1m 31s		the patch passed
+1 💚	spotbugs	3m 11s		the patch passed
+1 💚	shadedclient	34m 12s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	222m 24s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 46s		The patch does not generate ASF License warnings.
		382m 30s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDecommission

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/2/artifact/out/Dockerfile
GITHUB PR	#7141
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 95a19f8bcbbc 5.15.0-125-generic #135-Ubuntu SMP Fri Sep 27 13:53:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `cf7d519`
Default Java	Red Hat, Inc.-1.8.0_412-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/2/testReport/
Max. process+thread count	3820 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/2/console
versions	git=2.9.5 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Hexiaoqiao · 2024-12-09T13:40:44Z

Hi @eubnara do you mind to check if the failed unit test relate to this PR. Thanks.

eubnara · 2024-12-11T00:14:23Z

I checked it but I think the error in TestBalancerRPCDelay, TestReadStripedFileWithDecodingDeletedData is not related to this PR.
I tested it on my laptop with the same code but the two tests are passed.

hadoop-yetus · 2024-12-12T17:52:49Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 43s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	38m 2s		trunk passed
+1 💚	compile	1m 6s		trunk passed
+1 💚	checkstyle	1m 6s		trunk passed
+1 💚	mvnsite	1m 12s		trunk passed
+1 💚	javadoc	1m 42s		trunk passed
+1 💚	spotbugs	2m 59s		trunk passed
+1 💚	shadedclient	37m 58s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 7s		the patch passed
+1 💚	compile	1m 1s		the patch passed
+1 💚	javac	1m 1s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 56s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 122 unchanged - 0 fixed = 123 total (was 122)
+1 💚	mvnsite	1m 9s		the patch passed
+1 💚	javadoc	1m 33s		the patch passed
+1 💚	spotbugs	3m 13s		the patch passed
+1 💚	shadedclient	34m 1s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	222m 32s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 47s		The patch does not generate ASF License warnings.
		348m 52s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDecommission

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/3/artifact/out/Dockerfile
GITHUB PR	#7141
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux e77895c5c536 5.15.0-125-generic #135-Ubuntu SMP Fri Sep 27 13:53:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `cf7d519`
Default Java	Red Hat, Inc.-1.8.0_412-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/3/testReport/
Max. process+thread count	4060 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/3/console
versions	git=2.9.5 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

haiyang1987 · 2024-12-15T14:44:07Z

Thanks @eubnara work, please fix checkstyle.

…eGroup

hadoop-yetus · 2024-12-20T04:33:13Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	11m 47s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 16s		trunk passed
+1 💚	compile	1m 19s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	1m 10s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	1m 10s		trunk passed
+1 💚	mvnsite	1m 19s		trunk passed
+1 💚	javadoc	1m 9s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 39s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	3m 8s		trunk passed
+1 💚	shadedclient	34m 47s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 8s		the patch passed
+1 💚	compile	1m 10s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	1m 10s		the patch passed
+1 💚	compile	1m 2s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	1m 2s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 59s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 122 unchanged - 0 fixed = 123 total (was 122)
+1 💚	mvnsite	1m 9s		the patch passed
+1 💚	javadoc	0m 57s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 30s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	3m 2s		the patch passed
+1 💚	shadedclient	34m 30s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	221m 36s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 46s		The patch does not generate ASF License warnings.
		358m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/4/artifact/out/Dockerfile
GITHUB PR	#7141
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 7fe9be121523 5.15.0-125-generic #135-Ubuntu SMP Fri Sep 27 13:53:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `155fff0`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/4/testReport/
Max. process+thread count	3628 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7141/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Hexiaoqiao

LGMT. +1 from my side. Will commit if no more other comment while wait two work days.

Hexiaoqiao · 2024-12-26T03:58:01Z

Committed to trunk. Thanks @eubnara for your works.

github-actions bot added HDFS trunk labels Oct 31, 2024

HDFS-17655. Cannot run HDFS balancer with BlockPlacementPolicyWithNod…

155fff0

…eGroup

eubnara force-pushed the HDFS-17655 branch from cf7d519 to 155fff0 Compare December 19, 2024 22:33

Hexiaoqiao approved these changes Dec 21, 2024

View reviewed changes

Hexiaoqiao merged commit d5b836b into apache:trunk Dec 26, 2024
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-17655. Cannot run HDFS balancer with BlockPlacementPolicyWithNodeGroup #7141

HDFS-17655. Cannot run HDFS balancer with BlockPlacementPolicyWithNodeGroup #7141

eubnara commented Oct 31, 2024

hadoop-yetus commented Nov 1, 2024

Hexiaoqiao commented Dec 8, 2024

hadoop-yetus commented Dec 8, 2024

Hexiaoqiao commented Dec 9, 2024

eubnara commented Dec 11, 2024

hadoop-yetus commented Dec 12, 2024

haiyang1987 commented Dec 15, 2024

hadoop-yetus commented Dec 20, 2024

Hexiaoqiao left a comment •

edited

Loading

Hexiaoqiao commented Dec 26, 2024

HDFS-17655. Cannot run HDFS balancer with BlockPlacementPolicyWithNodeGroup #7141

HDFS-17655. Cannot run HDFS balancer with BlockPlacementPolicyWithNodeGroup #7141

Conversation

eubnara commented Oct 31, 2024

Description of PR

How was this patch tested?

For code changes:

hadoop-yetus commented Nov 1, 2024

Hexiaoqiao commented Dec 8, 2024

hadoop-yetus commented Dec 8, 2024

Hexiaoqiao commented Dec 9, 2024

eubnara commented Dec 11, 2024

hadoop-yetus commented Dec 12, 2024

haiyang1987 commented Dec 15, 2024

hadoop-yetus commented Dec 20, 2024

Hexiaoqiao left a comment • edited Loading

Choose a reason for hiding this comment

Hexiaoqiao commented Dec 26, 2024

Hexiaoqiao left a comment •

edited

Loading