HADOOP-19207: [ABFS][FNSOverBlob] Response Handling of Blob Endpoint APIs and Metadata APIs #7210

anujmodi2021 · 2024-12-05T07:47:38Z

Description of PR

Target Jira: https://issues.apache.org/jira/browse/HADOOP-19207

As part of the new Fns Over Blob Support, there is a need to handle responses from 2 service endpoints of Azure Storage differently. Blob Endpoint API response format is different from DFS Endpoint APIs. DFS Endpoint primarily has response body in Json format where as BLOB Endpoint has XML.

In this PR following changes are included:

Parsing List Status repsonse abstracted into DFS and Blob clients with additional schema files to support each of the endpoints due to difference in response body format
Parsing of request headers abstracted into DFS and Blob clients due to difference in response header formats.
Parsing on Storage Error repsonse abstracted into DFS and Blob clients due to difference in response body format.
Handling getPathStatus(), listStatus(), getXAttr(), setXAttr() calls on implicit paths on Blob Endpoint.
Handling Following behavioral differences from server in DFS and Blob Endpoint:
- List Status on Non-existing path returns 404 on DFS and empty listing on Blob
- List Status on File Path returns file in lisiting output on DFS and empty listing on Blob

How was this patch tested?

Existing test suite was ran and results added below to make sure existing DFS Endpoint support did not get disturbed.
New tests for FNS Blob need ingress on Blob endpoint to be supported which will be done in subsequest PRs.
Test Result: #7210 (comment)

Following JIRA tracks the test addition for new support: https://issues.apache.org/jira/browse/HADOOP-19234
Following JIRA tracks the ingress support for FNS Blob: https://issues.apache.org/jira/browse/HADOOP-19232

For code changes:

...rc/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/BlobListResultEntrySchema.java

...ure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/BlobListResultSchema.java

...ols/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java

anujmodi2021 · 2024-12-05T12:42:00Z

:::: AGGREGATED TEST RESULT ::::

============================================================
HNS-OAuth

[WARNING] Tests run: 157, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 653, Failures: 0, Errors: 0, Skipped: 85
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 25
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 23

============================================================
HNS-SharedKey

[WARNING] Tests run: 157, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 653, Failures: 0, Errors: 0, Skipped: 37
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 25
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 10

============================================================
NonHNS-SharedKey

[WARNING] Tests run: 157, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 637, Failures: 0, Errors: 0, Skipped: 277
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 27
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 11

============================================================
AppendBlob-HNS-OAuth

[WARNING] Tests run: 157, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 653, Failures: 0, Errors: 0, Skipped: 87
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 49
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 23

Time taken: 107 mins 7 secs.

snvijaya

+1 LGTM

hadoop-yetus · 2024-12-17T08:01:58Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 58s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 7 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	40m 7s		trunk passed
+1 💚	compile	0m 37s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 34s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 33s		trunk passed
+1 💚	mvnsite	0m 39s		trunk passed
+1 💚	javadoc	0m 43s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 34s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	1m 10s		trunk passed
+1 💚	shadedclient	34m 18s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 28s		the patch passed
+1 💚	compile	0m 31s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 31s		the patch passed
+1 💚	compile	0m 27s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 20s	/results-checkstyle-hadoop-tools_hadoop-azure.txt	hadoop-tools/hadoop-azure: The patch generated 1 new + 13 unchanged - 4 fixed = 14 total (was 17)
+1 💚	mvnsite	0m 30s		the patch passed
+1 💚	javadoc	0m 27s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 24s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	1m 3s		the patch passed
+1 💚	shadedclient	34m 2s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 20s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		122m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7210/5/artifact/out/Dockerfile
GITHUB PR	#7210
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 1c776a46fae2 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `cca8f1c`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7210/5/testReport/
Max. process+thread count	554 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7210/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

rakeshadr · 2024-12-26T04:43:07Z

+1 LGTM. Thanks @anujmodi2021 for the contribution.

anujmodi2021 added 3 commits November 28, 2024 18:24

Saving WIP

01091d6

Merge branch 'trunk' into HADOOP-19207_responseHandling

5a884a8

Response Parsing On Blob Endpoint

de2086e

github-actions bot added trunk TOOLS ABFS labels Dec 5, 2024

Fixing Tests in HNS-DFS

f050398

anmolanmol1234 reviewed Dec 5, 2024

View reviewed changes

...rc/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/BlobListResultEntrySchema.java Outdated Show resolved Hide resolved

anmolanmol1234 reviewed Dec 5, 2024

View reviewed changes

...ure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/BlobListResultSchema.java Outdated Show resolved Hide resolved

anmolanmol1234 reviewed Dec 5, 2024

View reviewed changes

...ols/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java Show resolved Hide resolved

anujmodi2021 added 2 commits December 5, 2024 03:46

EOF Fixes

b95b441

Javadocs

d881070

anujmodi2021 marked this pull request as ready for review December 5, 2024 12:41

This comment was marked as outdated.

Sign in to view

anujmodi2021 added 2 commits December 5, 2024 06:59

Yetus Checks

1935fe5

Build Check

f4d9b53

This comment was marked as outdated.

Sign in to view

snvijaya approved these changes Dec 11, 2024

View reviewed changes

Adding ASF License

cca8f1c

rakeshadr merged commit 7e67358 into apache:trunk Dec 26, 2024
4 checks passed

anujmodi2021 deleted the HADOOP-19207_responseHandling branch December 26, 2024 05:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-19207: [ABFS][FNSOverBlob] Response Handling of Blob Endpoint APIs and Metadata APIs #7210

HADOOP-19207: [ABFS][FNSOverBlob] Response Handling of Blob Endpoint APIs and Metadata APIs #7210

anujmodi2021 commented Dec 5, 2024 •

edited

Loading

anujmodi2021 commented Dec 5, 2024

This comment was marked as outdated.

This comment was marked as outdated.

snvijaya left a comment

hadoop-yetus commented Dec 17, 2024

rakeshadr commented Dec 26, 2024

HADOOP-19207: [ABFS][FNSOverBlob] Response Handling of Blob Endpoint APIs and Metadata APIs #7210

HADOOP-19207: [ABFS][FNSOverBlob] Response Handling of Blob Endpoint APIs and Metadata APIs #7210

Conversation

anujmodi2021 commented Dec 5, 2024 • edited Loading

Description of PR

How was this patch tested?

For code changes:

anujmodi2021 commented Dec 5, 2024

============================================================ HNS-OAuth

============================================================ HNS-SharedKey

============================================================ NonHNS-SharedKey

============================================================ AppendBlob-HNS-OAuth

This comment was marked as outdated.

This comment was marked as outdated.

snvijaya left a comment

Choose a reason for hiding this comment

hadoop-yetus commented Dec 17, 2024

rakeshadr commented Dec 26, 2024

anujmodi2021 commented Dec 5, 2024 •

edited

Loading

============================================================
HNS-OAuth

============================================================
HNS-SharedKey

============================================================
NonHNS-SharedKey

============================================================
AppendBlob-HNS-OAuth