[GLUTEN-8330][VL] Improve convert the viewfs path to hdfs path

[wangyum]

What changes were proposed in this pull request?

1. Avoid RPC calls.
2. Add a cache.

(Fixes: #8330)

How was this patch tested?

Manual tests, before: 225007 ms, after: 1330 ms:

import org.apache.hadoop.fs.viewfs.ViewFileSystemUtils
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{FileSystem, Path}

val fs = FileSystem.get(spark.sparkContext.hadoopConfiguration)
val paths = fs.listStatus(new Path("viewfs://my-cluster/path/to/table")).take(50000).map(_.getPath.toString)

val start1 = System.currentTimeMillis
paths.map(p => FileSystem.get(new Path(p).toUri, spark.sparkContext.hadoopConfiguration).resolvePath(new Path(p)))
println(s"Convert viewfs to HDFS using resolvePath took: ${System.currentTimeMillis - start1} ms")

val start2 = System.currentTimeMillis
paths.map(p => ViewFileSystemUtils.convertViewfsToHdfs(p, spark.sparkContext.hadoopConfiguration))
println(s"Convert viewfs to HDFS using convertViewfsToHdfs took: ${System.currentTimeMillis - start2} ms")

Output:

Convert viewfs to HDFS using resolvePath took: 225007 ms
Convert viewfs to HDFS using convertViewfsToHdfs took: 1330 ms

[github-actions]

#8330

[github-actions]

Run Gluten Clickhouse CI on x86

[zhouyuan]

Would this cache take too much memory?

[wangyum]

No, the size of this map is at most equal to the total number of partitions of these tables.

[zhouyuan]

👍

[JkSelf]

LGTM. Thanks for your work. Except two small comments.

[github-actions]

Run Gluten Clickhouse CI on x86

[JkSelf]

LGTM. Thanks.

[github-actions]

Run Gluten Clickhouse CI on x86

[JkSelf]

@wangyum Can you help to resolve the following compile issue? Thanks.

Error:  /__w/incubator-gluten/incubator-gluten/gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala:386: error: type mismatch;
Error:   found   : scala.collection.mutable.Buffer[String]
Error:   required: Seq[String]
Error:                    splitInfo.getPaths.asScala,
Error:                                       ^
Warning: [WARNING] 1 warning
Error: [ERROR] 1 error
Error:  exception compilation error occurred!!!

[github-actions]

Run Gluten Clickhouse CI on x86