Skip to content

Commit

Permalink
[SPARK-17339][CORE][BRANCH-2.0] Do not use path to get a filesystem i…
Browse files Browse the repository at this point in the history
…n hadoopFile and newHadoopFile APIs

## What changes were proposed in this pull request?

This PR backports apache#14960

## How was this patch tested?

AppVeyor - https://ci.appveyor.com/project/HyukjinKwon/spark/build/86-backport-SPARK-17339-r

Author: hyukjinkwon <[email protected]>

Closes apache#15008 from HyukjinKwon/backport-SPARK-17339.
  • Loading branch information
HyukjinKwon authored and shivaram committed Sep 8, 2016
1 parent 067752c commit 28377da
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions core/src/main/scala/org/apache/spark/SparkContext.scala
Original file line number Diff line number Diff line change
Expand Up @@ -988,7 +988,7 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli

// This is a hack to enforce loading hdfs-site.xml.
// See SPARK-11227 for details.
FileSystem.get(new URI(path), hadoopConfiguration)
FileSystem.getLocal(hadoopConfiguration)

// A Hadoop configuration can be about 10 KB, which is pretty big, so broadcast it.
val confBroadcast = broadcast(new SerializableConfiguration(hadoopConfiguration))
Expand Down Expand Up @@ -1077,7 +1077,7 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli

// This is a hack to enforce loading hdfs-site.xml.
// See SPARK-11227 for details.
FileSystem.get(new URI(path), hadoopConfiguration)
FileSystem.getLocal(hadoopConfiguration)

// The call to NewHadoopJob automatically adds security credentials to conf,
// so we don't need to explicitly add them ourselves
Expand Down

0 comments on commit 28377da

Please sign in to comment.