DeltaSource load all files to Spark Driver during every batch. For big Table it's a huge issue.

[GrigorievNick]

This line of code uploads all files from delta-log to spark driver.

delta/src/main/scala/org/apache/spark/sql/delta/files/DeltaSourceSnapshot.scala

Line 86 in 819bcd5

Seq("add")).as[IndexedFile].collect().toIterable

1. This creates a very big overhead for spark driver memory and GC for big topics.
2. It's pretty strange to see collect inside a function that returns an iterator. Does any concern exist to do not use toLocalIterator?

[zsxwing]

Good catch. Yep, it makes sense to use toLocalIterator. Feel free to submit a PR to fix it.

[GrigorievNick]

thx @zsxwing, right now we use spark 2.4.7, and the master branch of the delta.io compatible only with spark 3.
So I have two questions. Please suggest me.

1. Can I do the fix as the patch to 0.6.1 tag and expect that it will be published with 0.6.2 delta.io version?
2. I will do a fix to spark 3 as well, but right now I don't have the environment to test this fix, so does it ok If I will do the fix without internal testing? Do the delta.io test handle streaming API or it's better to publish MR with enhancement after my team adopts spark 3?

[fvaleye]

Hello @GrigorievNick, I submitted a PR related to your issue (only with the latest version of delta-io, with Spark 3). I hope it helps!

[GrigorievNick]

Hi @fvaleye Thank you.
This does not affect my current project, because I use spark 2, and will proceed for the next few months.
So I just have my own fork of delta.io 0.6.1.
But this saves my time for contributing to things that I don't use tight now at work.
And also help to improve project quality in general.
So again, thank you.
It was very helpful.