[Feature Request] merge metadata on mergeSchema=true

[zhukovgreen]

Feature request

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Overview

Motivation

This is follow up of the #2674 . Current behavior is counterintuitive. Metadata is a part of schema and it is logical to expect that it will be merged into the current table schema

Further details

The problem is here

delta/spark/src/main/scala/org/apache/spark/sql/delta/schema/SchemaMergingUtils.scala

Line 182 in 1609c38

currentField.metadata)

. If this line will be replaced with something like

current.Field.metadata ++ updateField.metadata

, then metadata will be merged.

Willingness to contribute

The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?

• Yes. I can contribute this feature independently.
• Yes. I would be willing to contribute this feature with guidance from the Delta Lake community.
• No. I cannot contribute this feature at this time.