[BUG][Kernel] delta-kernel-api pulls Jackson libraries

[YotillaAntoni]

Bug

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Describe the problem

Since this PR #3587 the delta-kernel-api includes a shaded version of Jackson to avoid dependency problems with connector developers.

But the published maven artifacts include jackson dependencies, so any project that uses delta-kernel-api might have conflicts

See for example:
https://oss.sonatype.org/content/repositories/iodelta-1181/io/delta/delta-kernel-api/3.3.0/delta-kernel-api-3.3.0.pom

<dependency>
  <groupId>com.fasterxml.jackson.core</groupId>
  <artifactId>jackson-databind</artifactId>
  <version>2.13.5</version>
</dependency>
<dependency>
  <groupId>com.fasterxml.jackson.core</groupId>
  <artifactId>jackson-core</artifactId>
  <version>2.13.5</version>
</dependency>
<dependency>
  <groupId>com.fasterxml.jackson.core</groupId>
  <artifactId>jackson-annotations</artifactId>
  <version>2.13.5</version>
</dependency>
<dependency>
  <groupId>com.fasterxml.jackson.datatype</groupId>
  <artifactId>jackson-datatype-jdk8</artifactId>
  <version>2.13.5</version>
</dependency>

Steps to reproduce

Add delta-kernel-api 3.3.0 dependency in a maven project

Observed results

Jackson dependencies are included

Expected results

Jackson dependencies are not included as they are shaded inside the delta-kernel-api jar

Further details

As a workaround one can explicitly exclude the jackson libraries when declaring the dependency, ie:

        <dependency>
            <groupId>io.delta</groupId>
            <artifactId>delta-kernel-api</artifactId>
            <version>${delta-kernel.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>com.fasterxml.jackson.core</groupId>
                    <artifactId>jackson-databind</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>com.fasterxml.jackson.core</groupId>
                    <artifactId>jackson-core</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>com.fasterxml.jackson.core</groupId>
                    <artifactId>jackson-annotations</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>com.fasterxml.jackson.datatype</groupId>
                    <artifactId>jackson-datatype-jdk8</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

Environment information

• Delta Lake version: 3.3.0 rc2
• Spark version:
• Scala version:

Willingness to contribute

The Delta Lake Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Delta Lake code base?

• Yes. I can contribute a fix for this bug independently.
• Yes. I would be willing to contribute a fix for this bug with guidance from the Delta Lake community.
• No. I cannot contribute a bug fix at this time.