AnalysisException: Attribute name contains invalid character(s) issue

[rambabu-posa]

Exception in thread "main" org.apache.spark.sql.AnalysisException: Attribute name "Code région" contains invalid character(s) among " ,;{}()\n\t=". Please use alias to rename it.;

[rambabu-posa]

Not only this column name (with special characters), normal column like "Transaction description" is also failing with same error.

That means if a Spark DataFrame reads a CSV file with column names( having spaces), its throwing same AnalysisException exception.

Is there any way to fix this issue with default character adding to that space like
"Transaction description"
to
"Transaction_description"

Or any other solution to solve this scenario please?

[brkyvz]

You can use the withColumnRenamed function of DataFrames to rename your columns, or you can alias columns in your select statement with alias, e.g.

select(col("Transacation description").alias("Transacation_description"))

[rambabu-posa]

Yes we can do it programatically. Do we have any option to fix it without doing so?
I think, we don't need to rename column names in Spark, it works very well without doing so.

[brkyvz]

Parquet doesn't allow storing such column names. I'd say it's better engineering practice for you to follow some convention yourself and fix the names rather than having some system arbitrarily fix it for you.

[rambabu-posa]

Im able to do the same operation using DataFrame API like
df.write.format("parquet").save("data/par_data")

I would say its a feature in "Delta Lake" that it is enforcing to follow that good practice.

[tdas]

That has indeed been one of our core design principle - an opinionated view of how to manage data without shooting yourself in the foot.

[zsxwing]

Closing this. Invalid characters are disallowed by design.

[surya1527]

Parquet doesn't allow storing such column names. I'd say it's better engineering practice for you to follow some convention yourself and fix the names rather than having some system arbitrarily fix it for you.

for c in df.columns:
df = df.withColumnRenamed(c, c.replace( ";" , ""))

That worked fine in case