TransposeDataFrame

Example of Spark Method to Transpose input DataFrame

What is Transpose?

The transpose of a Dataframe is a new DataFrame whose rows are the columns of the original DataFrame. (This makes the columns of the new DataFrame the rows of the original).

Here is the DataFrame Example:

Suppose we have Input DataFrame df as below:

  +--------+-----+------+-----+-------+
  |Products|Small|Medium|Large|ExLarge|
  +--------+-----+------+-----+-------+
  |Shirts  |10   |13    |34   |10     |
  |Trousers|11   |2     |30   |20     |
  |Pants   |70   |43    |24   |60     |
  |Sweater |101  |44    |54   |80     |
  +--------+-----+------+-----+-------+

Then below DataFrame will be Transpose DataFrame of df

  +--------+-----+------+-------+--------+
  |Products|Pants|Shirts|Sweater|Trousers|
  +--------+-----+------+-------+--------+
  |Medium  |43   |13    |44     |2       |
  |Small   |70   |10    |101    |11      |
  |ExLarge |60   |10    |80     |20      |
  |Large   |24   |34    |54     |30      |
  +--------+-----+------+-------+--------+

TransposeDF

TransposeDF is the Method written to convert input DataFrame into Transposed DataFrame. It take three below parameters and return new Transposed DataFrame:

TransposeDF(df: DataFrame, columns: Seq[String], pivotCol: String)

• First parameter is input DataFrame (eg: df in above example.)
• Second Parameter is Sequence of columns of Input DataFrame that need to transpose into rows. (eg: Seq("Small", "Medium", "Large", "ExLarge") in above example)
• Third Parameter is pivot column (column which rows required to transpose into columns). (eg. "Products" in above example)

How to use TransposeDF in Spark Scala

It is very easy to use. You just need to copy Scala TransposeDF method from here to your code and call it as below:

TransposeDF(df, Seq("Small", "Medium", "Large", "ExLarge"), "Products")

OR

val ColumnSeq:Seq[String] =  Seq("Small", "Medium", "Large", "ExLarge")   
val transDF = TransposeDF(df,ColumnSeq, "Products")

How to use TransposeDF in PySpark

Same as Scala, copy Python TransposeDF from here to your code and call it as below:

TransposeDF(df, ["Small", "Medium", "Large", "ExLarge"], "Products")

OR

ColumnList =  ["Small", "Medium", "Large", "ExLarge"]   
transDF = TransposeDF(df,ColumnList, "Products")