Transforming the data

Our Silver layer code will process our conversion events data, flattening the information there, and joining it to our dimensional tables. We’re focused on restructuring the data so it can be used in our Gold layer while cleaning and standardizing the naming conventions in our data platform.

We’ll start by creating a function to help read our data into a DataFrame.

def read(formatType: String, sourceLocation: String): DataFrame = {
    formatType match {
      case "delta" => spark.read.format(formatType).load(s"${sourceLocation}")
      case "csv" =>
        spark.read
          .format(formatType)
          .options(Map("delimiter" -> ",", "header" ->...