Upgrading pandas to PySpark using Koalas

pandas is the defacto standard for data processing in standard Python, the same as Spark has become the defacto standard for distributed data processing. The pandas API is Python-related and leverages a coding style that makes use of Python's unique features to write code that is readable and beautiful. However, Spark is based on the JVM, and even the PySpark draws heavily on the Java language, including in naming conventions and function names. Thus, it is not very easy or intuitive for a pandas user to switch to PySpark, and a considerable learning curve is involved. Moreover, PySpark executes code in a distributed manner and the user needs to understand the nuances of how distributed code works when intermixing PySpark code with standard single-node Python code. This is a deterrent to an average pandas user to pick up and use PySpark. To overcome this issue, the Apache Spark developer community came up with another open source library...