Pandas, PySpark, Koalas - Which library should you choose for your Data Science project?

This article compares three Python libraries for data analysis. Pandas is suitable for smaller volumes, PySpark handles large quantities through parallelization, and Koalas makes the transition from Pandas to Spark easier.

Introduction

Data science projects require specific tools depending on the project phase. This article provides an overview of 3 Python libraries often used in data analysis and compares their usage. Pandas, the foundation...

If you have ever done data analysis in Python, you couldn't have missed the Pandas library!
Indeed, this library is one of the most widely used for data manipulation. It has become a must-have due to its ease of use, practicality, and the full range of features available for data manipulation.

The core object of this library is the DataFrame (tables, see below) make it a very intuitive tool, as this object is very easy to grasp when working with structured data. Pandas makes it very easy to load or write data from a CSV file, Excel, or a SQL database. Furthermore, Pandas contains a multitude of optimized functions for manipulating this tabular data. In fact, critical parts of the code are sometimes written in Python or C to increase execution speed.

‍

‍

These various advantages make it the go-to choice for data processing in Python when working with limited data volumes.

For better performance: PySpark

Using a framework like Spark generally comes after the proof of concept, or PoC, phase. As a reminder, a PoC is a very short-term project aimed at exploring a data science topic. The PoC serves to prove the feasibility and added value of the approach on a small scale (with a reasonable amount of data). In this context, data scientists have every interest in using their favorite tools to move quickly.

When, in a second phase, you want to industrialize this PoC— that is, apply it on a larger scale — you generally run into performance issues. Indeed, the data volume may explode when moving to the full scope compared to the initial sample, which can create significant latency in calculation execution or even cause some to fail entirely.

To remedy this, it will be necessary to use frameworks that allow for parallelizing calculations, hence the use of Spark with its Python interface, PySpark. One of the advantages of PySpark is also the distinction between "lazy operation" and actions. Spark will only execute the code and therefore launch calculations when necessary.

Here is an example of PySpark code compared to Pandas; note that the syntax differs significantly and that an adjustment is required to move from Pandas to PySpark:

Pandas

import pandas as pd
df = pd.DataFrame({‘col1’: [1, 2], ‘col2’: [3, 4], ‘col3’: [5, 6]})
df[‘col4’] = df.col1 * df.col1

PySpark

df = spark.read.option(“inferSchema”, “true”).option(“comment”, True).csv(“my_data.csv”)
df = df.toDF(‘col1’, ‘col2’, ‘col3’)
df = df.withColumn(‘col4’, df.col1*df.col1)

For a smooth transition: Koalas?

Koalas is a very recent library (late 2019) that aims to write Spark programs using Pandas syntax. It allows for unifying the production of experimental and industrial code under the same tool, while benefiting from the flexibility of Pandas and the distributed performance of Spark.

It is therefore an intermediate solution that is quite relevant and particularly well-suited for Data Scientists who are already proficient in Pandas and wish to handle larger data volumes while sticking with familiar tools, thus avoiding the need to fully learn a new language. Note, however, that not all Pandas features are available in the Koalas library yet.
‍

‍

For example, using the same code as before with Koalas:

import databricks.koalas as ks
df = ks.DataFrame({‘col1’: [1, 2], ‘col2’: [3, 4], ‘col3’: [5, 6]})
df[‘col4’] = df.col1 * df.col1

You can clearly see the similarity to writing code in Pandas. Here, only the library name has changed, but the code remains exactly the same as it would be with Pandas.

Should we abandon PySpark, then? Not really.

Subtleties still exist between the two environments, and PySpark remains the reference framework in Big Data for the Data Engineering community. Data Engineers are often already familiar with the Spark language and will have very little interest in switching to Koalas. Furthermore, Koalas is an enhancement of the Spark DataFrame API to bring it closer to Pandas, meaning the underlying language remains Spark. If, for example, there is a need for additional performance, it may be necessary to revert to Spark without the wrapper. Additionally, Spark is designed to be easily integrated with a wide range of other tools, given its popularity.

In the graph below (produced by Databricks), we can see that PySpark still has superior performance compared to Koalas, even though Koalas is already very high-performing compared to Pandas. It is also interesting to note that for small datasets, Pandas is more efficient (due to the initialization and data transfer operations of distributed solutions) and therefore more suitable.

‍

Koalas can also be used as an approach to learn Spark gradually, but in any case, it is necessary to practice within the Spark environment.
‍

It is obviously possible to switch from one language to another using certain functions like to_pandas(), but it is recommended to use them as little as possible because they are very costly operations, as they impact the data storage format.

And where does Aqsone fit into all this?

Our data scientists use (Py)Spark and Koalas to process calculations on large volumes of data. If you would like to learn more about this technology and implement it in your big data projects, contact us!