Advanced techniques to efficiently process and load data
In this story, I want to talk about what I love about Pandas and what I often use in the ETL applications I create to process data. Exploratory data analysis, data cleansing, and data frame transformation will be covered. Here are some of my favorite techniques for using this library to optimize memory usage and efficiently process large amounts of data. When working with relatively small datasets in Pandas, this is rarely a problem. Easily manipulates data in data frames and provides a very useful set of commands to work with it. When it comes to data transformation on much larger data frames (1 GB or more), we typically use Spark and distributed computing clusters. It can process terabytes or petabytes of data, but it will probably cost a lot of money to run all that hardware. Therefore, Pandas may be a better choice if you need to work with medium-sized datasets in environments with limited memory resources.
Pandas and Python generators
In one of my previous articles, I wrote about how to use Python’s generators to efficiently process data. [1].
This is a simple trick to optimize memory usage. Imagine you have a huge dataset somewhere in external storage. It can be a database or just a large CSV file. Imagine you need to process this 2-3 TB file and apply some transformation to each row of data in this file. Assume that you have a service that performs this task, and that service only has 32 GB of memory. This limits data loading and prevents you from applying simple Python to load the entire file into memory and split it line by line. split(‘\n’) operator. The solution is to process line by line, yield Frees memory for the next memory each time. This helps create a continuous streaming flow of her ETL data to the final destination in the data pipeline. It can be anything: a cloud storage bucket, another database, a data warehousing solution (DWH), a streaming topic, etc.
