Pandas 대 Polars: 구문, 속도 및 메모리의 완전한 비교

Pandas vs. Polars: A Complete Comparison of Syntax, Speed, and Memory Need help choosing the right Python dataframe library? This article compares Pandas and Polars to help you decide. Image by Author # Introduction If you've been working with data in Python, you've almost certainly used pandas. It's been the go-to library for data manipulation for over a decade. But recently, Polars has been gaining serious traction. Polars promises to be faster, more memory-efficient, and more intuitive than pandas. But is it worth learning? And how different is it really? In this article, we'll compare pandas and Polars side-by-side. You'll see performance benchmarks, and learn the syntax differences. By the end, you'll be able to make an informed decision for your next data project. # Getting Started Let's get both libraries installed first: pip install pandas polars Note: This article uses pandas 2.2.2 and Polars 1.31.0. For this comparison, we'll also use a dataset that's large enough to see real performance differences. We'll use Faker to generate test data: pip install Faker Now we're ready to start coding. # Measuring Speed By Reading Large CSV Files Let's start with one of the most common operations: reading a CSV file. We'll create a dataset with 1 million rows to see real performance differences. First, let's generate our sample data: import pandas as pd from faker import Faker import random # Generate a large CSV file for testing fake = Faker() Faker.seed(42) random.seed(42) data = { 'user_id': range(1000000), 'name': [fake.name() for _ in range(1000000)], 'email': [fake.email() for _ in range(1000000)], 'age': [random.randint(18, 80) for _ in range(1000000)], 'salary': [random.randint(30000, 150000) for _ in range(1000000)], 'department': [random.choice(['Engineering', 'Sales', 'Marketing', 'HR', 'Finance']) for _ in range(1000000)] } df_temp = pd.DataFrame(data) df_temp.to_csv('large_dataset.csv', index=False) print("✓ Generated large_dataset.csv with 1M rows") This code creates a CSV file with realistic data. Now let's compare reading speeds: import pandas as pd import polars as pl import time # pandas: Read CSV start = time.time() df_pandas = pd.read_csv('large_dataset.csv') pandas_time = time.time() - start # Polars: Read CSV start = time.time() df_polars = pl.read_csv('large_dataset.csv') polars_time = time.time() - start print(f"Pandas read time: {pandas_time:.2f} seconds") print(f"Polars read time: {polars_time:.2f} seconds") print(f"Polars is {pandas_time/polars_time:.1f}x faster") Output when reading the sample CSV: Pandas read time: 1.92 seconds Polars read time: 0.23 seconds Polars is 8.2x faster Here's what's happening: We time how long it takes each library to read the same CSV file. While pandas uses its traditional single-threaded CSV reader, Polars automatically parallelizes the reading across multiple CPU cores. We calculate the speedup factor. On most machines, you'll see Polars is 2-5x faster at reading CSVs. This difference becomes even more significant with larger files. # Measuring Memory Usage During Operations Speed isn't the only consideration. Let's see how much memory each library uses. We'll perform a series of operations and measure memory consumption. Please pip install psutil if you don't already have it in your working environment: import pandas as pd import polars as pl import psutil import os import gc # Import garbage collector for better memory release attempts def get_memory_usage(): """Get current process memory usage in MB""" process = psutil.Process(os.getpid()) return process.memory_info().rss / 1024 / 1024 # — - Test with Pandas — - gc.collect() initial_memory_pandas = get_memory_usage() df_pandas = pd.read_csv('large_dataset.csv') filtered_pandas = df_pandas[df_pandas['age'] > 30] grouped_pandas = filtered_pandas.groupby('department')['salary'].mean() pandas_memory = get_memory_usage() - initial_memory_pandas print(f"Pandas memory delta: {pandas_memory:.1f} MB") del df_pandas, filtered_pandas, grouped_pandas gc.collect() # — - Test with Polars (eager mode) — - gc.collect() initial_memory_polars = get_memory_usage() df_polars = pl.read_csv('large_dataset.csv') filtered_polars = df_polars.filter(pl.col('age') > 30) grouped_polars = filtered_polars.group_by('department').agg(pl.col('salary').mean()) polars_memory = get_memory_usage() - initial_memory_polars print(f"Polars memory delta: {polars_memory:.1f} MB") del df_polars, filtered_polars, grouped_polars gc.collect() # — - Summary — - if pandas_memory > 0 and polars_memory > 0: print(f"Memory savings (Polars vs Pandas): {(1 - polars_memory/pandas_memory) * 100:.1f}%") elif pandas_memory == 0 and polars_memory > 0: print(f"Polars used {polars_memory:.1f} MB while Pandas used 0 MB.") elif polars_memory == 0 and pandas_memory > 0: print(f"Polars used 0 MB while Pandas used {pandas_memory:.1f} MB.") else: print("Cannot compute memory savings due to zero or negative memory usage delta in both frameworks.") This code measure