File I/O and Data Import

Loading data from files and databases is where every data science project begins. This lesson covers reading and writing structured data formats, file handling best practices, and techniques for working with large files.

Data Import Pipeline

File Basics

CSV format: Each row is a record, columns separated by delimiter: .

JSON format: Nested key-value pairs: where values can be arrays or nested objects.

File size: .

Opening and Closing Files

File Modes

"r"   Read (default). File must exist.
"w"   Write. Creates new or truncates existing.
"a"   Append. Creates new or appends to existing.
"x"   Exclusive create. Fails if file exists.
"rb"  Read binary.
"wb"  Write binary.
"r+"  Read and write. File must exist.
"w+"  Write and read. Truncates existing.

Reading Files

Writing Files

Reading CSV Files

CSV is the most common data format. Python's csv module and pandas handle it well.

With the csv Module

With pandas

Reading JSON Files

JSON is the standard for API responses and nested data.

JSON with pandas

Reading Excel Files

Reading from Databases

SQLite (Built-in)

With SQLAlchemy (Works with Most Databases)

Chunked Reading for Large Files

When files are too large to fit in memory, process them in chunks.

CSV Chunks

JSON Lines Chunks

Working with Compressed Files

Encoding Issues

Practical Pipeline

Key Takeaways

• Always use context managers (with statements) for file operations.
• Pandas provides the most convenient API for structured data files.
• Use chunked reading for files larger than available memory.
• JSON Lines format is ideal for streaming and large datasets.
• Choose the right encoding (UTF-8 is the safe default).
• Parquet is more efficient than CSV for columnar data (covered later).