Python Data Types & Structures Deep Dive

Why This Matters for Data Science

Before building machine learning models or analyzing datasets, you must fundamentally understand how Python organizes and stores data. Every data structure choice has computational consequences—a naive list operation might take 1000x longer than the equivalent NumPy array operation. This tutorial provides a rigorous exploration of Python's type system and data structures with focus on their performance characteristics and practical applications.

1. Primitive Data Types

1.1 The Type Hierarchy

Python's type system forms a hierarchy where every value is an object:

object
├── numbers.Number
│   ├── int (arbitrary precision)
│   ├── float (64-bit IEEE 754)
│   └── complex
├── bool (subclass of int)
├── str (immutable sequence of Unicode codepoints)
├── NoneType
└── collections.abc
    ├── Sequence → list, tuple, range
    ├── Set → set, frozenset
    └── Mapping → dict

1.2 Integer (int)

Python integers have arbitrary precision—they grow as large as memory allows.

# Memory representation
import sys

small_int = 42
big_int = 2 ** 1000

print(f"Small int: {small_int}")
print(f"  Bytes: {sys.getsizeof(small_int)}")
print(f"  Type: {type(small_int)}")
print()
print(f"Big int: {big_int}")
print(f"  Bytes: {sys.getsizeof(big_int)}")
print(f"  Digits: {len(str(big_int))}")

Output:

Small int: 42
  Bytes: 28
  Type: <class 'int'>

Big int: 10715086071862673209484250490600018105614048117055336074437503883703510511249361224931983788156958581275946729175531468251871452856923140435984577574698574803934567774824230985421074605062371141877954182153046474983581941267398767559165543946077062914571196477686542167660429831652624386837205668069376
  Digits: 302

Why arbitrary precision matters: Financial calculations, cryptography, and scientific computing require exact integer arithmetic without overflow.

1.3 Float (float)

Python floats are 64-bit IEEE 754 double-precision numbers.

import struct

value = 3.14159

# IEEE 754 representation
bytes_repr = struct.pack('d', value)
bits = ''.join(format(b, '08b') for b in bytes_repr)

print(f"Value: {value}")
print(f"IEEE 754 bits: {bits}")
print(f"  Sign:     {bits[0]} ({'negative' if bits[0] == '1' else 'positive'})")
print(f"  Exponent: {bits[1:12]} = {int(bits[1:12], 2) - 1023}")
print(f"  Mantissa: {bits[12:]}")
print(f"\nFloat precision limit:")
print(f"  0.1 + 0.2 = {0.1 + 0.2}")
print(f"  0.1 + 0.2 == 0.3? {0.1 + 0.2 == 0.3}")
print(f"  Decimal comparison: {round(0.1 + 0.2, 10) == 0.3}")

Output:

Value: 3.14159
IEEE 754 bits: 0100000000001001001000011111101101010100010001000010110111111000
  Sign:     0 (positive)
  Exponent: 10000000000 = 1
  Mantissa: 1001001000011111101101010100010001000010110111111000

Float precision limit:
  0.1 + 0.2 = 0.30000000000000004
  0.1 + 0.2 == 0.3? False
  Decimal comparison: True

Critical insight: Floating-point arithmetic is inherently approximate. Always use math.isclose() for float comparisons:

import math

a = 0.1 + 0.2
b = 0.3

# WRONG
print(f"Direct comparison: {a == b}")

# RIGHT
print(f"Close comparison: {math.isclose(a, b)}")
print(f"With tolerance:    {math.isclose(a, b, rel_tol=1e-9)}")

1.4 Boolean (bool)

Booleans are subclasses of integers with two values: True (1) and False (0).

# Boolean arithmetic
print(f"True + True = {True + True}")
print(f"True * 10 = {True * 10}")
print(f"False * 100 = {False * 100}")
print(f"True == 1: {True == 1}")
print(f"True is 1: {True is 1}")
print(f"\nIdentity vs Equality:")
print(f"  bool(1) == True: {bool(1) == True}")
print(f"  bool(1) is True: {bool(1) is True}")
print(f"  bool(0) == False: {bool(0) == False}")
print(f"  bool(0) is False: {bool(0) is False}")

1.5 String (str)

Strings in Python 3 are immutable sequences of Unicode codepoints.

s = "Hello, Data Science!"

print(f"String: {s}")
print(f"Length: {len(s)}")
print(f"First char: {s[0]}")
print(f"Last char: {s[-1]}")
print(f"Substring: {s[7:21]}")
print(f"Reversed: {s[::-1]}")
print(f"Memory per char: {sys.getsizeof('a')} bytes")
print(f"Total string memory: {sys.getsizeof(s)} bytes (includes overhead)")
print(f"\nImmutability demonstration:")
try:
    s[0] = 'h'
except TypeError as e:
    print(f"  Cannot modify: {e}")
print(f"  Must create new: {'h' + s[1:]}")

2. Type Casting and Type Hints

2.1 Why Type Casting Matters

Data scientists frequently encounter type mismatches when loading data:

# Common data type issues in data science
raw_data = ["1", "2.5", "3", "invalid", "4.0"]

# These will fail or give unexpected results
print("String to int (with validation):")
clean_ints = []
for x in raw_data:
    try:
        clean_ints.append(int(x))
    except ValueError:
        print(f"  Skipping '{x}' - not convertible to int")

print(f"Result: {clean_ints}")

# Float conversion is more forgiving
print("\nString to float (with validation):")
clean_floats = []
for x in raw_data:
    try:
        clean_floats.append(float(x))
    except ValueError:
        print(f"  Skipping '{x}' - not convertible to float")

print(f"Result: {clean_floats}")

Output:

String to int (with validation):
  Skipping '2.5' - not convertible to int
  Skipping 'invalid' - not convertible to int
  Skipping '4.0' - not convertible to int
Result: [1, 3]

String to float (with validation):
  Skipping 'invalid' - not convertible to float
Result: [1.0, 2.5, 3.0, 4.0]

2.2 Type Hints for Data Science

Type hints improve code readability and enable static analysis:

from typing import List, Dict, Optional, Union, Tuple
import numpy as np

# Function with proper type hints
def compute_statistics(
    data: List[float],
    weights: Optional[List[float]] = None
) -> Dict[str, float]:
    """
    Compute weighted mean and standard deviation.
    
    Args:
        data: List of numerical values
        weights: Optional weights for each value
        
    Returns:
        Dictionary with 'mean' and 'std' keys
    """
    if weights is None:
        weights = [1.0] * len(data)
    
    mean = sum(d * w for d, w in zip(data, weights)) / sum(weights)
    variance = sum(w * (d - mean) ** 2 for d, w in zip(data, weights)) / sum(weights)
    
    return {
        "mean": mean,
        "std": variance ** 0.5
    }

# Usage with type checking
data = [1.0, 2.0, 3.0, 4.0, 5.0]
result = compute_statistics(data)
print(f"Statistics: {result}")
print(f"Type: {type(result)}")

3. Lists vs Tuples vs Arrays

3.1 The Fundamental Difference

Memory Layout Comparison:

LIST (mutable):
┌──────────┬──────────┬──────────┐
│ pointer  │ pointer  │ pointer  │  ← List object (PyObject*)
└────┬─────┴────┬─────┴────┬─────┘
     │          │          │
     ▼          ▼          ▼
┌────────┐ ┌────────┐ ┌────────┐
│ int(1) │ │ int(2) │ │ int(3) │  ← Individual int objects
└────────┘ └────────┘ └────────┘

TUPLE (immutable):
┌──────────┬──────────┬──────────┐
│ int(1)   │ int(2)   │ int(3)   │  ← Values stored inline
└──────────┴──────────┴──────────┘

NUMPY ARRAY (contiguous):
┌──────────────────────────────┐
│ int(1) │ int(2) │ int(3)    │  ← Contiguous C array
└──────────────────────────────┘

3.2 Performance Comparison

import time
import numpy as np

def benchmark_list(n):
    start = time.perf_counter()
    lst = list(range(n))
    for i in range(n):
        lst[i] = lst[i] * 2
    return time.perf_counter() - start

def benchmark_tuple(n):
    start = time.perf_counter()
    tup = tuple(range(n))
    tup = tuple(x * 2 for x in tup)  # Must create new tuple
    return time.perf_counter() - start

def benchmark_numpy(n):
    start = time.perf_counter()
    arr = np.arange(n)
    arr = arr * 2  # Vectorized operation
    return time.perf_counter() - start

n = 1_000_000
print(f"Benchmarking with n={n:,}:")
print(f"  List:   {benchmark_list(n):.4f}s")
print(f"  Tuple:  {benchmark_tuple(n):.4f}s")
print(f"  NumPy:  {benchmark_numpy(n):.4f}s")

Output (typical):

Benchmarking with n=1,000,000:
  List:   0.0892s
  Tuple:  0.1245s
  NumPy:  0.0043s

3.3 Time Complexity Table

*NumPy arrays have amortized O(1) append when pre-allocated

3.4 When to Use Each

import numpy as np

# USE LIST when:
# - You need mixed types
# - You need to append frequently
# - You're building data incrementally
data_list = [1, "hello", 3.14, [1, 2, 3]]

# USE TUPLE when:
# - Data should not change (coordinates, RGB values)
# - Dictionary keys (hashable)
# - Function returns with multiple values
point = (3.0, 4.0)
rgb = (255, 128, 0)

# USE NUMPY ARRAY when:
# - Numerical computation on large datasets
# - Vectorized operations needed
# - Memory efficiency matters
data_array = np.array([1.0, 2.0, 3.0, 4.0, 5.0])

# Demonstrate hashability
try:
    hash(point)
    print(f"Tuples are hashable: hash({point}) = {hash(point)}")
except TypeError as e:
    print(f"Lists are NOT hashable: {e}")

# Demonstrate vectorization
print(f"\nVectorized operations:")
print(f"  data_array * 2 = {data_array * 2}")
print(f"  data_array ** 2 = {data_array ** 2}")
print(f"  np.mean(data_array) = {np.mean(data_array)}")

4. Dictionaries: Hash Tables

4.1 How Hash Tables Work

Dictionary Structure (simplified):

Key: "name" → hash("name") = 946783568
     ↓
Index: 946783568 % 8 = 0

┌───────┬─────────────────────────────┐
│ Index │ Bucket                      │
├───────┼─────────────────────────────┤
│   0   │ ("name", "Alice") ──────────┼──→ ("age", 30)
├───────┼─────────────────────────────┤
│   1   │ None                        │
├───────┼─────────────────────────────┤
│   2   │ ("city", "NYC")             │
├───────┼─────────────────────────────┤
│   3   │ None                        │
├───────┼─────────────────────────────┤
│   4   │ ("email", "a@b.com") ───────┼──→ ("phone", "555-0123")
├───────┼─────────────────────────────┤
│   5   │ None                        │
├───────┼─────────────────────────────┤
│   6   │ None                        │
├───────┼─────────────────────────────┤
│   7   │ None                        │
└───────┴─────────────────────────────┘

Hash collisions are resolved by chaining (linked lists).

4.2 Dictionary Operations

import sys

# Create dictionary
person = {
    "name": "Alice Johnson",
    "age": 30,
    "city": "New York",
    "skills": ["Python", "ML", "Statistics"]
}

print(f"Dictionary: {person}")
print(f"Memory: {sys.getsizeof(person)} bytes")
print()

# O(1) lookup
print("O(1) Lookups:")
print(f"  person['name'] = {person['name']}")
print(f"  person.get('age', 'N/A') = {person.get('age', 'N/A')}")
print(f"  person.get('salary', 'N/A') = {person.get('salary', 'N/A')}")
print()

# Dictionary comprehension
squares = {x: x**2 for x in range(10)}
print(f"Comprehension: {squares}")
print()

# Nested dictionary access
nested = {
    "user": {
        "profile": {
            "name": "Bob",
            "scores": [95, 87, 92]
        }
    }
}

# Safe nested access
def get_nested(d, keys, default=None):
    """Safely access nested dictionary keys."""
    current = d
    for key in keys:
        if isinstance(current, dict) and key in current:
            current = current[key]
        else:
            return default
    return current

print(f"Nested access: {get_nested(nested, ['user', 'profile', 'name'])}")
print(f"Missing key:   {get_nested(nested, ['user', 'address', 'city'])}")

4.3 Performance Characteristics

import time

def lookup_benchmark(n):
    """Benchmark dictionary vs list lookup."""
    # Create data
    data_dict = {f"key_{i}": i for i in range(n)}
    data_list = [(f"key_{i}", i) for i in range(n)]
    
    search_key = f"key_{n-1}"
    
    # Dict lookup
    start = time.perf_counter()
    for _ in range(1000):
        _ = data_dict[search_key]
    dict_time = time.perf_counter() - start
    
    # List search
    start = time.perf_counter()
    for _ in range(1000):
        _ = [x for x in data_list if x[0] == search_key]
    list_time = time.perf_counter() - start
    
    return dict_time, list_time

n = 100_000
dict_t, list_t = lookup_benchmark(n)
print(f"Lookup benchmark (n={n:,}, 1000 iterations):")
print(f"  Dict: {dict_t:.4f}s")
print(f"  List: {list_t:.4f}s")
print(f"  Speedup: {list_t/dict_t:.1f}x faster with dict")

The dictionary's O(1) average lookup time is achieved through hash-based indexing with collision resolution, making it the foundation of Python's data model.

5. Sets: Membership Testing and Deduplication

5.1 Set Operations

# Create sets
skills_ml = {"Python", "TensorFlow", "Scikit-learn", "Pandas"}
skills_data = {"Python", "SQL", "Pandas", "Tableau", "Excel"}

# Set operations
print("Set Operations:")
print(f"  Union:        {skills_ml | skills_data}")
print(f"  Intersection: {skills_ml & skills_data}")
print(f"  Difference:   {skills_ml - skills_data}")
print(f"  Symmetric:    {skills_ml ^ skills_data}")
print()

# Deduplication
data = [1, 2, 2, 3, 3, 3, 4, 4, 4, 4, 5, 5, 5, 5, 5]
unique = list(set(data))
print(f"Original: {data}")
print(f"Unique:   {sorted(unique)}")
print(f"Count:    {len(data)} → {len(unique)}")
print()

# Membership testing benchmark
import time

large_list = list(range(1_000_000))
large_set = set(range(1_000_000))

# Test membership
test_value = 999_999

start = time.perf_counter()
for _ in range(100):
    _ = test_value in large_list
list_time = time.perf_counter() - start

start = time.perf_counter()
for _ in range(100):
    _ = test_value in large_set
set_time = time.perf_counter() - start

print(f"Membership test (1M elements, 100 iterations):")
print(f"  List: {list_time:.4f}s")
print(f"  Set:  {set_time:.4f}s")
print(f"  Speedup: {list_time/set_time:.1f}x faster with set")

6. Mutable vs Immutable

6.1 Why Immutability Matters

Mutable Object (list):
┌──────────────┐
│ id: 14023456 │──→ [1, 2, 3, 4, 5]
└──────────────┘
     ↓ modified in place
┌──────────────┐
│ id: 14023456 │──→ [1, 2, 3, 4, 5, 6]  ← Same object
└──────────────┘

Immutable Object (tuple):
┌──────────────┐
│ id: 14023456 │──→ (1, 2, 3, 4, 5)
└──────────────┘
     ↓ "modification" creates new object
┌──────────────┐
│ id: 14023520 │──→ (1, 2, 3, 4, 5, 6)  ← Different object
└──────────────┘

import sys

# Mutable: in-place modification
list_a = [1, 2, 3]
list_b = list_a  # Both point to same object

print("Mutable (list):")
print(f"  list_a id: {id(list_a)}")
print(f"  list_b id: {id(list_b)}")
print(f"  Same object: {list_a is list_b}")

list_b.append(4)
print(f"  After append(4) to list_b:")
print(f"  list_a: {list_a}")  # Also changed!
print(f"  list_b: {list_b}")
print()

# Immutable: new object created
tuple_a = (1, 2, 3)
tuple_b = tuple_a  # Both point to same object

print("Immutable (tuple):")
print(f"  tuple_a id: {id(tuple_a)}")
print(f"  tuple_b id: {id(tuple_b)}")
print(f"  Same object: {tuple_a is tuple_b}")

tuple_b = tuple_b + (4,)  # Creates new tuple
print(f"  After concatenation:")
print(f"  tuple_a: {tuple_a}")  # Unchanged!
print(f"  tuple_b: {tuple_b}")
print(f"  Same object now: {tuple_a is tuple_b}")
print()

# Immutability enables safe dictionary keys
location_data = {
    (40.7128, -74.0060): "New York",
    (34.0522, -118.2437): "Los Angeles",
    (41.8781, -87.6298): "Chicago"
}

print("Dictionary with tuple keys:")
for coords, city in location_data.items():
    print(f"  {coords}: {city}")

6.2 String Interning (Caching)

# Python caches small integers and some strings
a = "hello"
b = "hello"
c = "".join(["h", "e", "l", "l", "o"])

print(f"a is b: {a is b}")  # True (interned)
print(f"a is c: {a is c}")  # False (created dynamically)
print(f"a == c: {a == c}")  # True (value equality)
print()

# Integer caching (-5 to 256)
x = 256
y = 256
print(f"256 is 256: {x is y}")  # True (cached)

x = 257
y = 257
print(f"257 is 257: {x is y}")  # False (not cached)

7. Memory Layout Visualization

7.1 Stack vs Heap

Memory Layout:

STACK (fast, limited size):
┌─────────────────────────────────────────────┐
│ Frame: main()                               │
│  ┌─────────┐ ┌─────────┐ ┌─────────┐       │
│  │ x = 5   │ │ y = 3.14│ │ s = ────┼──┐    │
│  └─────────┘ └─────────┘ └─────────┘  │    │
│  ┌─────────┐ ┌─────────┐              │    │
│  │ flag=True│ │ temp = ─┼────┐        │    │
│  └─────────┘ └─────────┘    │        │    │
└──────────────────────────────│────────│────┘
                               │        │
HEAP (slow, large):            │        │
┌──────────────────────────────│────────│────┐
│  ┌─────────┐                 │        │    │
│  │ "Hello" │ ←───────────────┘        │    │
│  └─────────┘                          │    │
│  ┌─────────┐                          │    │
│  │ [1,2,3] │ ←────────────────────────┘    │
│  └─────────┘                               │
│  ┌─────────┐                               │
│  │ {"a": 1}│                               │
│  └─────────┘                               │
└─────────────────────────────────────────────┘

7.2 Reference Counting

import sys

# Reference counting
a = [1, 2, 3]
print(f"Initial refcount: {sys.getrefcount(a) - 1}")  # -1 for getrefcount arg

b = a  # Increases refcount
print(f"After b = a: {sys.getrefcount(a) - 1}")

c = a  # Increases refcount
print(f"After c = a: {sys.getrefcount(a) - 1}")

del b  # Decreases refcount
print(f"After del b: {sys.getrefcount(a) - 1}")

del c  # Decreases refcount
print(f"After del c: {sys.getrefcount(a) - 1}")

# Circular reference detection
class Node:
    def __init__(self):
        self.reference = None

x = Node()
y = Node()
x.reference = y
y.reference = x  # Circular reference

print(f"\nCircular reference: x→y→x")
print(f"  x refcount (includes getrefcount): {sys.getrefcount(x) - 1}")
print(f"  y refcount (includes getrefcount): {sys.getrefcount(y) - 1}")
print(f"  Garbage collector handles circular refs")

8. Complete Code Example: Data Science Utilities

import numpy as np
from typing import List, Dict, Any, Optional
from collections import defaultdict

class DataScienceToolkit:
    """Practical data structure utilities for data science."""
    
    @staticmethod
    def safe_convert(value: Any, target_type: type, default: Any = None) -> Any:
        """Safely convert value to target type."""
        try:
            return target_type(value)
        except (ValueError, TypeError):
            return default
    
    @staticmethod
    def flatten_dict(d: Dict, parent_key: str = '', sep: str = '.') -> Dict:
        """Flatten nested dictionary."""
        items = []
        for k, v in d.items():
            new_key = f"{parent_key}{sep}{k}" if parent_key else k
            if isinstance(v, dict):
                items.extend(DataScienceToolkit.flatten_dict(v, new_key, sep).items())
            else:
                items.append((new_key, v))
        return dict(items)
    
    @staticmethod
    def chunk_list(lst: List, chunk_size: int) -> List[List]:
        """Split list into chunks."""
        return [lst[i:i + chunk_size] for i in range(0, len(lst), chunk_size)]
    
    @staticmethod
    def weighted_average(values: List[float], weights: List[float]) -> float:
        """Compute weighted average."""
        return sum(v * w for v, w in zip(values, weights)) / sum(weights)
    
    @staticmethod
    def detect_outliers_iqr(data: List[float], factor: float = 1.5) -> List[int]:
        """Detect outliers using IQR method."""
        sorted_data = sorted(data)
        n = len(sorted_data)
        q1 = sorted_data[n // 4]
        q3 = sorted_data[3 * n // 4]
        iqr = q3 - q1
        lower = q1 - factor * iqr
        upper = q3 + factor * iqr
        return [i for i, x in enumerate(data) if x < lower or x > upper]

# Usage example
toolkit = DataScienceToolkit()

# Type conversion
print("Safe type conversion:")
print(f"  '42' → int: {toolkit.safe_convert('42', int)}")
print(f"  'abc' → int: {toolkit.safe_convert('abc', int, default=0)}")
print(f"  '3.14' → float: {toolkit.safe_convert('3.14', float)}")
print()

# Dictionary flattening
nested = {
    "user": {
        "profile": {
            "name": "Alice",
            "age": 30
        },
        "settings": {
            "theme": "dark",
            "notifications": True
        }
    }
}
flattened = toolkit.flatten_dict(nested)
print("Flattened dictionary:")
for key, value in flattened.items():
    print(f"  {key}: {value}")
print()

# List chunking
data = list(range(1, 11))
chunks = toolkit.chunk_list(data, 3)
print(f"Chunked list ({data}):")
for i, chunk in enumerate(chunks):
    print(f"  Chunk {i}: {chunk}")
print()

# Outlier detection
np.random.seed(42)
normal_data = np.random.normal(0, 1, 100).tolist()
normal_data.extend([10, -10, 15])  # Add outliers
outliers = toolkit.detect_outliers_iqr(normal_data)
print(f"Outlier detection: found {len(outliers)} outliers")
print(f"  Outlier indices: {outliers}")
print(f"  Outlier values: {[normal_data[i] for i in outliers]}")

Key Takeaways

Practice Exercises

Exercise 1: Performance Analysis

Write a function that benchmarks list vs dictionary lookup for finding an element in a collection of 1 million items. Include timing and discuss the results.

Exercise 2: Nested Data Processing

Given a nested dictionary representing a student database, write a function to:

1. Flatten the dictionary
2. Extract all grades
3. Calculate statistics (mean, median, std)
4. Identify students with grades below threshold

Exercise 3: Memory Optimization

Create a data structure that stores 10,000 records with name, age, and email. Compare memory usage between:

• List of dictionaries
• List of tuples
• NumPy structured array
• Pandas DataFrame

Discuss which approach is most memory-efficient and why.

Exercise 4: Type Conversion Pipeline

Build a robust type conversion function that handles:

• Mixed types in input
• Missing values (None, NaN)
• Invalid conversions
• Logging of conversion failures

Test with a dataset containing 1000 mixed-type values.

Exercise 5: Set Operations for Data Analysis

Using set operations, write a function to find:

• Common features between two datasets
• Features unique to each dataset
• The symmetric difference Discuss why set operations are preferred over list operations for this use case.