Python Logging — Production-Ready Logging

The logging module provides flexible event logging for applications and libraries. Proper logging is essential for debugging, monitoring, and auditing production systems.

Learning Objectives

• Configure logging with basicConfig
• Create custom loggers with handlers and formatters
• Implement structured logging for production
• Use logging best practices

Why Not Just Use Print?

# print() problems:
print("Starting process")           # No timestamp
print("Error: file not found")      # No severity level
print(f"User {user_id} logged in")  # No way to filter

# logging solves all of these:
logging.info("Starting process")
logging.error("Error: file not found")
logging.info(f"User {user_id} logged in")

Basic Configuration

import logging

# Simple setup — good for development
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S'
)

# Create logger
logger = logging.getLogger(__name__)

# Log at different levels
logger.debug("Debug message")      # Not shown (below INFO)
logger.info("Info message")        # Shown
logger.warning("Warning message")  # Shown
logger.error("Error message")      # Shown
logger.critical("Critical message") # Shown

Log Levels Explained

Level    When to Use
─────    ──────────────────────────────────────────
DEBUG    Detailed info for developers (loop vars, API responses)
INFO     Confirmation that things are working (startup, requests)
WARNING  Something unexpected but not broken (disk 90% full)
ERROR    Something broke (failed to connect, exception)
CRITICAL Program may stop (database down, out of memory)

Custom Logger with Handlers

import logging
from logging.handlers import RotatingFileHandler

def setup_logger(name):
    logger = logging.getLogger(name)
    logger.setLevel(logging.DEBUG)

    # Console handler — show INFO and above
    console = logging.StreamHandler()
    console.setLevel(logging.INFO)
    console_fmt = logging.Formatter(
        '%(asctime)s - %(levelname)s - %(message)s',
        datefmt='%H:%M:%S'
    )
    console.setFormatter(console_fmt)

    # File handler — record DEBUG and above
    file_handler = RotatingFileHandler(
        f'{name}.log',
        maxBytes=5*1024*1024,  # 5MB per file
        backupCount=5           # Keep 5 backups
    )
    file_handler.setLevel(logging.DEBUG)
    file_fmt = logging.Formatter(
        '%(asctime)s - %(name)s - %(levelname)s - %(funcName)s:%(lineno)d - %(message)s'
    )
    file_handler.setFormatter(file_fmt)

    logger.addHandler(console)
    logger.addHandler(file_handler)

    return logger

# Usage
logger = setup_logger("myapp")
logger.info("Application started")
logger.debug("Loading configuration from %s", "config.json")
logger.error("Failed to connect to database: %s", "Connection refused")

Structured Logging (JSON)

Structured logs are machine-readable, making them easy to search and analyze in log aggregation tools (ELK, Datadog, CloudWatch).

import logging
import json
from datetime import datetime

class JSONFormatter(logging.Formatter):
    def format(self, record):
        log_entry = {
            "timestamp": datetime.utcnow().isoformat(),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
            "module": record.module,
            "function": record.funcName,
            "line": record.lineno,
        }
        if record.exc_info:
            log_entry["exception"] = self.formatException(record.exc_info)
        return json.dumps(log_entry)

# Setup structured logger
logger = logging.getLogger("api")
handler = logging.StreamHandler()
handler.setFormatter(JSONFormatter())
logger.addHandler(handler)

# Usage
logger.info("User logged in", extra={"user_id": 123, "ip": "192.168.1.1"})
# Output: {"timestamp": "2024-01-15T10:30:00", "level": "INFO", "message": "User logged in", ...}

Context Managers for Logging

import logging
import time
from contextlib import contextmanager

@contextmanager
def log_operation(logger, operation_name):
    """Log the start, duration, and result of an operation."""
    logger.info(f"Starting: {operation_name}")
    start = time.perf_counter()
    try:
        yield
        duration = time.perf_counter() - start
        logger.info(f"Completed: {operation_name} ({duration:.2f}s)")
    except Exception as e:
        duration = time.perf_counter() - start
        logger.error(f"Failed: {operation_name} ({duration:.2f}s) - {e}")
        raise

# Usage
logger = setup_logger("myapp")
with log_operation(logger, "data import"):
    import_data()  # Your operation here

Key Takeaways

1. Use getLogger(__name__) for module-level loggers
2. Set appropriate levels for different environments
3. Use structured logging (JSON) for production
4. Configure handlers for console, file, and external services
5. Use RotatingFileHandler to prevent huge log files
6. Include function name and line number in file logs
7. Never log sensitive data (passwords, tokens, PII)
8. Logging is better than print for production code