Logging in Python

Logging in Python is a module provided in the language, allowing it to have flexibility in emitting log messages from programs. It is mainly used to track events, debug, and record activities in programs for further analysis.

What is Logging?

Logging is recording events, warnings, errors, or other messages about the program’s execution. These logs, which contain messages, assist the developers in monitoring the program:

1. Track Program Behaviour: Understand what the program is doing.
2. Debugging issues: Identify issues and locate errors.
3. Audit: History of an action or any transaction by the system.

Unlike using the print() function, logging provides:

• Configurability: You can control where logs go (console, files, etc.) and what gets logged (severity levels).
• Structure: Log messages are formatted with timestamps, levels, and other context.
• Scalability: Logging works well for simple scripts and large, distributed systems.

Core Elements of Python’s Logging Framework

1. Logger

The logger is the entry point to use the logging framework, responsible for:

• Receiving log messages.
• Deciding their importance (based on the log level).
• Forwarding messages to Handlers for output.

A logger has a name. Most of the time, you will use logging.getLogger(name) to obtain or create a logger, such as:

import logging

logger = logging.getLogger("MyApp")
logger.info("An informational message from MyApp")

2. Log Levels

Log levels are used to categorize the severity of messages. Each level has a numeric value:

Example:

import logging

logging.basicConfig(level=logging.DEBUG)  # Set the lowest log level (DEBUG)

logging.debug("This is for debugging")
logging.info("General information")
logging.warning("This is a warning")
logging.error("An error occurred")
logging.critical("Critical failure!")

3. Handlers

Handlers define where log messages go. Examples:

• StreamHandler: Which sends logs to the console (stdout/stderr).
• FileHandler: Which logs to a file.
• RotatingFileHandler: When the log file reaches a certain size, this handler rotates it.
• SMTPHandler: Sends logs via an email.

A logger can have multiple handlers, and each handler can define its log level.

Example:

import logging

# Create logger
logger = logging.getLogger("ExampleLogger")
logger.setLevel(logging.DEBUG)

# Create console handler
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.DEBUG)

# Create file handler
file_handler = logging.FileHandler("example.log")
file_handler.setLevel(logging.ERROR)

# Add handlers to the logger
logger.addHandler(console_handler)
logger.addHandler(file_handler)

# Log messages
logger.debug("This appears in the console only")
logger.error("This appears in both the console and file")

4. Formatters

Formatters define how log messages are displayed. You can include details like:

• Timestamp (%(asctime)s)
• Logger name (%(name)s)
• Log level (%(levelname)s)
• Message (%(message)s)

Example:

import logging

# Custom formatter
formatter = logging.Formatter(
    fmt='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S'
)

# Apply formatter to handlers
console_handler = logging.StreamHandler()
console_handler.setFormatter(formatter)

logger = logging.getLogger("FormattedLogger")
logger.addHandler(console_handler)
logger.setLevel(logging.INFO)

logger.info("This is a formatted log message")

Output:

2025-01-09 13:00:00 - FormattedLogger - INFO - This is a formatted log message

5. Filters

Filters provide fine-grained control over which log records get passed to handlers.

Example:

import logging

class CustomFilter(logging.Filter):
    def filter(self, record):
        # Only allow messages that contain the word 'Important'
        return 'Important' in record.msg

logger = logging.getLogger("FilteredLogger")
logger.setLevel(logging.DEBUG)

# Add filter to a console handler
console_handler = logging.StreamHandler()
console_handler.addFilter(CustomFilter())
logger.addHandler(console_handler)

logger.info("This is not important")  # Will NOT be logged
logger.info("This is Important!")     # Will be logged

Configuration Methods

1. basicConfig (Quick Setup)

basicConfig is the simplest way to configure logging. Example:

import logging

logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s - %(levelname)s - %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S'
)

logging.info("Logging configured with basicConfig")

2. Configuring Multiple Loggers

For larger applications, you may want to configure multiple loggers differently.

import logging

# Logger for module1
logger1 = logging.getLogger("module1")
logger1.setLevel(logging.DEBUG)
console_handler = logging.StreamHandler()
logger1.addHandler(console_handler)

# Logger for module2
logger2 = logging.getLogger("module2")
logger2.setLevel(logging.WARNING)
file_handler = logging.FileHandler("module2.log")
logger2.addHandler(file_handler)

logger1.debug("Debug from module1")  # Console
logger2.warning("Warning from module2")  # File

3. Using Configuration Files

Logging can be configured using a dictionary or a file (YAML, INI).

YAML Example:

version: 1
formatters:
  detailed:
    format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
handlers:
  console:
    class: logging.StreamHandler
    level: DEBUG
    formatter: detailed
  file:
    class: logging.FileHandler
    filename: app.log
    level: ERROR
    formatter: detailed
loggers:
  MyApp:
    level: DEBUG
    handlers: [console, file]

Python code to load the configuration:

import logging.config
import yaml

with open("logging_config.yaml", "r") as f:
    config = yaml.safe_load(f)
logging.config.dictConfig(config)

logger = logging.getLogger("MyApp")
logger.debug("This will appear in the console")
logger.error("This will appear in the console and app.log")

Best Practices for Logging

1. Use Log Levels Wisely: Avoid using DEBUG or INFO in production.
2. Centralize Logging Configuration: Keep all settings in a configuration file.
3. Avoid Sensitive Data: Never log passwords or personal information.
4. Use Rotating Logs: Prevent disk space issues by using RotatingFileHandler.
5. Custom Loggers for Modules: Use separate loggers for different parts of the application.