Logging Guidelines
Here’s a set of detailed logging guidelines for DRISTI implementation to implement structured, configurable, and secure logging.
1. Define Log Levels Clearly
INFO: General information about application progress, such as starting and stopping services, user actions, or major events. Use this sparingly to avoid log bloat.
WARNING: Non-critical issues that do not cause a failure but may require attention, like a fallback operation or a near-limit resource usage.
ERROR: Issues that have caused a failure or have interrupted expected operations, such as an unhandled exception or a failed service call.
DEBUG: Detailed technical information useful for debugging, such as internal state or configuration details.
TRACE: Most granular level, showing step-by-step execution, typically useful only in development environments.
Configurable Logging: Enable these log levels based on environment and purpose (e.g., DEBUG in dev and INFO/WARN/ERROR in production). Log levels should be configurable in external settings files, such as log4j2.xml
in Java or log4net.config
in .NET.
2. Use Structured Logging
Implement structured logging (e.g., JSON format) to make logs easier to parse and analyze by log aggregation tools.
Log as key-value pairs where possible (e.g.,
{"event": "UserLogin", "username": "jdoe", "status": "Success"}
), to allow structured searches and insights in log monitoring systems.
3. Avoid Logging Sensitive or Confidential Information
Avoid Personal Identifiable Information (PII) like full names, emails, addresses, social security numbers, credit card details, or any other sensitive data.
Use hashing or masking for sensitive fields if logging them is unavoidable (e.g.,
userId: "***1234"
instead ofuserId: "12345678"
).Do not log credentials, security tokens, or session cookies.
Configure logs to exclude database schema information, internal service endpoints, file paths, and internal error messages.
4. Standardize Log Message Format
Use a consistent message format across all logs, which should ideally include:
Timestamp: Since the current implementation is India based, use IST date time format. While applications use EPOC time, avoid using them in logging, to make logging easier to quickly understand
Log Level: INFO, ERROR, etc.
Component/Service Name: Identify which part of the application generated the log.
Unique Request/Transaction ID: Correlate logs across services, especially in microservices architectures.
User Context (Anonymized ID): Where applicable, to trace issues back to a session or user without exposing sensitive information.
Message: Summary of the event.
Additional Metadata: Where needed, to provide more context without leaking internal details.
5. Use Exception Logging Carefully
Log only the necessary information from exceptions. Include the stack trace at ERROR level but remove or mask any internal details, such as file paths or line numbers, that could expose implementation details.
For nested exceptions, avoid logging the same error multiple times in the stack to prevent log bloat.
6. Log Application and Business Events
Log significant events like login attempts, data processing activities, or transaction completions to get an overview of application usage.
Use INFO level for these events, especially for actions like user logins, configuration changes, or permissions updates, which may be useful for security audits or tracking purposes.
Distinguish between logs for generic service events that are more useful during initial builds or debugging in DEV environment. In PROD need to ensure that initially only ERROR logging is turned on and in case of any exceptions, or error that needs debugging, enable additional logs by modifying the setting on configuration file.
Logging level can be dynamically changed using spring boot actuator endpoints. See this for reference.
7. Consider Log Rotation and Retention Policies
Configure log rotation to prevent log files from growing indefinitely. Use time-based (e.g., daily rotation) or size-based rotation.
Define retention policies to keep critical logs for auditing but purge older or less important logs to manage storage and comply with data protection policies.
8. Use Correlation IDs for Distributed Systems
In microservices architectures, propagate a correlation ID across all services for a single request. This ID helps trace the entire flow of the request across multiple services, which is crucial for debugging and incident response.
9. Use Centralized Logging and Monitoring
Set up centralized logging (e.g., ELK Stack, Splunk, Jaeger) to collect logs from various services and make it easier to search, filter, and analyze logs.
Use real-time monitoring tools to generate alerts on certain conditions, such as ERROR spikes, so that issues can be resolved proactively.
10. Secure Access to Logs
Ensure only authorized personnel have access to logs, especially those containing sensitive information.
Encrypt log files if they contain any sensitive or restricted-access information.
Limit access to the log aggregation and monitoring systems to authorized users and roles.
Example Logging Guidelines in Code
Bad Example
Good Example
Summary
Following these guidelines will help maintain an effective logging strategy that is secure, informative, and scalable. By focusing on clarity, security, and consistency, you create logs that are meaningful for troubleshooting and insights without exposing internal or sensitive information.
Last updated