Logs: store correctly and find fast
Structure
Fields, not sentences: time, level, service, version, request identifier, a masked user identifier, and an event. That way you can filter and aggregate instead of searching text.
Levels used consistently: error for what needs attention, warning for what's exceptional but handled, info for business events, debug only in development.
How much to keep
Logs are a significant cost item. Keep detailed for a short time, aggregated for a long time, and a cheap archive for what's needed for compliance. Set this once, or it's decided on its own — upward.
What not to write
Passwords, tokens, card numbers, identity numbers and private message content. A log is a database in every way, with fewer controls — so a leak from it is especially common.
Going deeper
Add a request identifier generated at the edge and passed to all components and to the user's response. When a customer reports an error with an identifier, the investigation starts from the right line — instead of scanning millions of lines around an estimated hour.