Monitoring and observability: knowing something broke before the customer
What to measure
Start with four: request rate, error rate, response time at the extremes, and resource saturation. Add one business metric — orders per hour, signups, sends — that will catch failures the infrastructure doesn't see.
Average response time is misleading. Look at the 95th and 99th percentiles; that's where the angry users are.
Useful logs
Structured, with a request identifier that follows a call across the whole system, and without needless personal information. A free-text log is an archive, not an investigation tool.
Alerts that don't wear you out
Alert on symptoms the user feels, not on every technical anomaly. Every alert should include what happened, the impact, and the first step. An alert with no action — delete it. Alert fatigue is the number-one cause of missing a real incident.
Going deeper
Add distributed tracing that shows a request's path across components with timings. In split systems that's the only way to answer “why did this take three seconds” without guessing, and it shortens investigations from hours to minutes.