logging

Volatile by default on many distros, size-capped by silent policy, binary on disk. Know the storage modes, the vacuum knobs, and how journald feeds central logging.

The friendliest DaemonSet question there is, and it still separates people who have operated one from people who have read about one. The scheduling rules hide the traps.

The instinct is to delete old logs. That is backwards: the waste lives in what you ingest, not how long you keep it. This answer walks the levers in the order that actually pays.

The data-path question separates people who have sized a cluster from people who installed one. Shard sizing, refresh versus flush, and each ILM phase carry one specific failure mode apiece.

Sampling keeps proportions and loses rare individuals; filtering keeps chosen classes complete and loses whatever you misclassified. Decide retention at write time, keep counters always, rehydrate the rest.

Nothing errors because dropping unmatched lines is documented behaviour. Counting the tagged failures converts an invisible leak into a number, and numbers have owners.

The hard parts are not search. They are deciding what to keep before you pay to store it, isolating one team's query from everyone else's, and making the cost visible to the people generating it.

Prompts contain whatever the user typed, which in a support product is names, account numbers and medical detail. Full-text logging is a data-protection decision, not a debugging preference.

Most of the obligation lands on the product, but three parts land on infrastructure: an inventory of what is deployed, evidence of what it did, and the ability to turn it off. Those are platform capabilities and they take months to retrofit.