I worked for almost 4 years in the edge compute space, in a Cloudflare competitor running global Kubernetes clusters of clusters.
Our company, since acquired, ran multi cloud (AWS, Digital Ocean, GCP, others) and multi-locations (Sydney, Melbourne, San Francisco, New York, Ohio, London) etc.
These K8 clusters communicated via a message bus and all the Docker Pod containers had logging sent to centralized Elastic Search (ELK) stacks.
Logs are JSON, JSON is a very inefficient and verbose format. Logs are mostly noise. Logs are mostly the same or low information. Logs are only useful when they contain errors. The tragedy of logs is that you only want logs when things go wrong, but when there are no errors, you dont want them.
Not only is it expensive to log at the application level, its also more expensive to stream the logs to the central servers.
You get hit with log compute hosting fees and log streaming fees.Its a lot. In fact, log compute and log network fees can be actually more expensive than delivery.
I can guarantee this is true.
