I’ve come across this issue professionally more than a few times now, and decided to write it down. A lot of latency graphs are unusable. Observability is hard, and talking about performance is even harder. But it is made a lot harder than needed by the platforms built for this, than it should. This article is for everyone creating dashboards, especially when you use DataDog & GCP together.
Stop averaging your GCP percentiles in Datadog If you pull Google Cloud metrics into Datadog, there’s a good chance one of your latency graphs is lying to you.
Coming from a Google (GCP) ecosystem, there are many things that AWS does not do, that you’ll miss. One of these things is automatically mounting secrets from Secret Manager on your serverless Cloud Run instances like so:
gcloud run deploy SERVICE --update-secrets=PATH=SECRET_NAME:VERSION This is extremely convenient, as the application itself does not need to be aware of the complexities of retrieving the secret. This is a form of separation of concerns, and has the advantage that locally you can just use .
Sometimes your hobby project needs to run something continuouly, even in this ‘serverless-era’. In most situations you could get away with CronJobs (Google Cloud Scheduler) and static hosting & serverless functions, but not always. For example, if you consume an API that is only available over WebSockets, like when building a chat bot for example.
These applications often sit idle most of the time. They simply leave the connection running, “awaking” only when a message comes in.
When me and my collegues at Q42/Hue inspect our applications and there error messages they produce, we sometimes see this gem:
Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information. This seems to be a nice error message, as it clearly explains what happens and where more information can be found. However, the message and the background information do not capture all peculiarities that can cause this error.