On the beginning… I wrote this blog post for myself. Myself from 10 years ago, to be precise. This is a post I wish I could send back in time and show to myself. I’m sure it would change my direction and career. If you have heard about Erlang but are not sure if it is worth giving it a try…or you just love Erlang and would like to hear my praise for it, please read on.
gen_event confused me from the beginning, so I wanted to investigate the topic more deeply. I did that here. Then I left that topic, and it returned recently to me when I was wondering how the situation changed since then. Here is the updated version of the initial investigation, which started with the following statement: I never used a gen_event, I think it is a bad pattern. At first, it may look like a controversial statement, but I heard a lot of those complaints from other people.
Observability-first Approach Tracing as a tool is immensely productive. It cannot be overestimated - especially in production, but also in development when debugging our own mistakes. In the previous article I covered why we should use that facility and why they invented it in the first place. After first sight and couple of rounds, you fall in love with that tool. Right after you gain some experience, more questions start to arise.
Debugging Concurrency Let me tell you a story. We have deployed a new version of software on the whole fleet. Hundreds of machines in 4 different regions around the world. We observed our metrics, inspected our logs - nothing there. Success, another deployment without any issues. Fast forward two days, and we received a customer support ticket to investigate. One of our integrations complained that they are observing a significant amount of errors when calling our endpoints, in two distinct regions.