Remote Debugging

How would you debug a highly concurrent system?

Debugging Concurrency Let me tell you a story. We have deployed a new version of software on the whole fleet. Hundreds of machines in 4 different regions around the world. We observed our metrics, inspected our logs - nothing there. Success, another deployment without any issues. Fast forward two days, and we received a customer support ticket to investigate. One of our integrations complained that they are observing a significant amount of errors when calling our endpoints, in two distinct regions.

Continue reading