In the dashboard which I hacked together at MMC RTV Slovenia, I display a set of related flickr searches based on the current state of the system. So far, there’s a distinct lack of snakes, volcanoes, lava, fire, explosions and other things that might freak an engineer out as much as a server outage.

Thoughts:

I like how the hosts are clearly marked when one of their metrics goes over a threshold. The CPU based hosts are on the left two columns, network is in the third, server load in the fourth. The right columns (disk and swap usage) are descriptive but only for a daily report, not so much for a live dashboard. I can clearly see the marked hosts across the room, even if I would miss the background.

I’m currently applying for a Flickr API key, and this is what I put into the description for “HQ Dashboard”:

“I’m building a monitoring dashboard which includes statistics data from internal sources, some tweets and, hopefully, an assortment of Flickr backgrounds based on current monitoring conditions (ie., search for fire/destruction/storm/tornado when servers are being overloaded, and grass/meadows/happy/coffee if the monitoring says everything is fine). In this way I plan to use the Flickr search to provide a visual representation of the severity level currently being experienced.”

In many ways I consider that the biggest strain with being a software developer is the intensive intellectual work. It’s draining. In my case, I think about problems, possible solutions. I code, test, fix and deploy changes. It’s stressful, demanding and has the tendency to consume your waking time even outside of work. There is no guaranteed off switch.

A man sits at a bar, his friends walk in. He’s thinking about efficiently pushing hundreds of millions of rows of data into a database. There is no punchline. This is literally my life most of the time, the only change is a bit different thought exercise.

Green lines are code deploys and red ones are requests going to a part of the cluster. I had a big deploy today, and I’m glad to come out of it relatively unscathed.

1. Frameworks now use Composer packages, more code reuse now
2. Updated structural framework, base classes
3. Relatively smart and simple dependency injection
4. Proper exception handling and notification

Obviously there was some breakage. I had to fix about 5 incorrectly implemented interfaces, out of which one was affecting work (3 minutes to fix in production!). There were some conflicts with legacy systems like WAP and a few other breaks due to some structural inconsistencies created over the years. Most of it was a bunch of quick fixes.

If I could give advice to any software developer it would be this:

Plan your software wisely. If the client needs new features - think about them. Don’t just think about if they scale, think about where the project is going. Think about the work you’ll do now, and the work you or someone else might have to do down the line because of your decisions. Having a procedure how to scale the work that the client sees is better than having to scale the part of your software which the client never gets to see. Enable your client, don’t let your client disable you. Sometimes rebuilding is the only option - and if it comes to that, you most certainly didn’t follow my advice up to this point.

It’s days like this where I love my job. I’m implementing back pressure for API calls that communicate to an external service. Depending on how overloaded the external service is, my API interface adapts to use caching more extensively, or to use a service friendly request retry strategy, minimizing impact on infrastructure and possibly even resolving problems when they occur. This is done by keeping track of a lot of data - timeouts, error responses, request duration, ratios between failed and successful requests,…