BaaS: Burnout as a Service

I wanna take a moment to to address what I like to call Burnout as a Service; how I see burnout as a product of the tech industry and culture. My friends, coworkers, and I have all experienced it in various levels, sometimes to crippling physical side effects. In this article I'm going to use strong language like need, and should, and I won't work with you if you [don't|do] $thing. While I have strong opinions about this subject and feel I have developed a powerful framework for helping avoid burnout it’s bound not to be perfect or complete. I do study this subject at great length in the name of personal development and productivity/energy management so if you have thoughts, opinions, feedback, or insights further into this topic I'd be happy to hear about them in the comments.

I'd like to point to this article, written by an amazing operations expert who has reached the jaded level where all things are approached solely by the perceived possible failure points, causing decision paralysis. Having lived the fear of doing for the failure it may cause down the road it I personally identify with this brand of burnout myself.

In my experience I have come to see burnout as a product of our tech industry's culture more than anything else. There is a lot of different little causes that all stack up on each other; the fast evolution of technology, the constant threat from bad actors, high stakes companies built on investors, product first revenue second business models, fickle customers caused by untested in market products, unheard of before uptime requirements on hastily developed systems, etc. The list could go on listing reasons why the industry itself is a primary source of maximum energy drain.

My mother is a healthcare worker who often asks why I make as much if not more than her working on computers. I often have to remind her that every moment, at work or not, I can be more or less responsible for the shut down of an entire company's revenue stream (possibly permanently) not just through negligence but from lack of anticipating the next failure or attack and guarding against it properly.

Let that sink in for a few moments. As an Operations Engineer or Developer not just my negligence but my lack of constant vigilance, research, and forethought can destroy an entire company. From bugs that open security holes, infrastructure mistakes that allow hidden-until-failure single points of failure, to a seemingly solid design choice that causes massive cascading failures in ways I never expected1. Let's not even address what happens when someone makes an oops uh-oh and that backup you really need is coming back corrupt.

However all this is the nature of the industry we buy into often with the understanding of that this is how it works. I love having a job where mentally jogging day in and day out to keep pace or even get ahead of developing technology and security vectors. This is my great mental stimulation and when I tire of it I'll go pour drinks at a bar.

However we all need to both expend and recover energy in equal amounts. This is a core biological imperative that we don't often think about. Our desire to always be creating (which expends great amounts of mental energy) drives us all to burnout. An amount of rest and recuperation in equal volume and type to our expenditures needs to be done regularly. So many of us ignore the daily, weekly, and monthly cycles of stress and rest that are necessary to work at peak optimal shape. Instead we push for days, week, or even months to try to reach a constantly moving goalpost with a promise that we will someday maybe take that vacation we need. This is only assisted by our 24/7/365 pagers and systems that take no rest, constantly waiting to fail or fall over from the ever present threat of bad actors or full hard drives.

I have watched pager fatigue alone completely destroy someone mentally. A poorly managed monitoring system that pages over things that aren't absolutely actionable and urgent or doesn't soft notify well enough in advance things that could be resolved before they become a critical issue is psychological abuse when delivered at the right volume. If someone can't disconnect because of the ever present pages that may or may not be actually actionable and critical then they are being slowly tortured, nerves frayed down with the rasp of their own phones.

At an even higher level I think we actively foster burnout amongst our peers and even ourselves with this great rockstar solo act so many of us pull. The concept of the Bastard Operator From Hell (BOFH) is the singlehanded "everything IT" person who has built everything from ground up and maintains everything even in the face of his users "always breaking everything". Because of this they become so jaded that they begin to torment their own end users and customers for mental relief. The worst part of this is we have formed a whole worship culture about being the lone gunman tech asshole to a point where I have seen a lot of people glorifying and emulating it well above and beyond their own time.

If we built up the proper support systems both mentally and technically we would be able to weather the storms the environment and systems rain down on us much better. If we worked together as much as possible instead of competing we would be well armed against the ever-present threat of burnout. It tends to be a lot to do with the personality types we pulled to technology for years; high on technical knowledge, low on social skills, lots of communication through digital means and not a lot of interpersonal interactions. Mix this in with the E/INTJ Type A personalities that are drawn to this higher stakes world of startups and high payoff companies and we develop this culture where we think everyone needs to be a rockstar or a ninja. We slim down staffing and just "hire the most brilliant mind in tech" to not just design but also implement and support these companies ad infinitum or more realistically, until they burn out, quit, or both.

I understand staffing is expensive and money is tight when your product is still only on the verge of success but in so much of operations and development we are paid to think, not to turn cogs. We design, develop, and foster ideas and solutions to problems no one else has solved. These kinds of ideas are not easily grown in a vacuum, but best cultivated through discourse and experimentation. However difficult it is to measure these expenditures or notice when they are getting strained there are ways we can approach them that helps identify issues faster as well as spread the mental load out more safely.

The best thing that ever happened to me in my experience in operations was learning to foster an interpersonal technical rapport with my co-workers and keeping it open. Constantly jogging ideas back and forth, never letting myself, or them stick on a problem and instead kicking it out to jog between us. I've done it twice and now it's a job requirement for me. The ability to "pair up" with operations to constantly foster and develop the most efficiently.

You see a lot of this starting to bud up in the tech world these days actually;

  • Open floor plans and chat based teams open up as much quick and easy discussion on issues as possible.
  • Code reviews are becoming the norm not just in development but in several forms of operations2.
  • The whole DevOps movement has large parts about enhancing communication and working together with others to help pool strengths and minimize weaknesses.
  • Some parts of agile/scrum are all about raising concerns and roadblocks as quickly as possible to put them up to the whole group, not trying to stick a single developer to solve a hard problem.
  • Pair programming is the next evolution of this, literally putting two minds to a single problem to solve it as efficiently and quickly as possible.

Even with all these trends though I still don't see enough of it, enough brainstorming, enough idea swapping, enough "Hey man I'm trying to do this like this but..." and that's why we are stuck staring at the flaws in every system. We work in our closed loops assuming that Issac Newton really did just sit under a tree staring at apples until he invented the Law of Gravitation so if we stare at our Apples and burn up all our mental energy the best way to handle this new package deploy will come to us. This is really where burnout comes from.

In closing I'd like to talk a little bit more about my mother. She's been a registered nurse her whole professional life. She's started in emergency rooms and for as long as I remember has been a hospice nurse. As you can imagine she's best at dealing with emergencies and the dying. Having dealt with the maimed and dying her whole life she's gone though mandatory on the clock grief therapy all of her professional life as well. In heath care when you deal with extreme emotional situations they regularly put you through therapy to help deal with and recover from what you work with. Whenever I ask someone who works about it it feels obvious to them that someone put though that level of emotional strain regularly would snap and do something horrible if they didn't receive regular emotional therapy. So as someone who goes through regular mental strain what are you doing for mental therapy to prevent your own burnout?


tl;dr

In my opinion burnout is a natural product of the tech industry culture. In order help combat this you need to do all of the following, which is not a panacea but a powerful preventive framework;

  • Work with others to help solve problems instead of trying to be a rockstar and solve them all yourself.
  • Create review processes around configuration and software changes so that you have reduced liability and risk.
  • Work with someone to develop intelligent alerting as well as a support rotation that allows rest and relaxation cycles as frequently as possible to reduce pager fatigue.
  • if you have critical failures more than once a quarter you need to review your infrastructure & procedures and ask yourself what your company is doing that's more important than having a reliable, usable product.
  • Rest yourself mentally every day, take breaks when you are locked in to a problem, and take regular vacations.
  • Bring in as much positivity to your workplace and your life as possible.
  • Fight negativity by analyzing it down to its root causes instead of superficially dismissing or accepting it.

And I hear you saying it already; "This isn't important right now", "I/we don't have the bandwidth for this at the moment", "I'm doing fine right now", or the worst "I'm not gonna burn out".

But know that when you do burn out I can promise you that you absolutely will not have the bandwidth or mental energy to do these things. By the time you are so far into burnout you actually become self aware of it deadlines will have slipped, everyone's already jaded and unhappy, and maybe you won't even care enough to implement this change.


  1. I have some amazing stories about Sybase databases running on a failover Sun Cluster using Veritas Volume Manager and the catastrophic problem with SCSI-3 reservations preventing disks to randomly not be mountable during a fail over event caused only when the LUNs are provided from newer NetApp hardware. Long story short, the database will start without half it's disks and it is not a pretty sight where it goes from there. 

  2. The third best thing I have ever done in my career so far was take all of a companies critical configurations (haproxy, dns, maintenance crons, backup scripts, monitoring, ect) and put them in git, created a code review process, and used jenkins to deploy to production. Call it CI/CD, call it change control, call it whatever you want; I call it almost never again making that one line stupid oops uh oh that breaks an entire system again. 

Being Happy

In my personal experience not everyone wants to be happy. When I say that I'm not talking about that one friend either. I'm talking about me. For a very long time I didn't wanna be happy. It was a very frustrating time. I can't say I knew I wanted to be unhappy, I just was all the time. I was angry, depressed, and off and on apathetic. It was actually my mother who coined the phrase "You just need to be unhappy." somewhere in my early to mid 20s.

Above and beyond my own internal strife one of my problems was that those around me were constantly interested in trying to make me happy. Maybe that doesn't sound like a problem to you but it was aggravating to me. It wasn't that people wanted to help it was this frustrating struggle of people offering what I saw as cookie cutter back pats and weak attempts to stoke a nonexistent ego instead of just facing the truth and seeing the world like I saw it. They didn't understand my problems. I thought about how miserable life is and how horrific this whole world works. I thought all the time about how my mediocre skill set in an over saturated field that shared a job pool with savants & geniuses meant I would going to toil and claw against an over abundant and under paid workforce. I thought about how compared to my peers my nothing-to-offer existence meant I get to writhe away in this world alone with a few cynical friends to cheerlead ourselves along to the grave. I looked at the ugly side of every story and trust me when you stop sugar coating everything you realize that there is millions of fucked up things going on every day and we do our best to gloss over it and put lipstick on this pig so that we can wake up and say today is an awesome day. This whole world is a corrupt Masque of the Red Death; an extravagant gala thrown by the privileged to hide away the social plague destroying everyone else around us.

Over time I learned how to communicate with others about my feelings in ways that helped prevent the constant fawning over my state. Of course I had to be highly selective of who I chose to associate with since I didn't want to have to go through this rigmarole constantly. I wasn't happy at that point but I reached this semi-content equilibrium where I got by with thick sense of dark humor and snark, as much as one possibly could exist in this sickness.

After a long while doing this I reached this weird nirvana where I was just me, and everyone was ok with it and I was ok with it to. I was just ok. Then the weird shit happened.

Through having this small social group, literally six to eight people deep, I was able to find this confidence in myself. Maybe what I could do was shit to anyone else but I could do things that made a difference to my friends, or at least impressed them. Over a period of about ten years give or take this grew, as well as my social circle. I often felt like the imposter in the room but through all these people I started to realize my own potential.

I can't say the exact moment it hit me. I know it was when I was working for Stephens. I had stuck my foot out enough times and somehow not gotten the door slammed on it enough times that I had made it somewhere in the company. The group I was running hadn't completely imploded on itself around me yet and I was a pretend famous DJ. Somewhere around this time it dawned on me that life isn't the Olympics but an Industry and even if I sucked the fact I wasn't going to stop trying made me valuable.

It was around that time I stopped reading fiction and switched to non-fiction. I read a lot of 90s-00s new era "be awesome at life" self help books and started implementing all these systems and tricks I read about.

My personal mantra around then was "It Never Hurts to Help"; a tongue in cheek reference to a cartoon from my youth called Eek the Cat. It was a morbid tale about an anthropomorphic feline whose overly sunny attitude and unflappable willingness to help other constantly ended him up in the hospital. That was his catch phrase, the one he said right before he was mauled by something. I like to say I was using it ironically since in the end I rarely caught fire after saying it but I did end up getting places professional and personally.

Needless to say life started moving really fast when I became truly motivated to help and get things done. I can't say I was happy... but I was really busy.

It was around then the shift really happened. I don't know if my attitude shifted first or those around me but things became nightmarishly disjointed and stressful at work over bad management, the club promoters I was working with went to war with another promotions group, and the community I was dealing with collapsed around the time someone slept with a minor unknowing and then someone else killed themselves. Needless to say these were dark times. However through all this stress I had a mantra and I stuck to it. Suddenly I was a too positive person for those I was around.

I was right back to where I was before in the reverse way. Everyone told me I need to "take it down a notch" and accused me of being disingenuous and sarcastic simply because I'm living my live the way I chose. I can't possibly feel like that, I can't do this, and we can't do that.

That's when my mantra changed to "I Only Have Cans". Just like before I had to adjust who is important in my life since I don't want to surround myself with those who are going to try to slow my progress and scowl at my outlook. This new mantra isn't just a tongue and cheek spite of never giving up and always helping. This one is only having the positive, always being able to do something. I guess that's when I decided to be happy?

I can't say I'm always happy. In fact I'd go so far as to say no one really stops dealing with depression. I still have the eternal funeral procession of self doubt, loathing, paranoia, and ill wishes flickering through my mind like an unending film. However I can decide it doesn't control me and I have way way better things to do with my life than be consumed by my own innate apathy. I can say that I'm in control of my outlook and what is important to me and I can control the world around me enough to decide I'm gonna be happy this day.

My mantra has been changing lately. I didn't have a mantra for over 26 years and now suddenly in eight years I have gone through three of them. Like I said, things started moving fast. It's not final, nothing is, but these days I'm sticking with "Be Fucking Amazing".

Not bad for someone who needed to be unhappy almost his entire life.

For those who actually read this far, I didn't write this to publicly stoke my dick at everyone or at least that wasn't the original intention. I've been thinking a lot lately about those around me who are unhappy now. I get a little sad and want to go make them happy which reminds me of where I stood not that long ago. I'm not going to pretend my story applies to anyone else and this should be shared around facebook by duck lip hotties as some overly winded it gets better back pat. However I'd like to hope that there are plenty of people who struggle with needing to be unhappy that will learn to take control of their own world and be fucking awesome in their own right.

Quick Note on GnuRadio on Pentoo

Not a big blog, but a quick problem I got solved on IRC that I thought might help others.

I have a Gateway LT4009u with an Atom N2600. It's my "hacker/workshop" laptop. The atom N processors are a bit gimpy so sometimes things don't run right. One thing is GNURadio on Pentoo. Pentoo runs hardened and this pisses off the atom n.

So if you get the following error.

LLVM ERROR: Allocation failed when allocating new memory in the JIT
Can't Allocate RWX Memory: Operation not permitted

Then you need to soft disable hardened with the following command

sudo toggle_hardened

I hope that helps anyone else on the internet.

Thanks to Zero_Chaos in #pentoo on irc.freenode.net for the fix (and pentoo)

Quick Update: This also happens when running in VirtualBox 5 on my 2015 MacBook i7, but the fix is the same

Tagged