Filed under ramblings

BaaS: Burnout as a Service

I wanna take a moment to to address what I like to call Burnout as a Service; how I see burnout as a product of the tech industry and culture. My friends, coworkers, and I have all experienced it in various levels, sometimes to crippling physical side effects. In this article I'm going to use strong language like need, and should, and I won't work with you if you [don't|do] $thing. While I have strong opinions about this subject and feel I have developed a powerful framework for helping avoid burnout it’s bound not to be perfect or complete. I do study this subject at great length in the name of personal development and productivity/energy management so if you have thoughts, opinions, feedback, or insights further into this topic I'd be happy to hear about them in the comments.

I'd like to point to this article, written by an amazing operations expert who has reached the jaded level where all things are approached solely by the perceived possible failure points, causing decision paralysis. Having lived the fear of doing for the failure it may cause down the road it I personally identify with this brand of burnout myself.

In my experience I have come to see burnout as a product of our tech industry's culture more than anything else. There is a lot of different little causes that all stack up on each other; the fast evolution of technology, the constant threat from bad actors, high stakes companies built on investors, product first revenue second business models, fickle customers caused by untested in market products, unheard of before uptime requirements on hastily developed systems, etc. The list could go on listing reasons why the industry itself is a primary source of maximum energy drain.

My mother is a healthcare worker who often asks why I make as much if not more than her working on computers. I often have to remind her that every moment, at work or not, I can be more or less responsible for the shut down of an entire company's revenue stream (possibly permanently) not just through negligence but from lack of anticipating the next failure or attack and guarding against it properly.

Let that sink in for a few moments. As an Operations Engineer or Developer not just my negligence but my lack of constant vigilance, research, and forethought can destroy an entire company. From bugs that open security holes, infrastructure mistakes that allow hidden-until-failure single points of failure, to a seemingly solid design choice that causes massive cascading failures in ways I never expected1. Let's not even address what happens when someone makes an oops uh-oh and that backup you really need is coming back corrupt.

However all this is the nature of the industry we buy into often with the understanding of that this is how it works. I love having a job where mentally jogging day in and day out to keep pace or even get ahead of developing technology and security vectors. This is my great mental stimulation and when I tire of it I'll go pour drinks at a bar.

However we all need to both expend and recover energy in equal amounts. This is a core biological imperative that we don't often think about. Our desire to always be creating (which expends great amounts of mental energy) drives us all to burnout. An amount of rest and recuperation in equal volume and type to our expenditures needs to be done regularly. So many of us ignore the daily, weekly, and monthly cycles of stress and rest that are necessary to work at peak optimal shape. Instead we push for days, week, or even months to try to reach a constantly moving goalpost with a promise that we will someday maybe take that vacation we need. This is only assisted by our 24/7/365 pagers and systems that take no rest, constantly waiting to fail or fall over from the ever present threat of bad actors or full hard drives.

I have watched pager fatigue alone completely destroy someone mentally. A poorly managed monitoring system that pages over things that aren't absolutely actionable and urgent or doesn't soft notify well enough in advance things that could be resolved before they become a critical issue is psychological abuse when delivered at the right volume. If someone can't disconnect because of the ever present pages that may or may not be actually actionable and critical then they are being slowly tortured, nerves frayed down with the rasp of their own phones.

At an even higher level I think we actively foster burnout amongst our peers and even ourselves with this great rockstar solo act so many of us pull. The concept of the Bastard Operator From Hell (BOFH) is the singlehanded "everything IT" person who has built everything from ground up and maintains everything even in the face of his users "always breaking everything". Because of this they become so jaded that they begin to torment their own end users and customers for mental relief. The worst part of this is we have formed a whole worship culture about being the lone gunman tech asshole to a point where I have seen a lot of people glorifying and emulating it well above and beyond their own time.

If we built up the proper support systems both mentally and technically we would be able to weather the storms the environment and systems rain down on us much better. If we worked together as much as possible instead of competing we would be well armed against the ever-present threat of burnout. It tends to be a lot to do with the personality types we pulled to technology for years; high on technical knowledge, low on social skills, lots of communication through digital means and not a lot of interpersonal interactions. Mix this in with the E/INTJ Type A personalities that are drawn to this higher stakes world of startups and high payoff companies and we develop this culture where we think everyone needs to be a rockstar or a ninja. We slim down staffing and just "hire the most brilliant mind in tech" to not just design but also implement and support these companies ad infinitum or more realistically, until they burn out, quit, or both.

I understand staffing is expensive and money is tight when your product is still only on the verge of success but in so much of operations and development we are paid to think, not to turn cogs. We design, develop, and foster ideas and solutions to problems no one else has solved. These kinds of ideas are not easily grown in a vacuum, but best cultivated through discourse and experimentation. However difficult it is to measure these expenditures or notice when they are getting strained there are ways we can approach them that helps identify issues faster as well as spread the mental load out more safely.

The best thing that ever happened to me in my experience in operations was learning to foster an interpersonal technical rapport with my co-workers and keeping it open. Constantly jogging ideas back and forth, never letting myself, or them stick on a problem and instead kicking it out to jog between us. I've done it twice and now it's a job requirement for me. The ability to "pair up" with operations to constantly foster and develop the most efficiently.

You see a lot of this starting to bud up in the tech world these days actually;

  • Open floor plans and chat based teams open up as much quick and easy discussion on issues as possible.
  • Code reviews are becoming the norm not just in development but in several forms of operations2.
  • The whole DevOps movement has large parts about enhancing communication and working together with others to help pool strengths and minimize weaknesses.
  • Some parts of agile/scrum are all about raising concerns and roadblocks as quickly as possible to put them up to the whole group, not trying to stick a single developer to solve a hard problem.
  • Pair programming is the next evolution of this, literally putting two minds to a single problem to solve it as efficiently and quickly as possible.

Even with all these trends though I still don't see enough of it, enough brainstorming, enough idea swapping, enough "Hey man I'm trying to do this like this but..." and that's why we are stuck staring at the flaws in every system. We work in our closed loops assuming that Issac Newton really did just sit under a tree staring at apples until he invented the Law of Gravitation so if we stare at our Apples and burn up all our mental energy the best way to handle this new package deploy will come to us. This is really where burnout comes from.

In closing I'd like to talk a little bit more about my mother. She's been a registered nurse her whole professional life. She's started in emergency rooms and for as long as I remember has been a hospice nurse. As you can imagine she's best at dealing with emergencies and the dying. Having dealt with the maimed and dying her whole life she's gone though mandatory on the clock grief therapy all of her professional life as well. In heath care when you deal with extreme emotional situations they regularly put you through therapy to help deal with and recover from what you work with. Whenever I ask someone who works about it it feels obvious to them that someone put though that level of emotional strain regularly would snap and do something horrible if they didn't receive regular emotional therapy. So as someone who goes through regular mental strain what are you doing for mental therapy to prevent your own burnout?


tl;dr

In my opinion burnout is a natural product of the tech industry culture. In order help combat this you need to do all of the following, which is not a panacea but a powerful preventive framework;

  • Work with others to help solve problems instead of trying to be a rockstar and solve them all yourself.
  • Create review processes around configuration and software changes so that you have reduced liability and risk.
  • Work with someone to develop intelligent alerting as well as a support rotation that allows rest and relaxation cycles as frequently as possible to reduce pager fatigue.
  • if you have critical failures more than once a quarter you need to review your infrastructure & procedures and ask yourself what your company is doing that's more important than having a reliable, usable product.
  • Rest yourself mentally every day, take breaks when you are locked in to a problem, and take regular vacations.
  • Bring in as much positivity to your workplace and your life as possible.
  • Fight negativity by analyzing it down to its root causes instead of superficially dismissing or accepting it.

And I hear you saying it already; "This isn't important right now", "I/we don't have the bandwidth for this at the moment", "I'm doing fine right now", or the worst "I'm not gonna burn out".

But know that when you do burn out I can promise you that you absolutely will not have the bandwidth or mental energy to do these things. By the time you are so far into burnout you actually become self aware of it deadlines will have slipped, everyone's already jaded and unhappy, and maybe you won't even care enough to implement this change.


  1. I have some amazing stories about Sybase databases running on a failover Sun Cluster using Veritas Volume Manager and the catastrophic problem with SCSI-3 reservations preventing disks to randomly not be mountable during a fail over event caused only when the LUNs are provided from newer NetApp hardware. Long story short, the database will start without half it's disks and it is not a pretty sight where it goes from there. 

  2. The third best thing I have ever done in my career so far was take all of a companies critical configurations (haproxy, dns, maintenance crons, backup scripts, monitoring, ect) and put them in git, created a code review process, and used jenkins to deploy to production. Call it CI/CD, call it change control, call it whatever you want; I call it almost never again making that one line stupid oops uh oh that breaks an entire system again. 

Being Happy

In my personal experience not everyone wants to be happy. When I say that I'm not talking about that one friend either. I'm talking about me. For a very long time I didn't wanna be happy. It was a very frustrating time. I can't say I knew I wanted to be unhappy, I just was all the time. I was angry, depressed, and off and on apathetic. It was actually my mother who coined the phrase "You just need to be unhappy." somewhere in my early to mid 20s.

Above and beyond my own internal strife one of my problems was that those around me were constantly interested in trying to make me happy. Maybe that doesn't sound like a problem to you but it was aggravating to me. It wasn't that people wanted to help it was this frustrating struggle of people offering what I saw as cookie cutter back pats and weak attempts to stoke a nonexistent ego instead of just facing the truth and seeing the world like I saw it. They didn't understand my problems. I thought about how miserable life is and how horrific this whole world works. I thought all the time about how my mediocre skill set in an over saturated field that shared a job pool with savants & geniuses meant I would going to toil and claw against an over abundant and under paid workforce. I thought about how compared to my peers my nothing-to-offer existence meant I get to writhe away in this world alone with a few cynical friends to cheerlead ourselves along to the grave. I looked at the ugly side of every story and trust me when you stop sugar coating everything you realize that there is millions of fucked up things going on every day and we do our best to gloss over it and put lipstick on this pig so that we can wake up and say today is an awesome day. This whole world is a corrupt Masque of the Red Death; an extravagant gala thrown by the privileged to hide away the social plague destroying everyone else around us.

Over time I learned how to communicate with others about my feelings in ways that helped prevent the constant fawning over my state. Of course I had to be highly selective of who I chose to associate with since I didn't want to have to go through this rigmarole constantly. I wasn't happy at that point but I reached this semi-content equilibrium where I got by with thick sense of dark humor and snark, as much as one possibly could exist in this sickness.

After a long while doing this I reached this weird nirvana where I was just me, and everyone was ok with it and I was ok with it to. I was just ok. Then the weird shit happened.

Through having this small social group, literally six to eight people deep, I was able to find this confidence in myself. Maybe what I could do was shit to anyone else but I could do things that made a difference to my friends, or at least impressed them. Over a period of about ten years give or take this grew, as well as my social circle. I often felt like the imposter in the room but through all these people I started to realize my own potential.

I can't say the exact moment it hit me. I know it was when I was working for Stephens. I had stuck my foot out enough times and somehow not gotten the door slammed on it enough times that I had made it somewhere in the company. The group I was running hadn't completely imploded on itself around me yet and I was a pretend famous DJ. Somewhere around this time it dawned on me that life isn't the Olympics but an Industry and even if I sucked the fact I wasn't going to stop trying made me valuable.

It was around that time I stopped reading fiction and switched to non-fiction. I read a lot of 90s-00s new era "be awesome at life" self help books and started implementing all these systems and tricks I read about.

My personal mantra around then was "It Never Hurts to Help"; a tongue in cheek reference to a cartoon from my youth called Eek the Cat. It was a morbid tale about an anthropomorphic feline whose overly sunny attitude and unflappable willingness to help other constantly ended him up in the hospital. That was his catch phrase, the one he said right before he was mauled by something. I like to say I was using it ironically since in the end I rarely caught fire after saying it but I did end up getting places professional and personally.

Needless to say life started moving really fast when I became truly motivated to help and get things done. I can't say I was happy... but I was really busy.

It was around then the shift really happened. I don't know if my attitude shifted first or those around me but things became nightmarishly disjointed and stressful at work over bad management, the club promoters I was working with went to war with another promotions group, and the community I was dealing with collapsed around the time someone slept with a minor unknowing and then someone else killed themselves. Needless to say these were dark times. However through all this stress I had a mantra and I stuck to it. Suddenly I was a too positive person for those I was around.

I was right back to where I was before in the reverse way. Everyone told me I need to "take it down a notch" and accused me of being disingenuous and sarcastic simply because I'm living my live the way I chose. I can't possibly feel like that, I can't do this, and we can't do that.

That's when my mantra changed to "I Only Have Cans". Just like before I had to adjust who is important in my life since I don't want to surround myself with those who are going to try to slow my progress and scowl at my outlook. This new mantra isn't just a tongue and cheek spite of never giving up and always helping. This one is only having the positive, always being able to do something. I guess that's when I decided to be happy?

I can't say I'm always happy. In fact I'd go so far as to say no one really stops dealing with depression. I still have the eternal funeral procession of self doubt, loathing, paranoia, and ill wishes flickering through my mind like an unending film. However I can decide it doesn't control me and I have way way better things to do with my life than be consumed by my own innate apathy. I can say that I'm in control of my outlook and what is important to me and I can control the world around me enough to decide I'm gonna be happy this day.

My mantra has been changing lately. I didn't have a mantra for over 26 years and now suddenly in eight years I have gone through three of them. Like I said, things started moving fast. It's not final, nothing is, but these days I'm sticking with "Be Fucking Amazing".

Not bad for someone who needed to be unhappy almost his entire life.

For those who actually read this far, I didn't write this to publicly stoke my dick at everyone or at least that wasn't the original intention. I've been thinking a lot lately about those around me who are unhappy now. I get a little sad and want to go make them happy which reminds me of where I stood not that long ago. I'm not going to pretend my story applies to anyone else and this should be shared around facebook by duck lip hotties as some overly winded it gets better back pat. However I'd like to hope that there are plenty of people who struggle with needing to be unhappy that will learn to take control of their own world and be fucking awesome in their own right.

Chef Frustrations

I've spent the last week working on implementing chef. The experience is frustrating to say the least. Instead of whining I wanted to take the time to write out some of my pain points and hopefully offer some constructive fixes to what I see as the wall in the learning curve.

Now to be clear up front. Most of my problems aren't with Chef, Ruby, or most of the core product; it's with implementing it. To be more precise I think the failure REALLY is documentation.

Anti-pattern One: Getting Started (into a corner)

Also known as the "Just enough to be dangerous but not useful" anti-pattern

I really liked the new learn chef. I have to give them a ton of credit for all the work but underneath all the new splash and presentation it's still the exact same old Chef 101 it was two years ago; it teaches you the barest of all basics and then drops you off to docs.opscode.com

I know that most would feel that statement isn't fair, since it teaches you all about the design and system behind how chef works, and that it does; but it still feels like not enough to be useful and here is why.

Anti-pattern Two: We Have no Patterns...

Learn Chef teaches you how chef works but not really how to use it at any level of scale; There is no real world usage taught anywhere. It teaches you to set up a Chef Enterprise server and then re-inventing the wheel with a homemade apache or ntp cookbook, and push it all to a vm but you would rarely do this in practice right?

When you leave Chef's documentation you learn about many very important Chef Patterns;

  • wrapper cookbooks
  • berkshelf way
  • one repo per cookbook vs monolithic repo
  • application cookbooks
  • service cookbooks

Why doesn't chef teach us these? Is this something we save for consultants to teach us at thousands of dollars an hour? Is it that Chef wants to avoid teaching patterns in order to remain as flexible as possible1?

It's not just chef either. Go to http://berkshelf.com and tell me how to use this tool assuming you've never done such before. If I was trying to remember a few commands or learn a new trick on top of something this tools docs would be great but it's missing the meat of what this tool is designed for and how to use it. A lot of chef's tools are treated this way.

Anti-pattern Three: ...So please learn everyone else's anti-patterns

This is my biggest frustration, OPD; Other People's Docs. As someone who has been working in Systems for 10+ years I have lived and learned so much from everyone else's blogs, which is why I feel the need to blog all my own lessons and information.

I feel that chef relies too much on OPD though. Especially because chef is such a fast moving target. It's amazing how many people who use chef that I talk to that use it in some odd, bizarre, and or generally 'not correct' way. It's usually because they learned a bad habit from a predecessor or found a bug in a long ago version and found some OPD that convinced them that "oh no you have to run everything chef-solo with your own special bootstraps, that is the ONE TRUE WAY™". I'm not saying that patten doesn't work but I doubt it's the best way for many infrastructures2.

I plan on documenting plenty of chef like things myself; in fact I plan on posting as much of my own OPD as possible but with how fast chef evolves as a product and with the large variance of methods for different environments I really hope people take everything with a grain of salt and read the date on the post when consitering my advice.

Here is a great example; where about 2014-07 I went into #chef and asked about some methods for setting things up and was linked to this blog which is treated like a defacto example of how to do things. But read all those updates... and then notice how it's using a lot of deprecated methods. I was linked to an article that could be titled "How to develop some really bad habits, but learn important things while you are at it." It's not Mischa's fault, It doesn't seem like he is a docs writer for Chef. Honestly I feel the best thing that could be done is this document be updated to the latest methodologies and tacked on to the end of learn chef as "One good method to get your enviroment up and going".

As a chef user do you even know about chef-dk? you probably should take a break from what you are doing, read this and then do this. Seriously don't you feel much better? This also should be on the end of learn chef guide. Hell this should probably be the first half of the learn chef guide.

I get that maybe they don't want to declare a "chef way" to do things... but at least give us some better hints.

Next Actions

Just to recap;

  • I believe chef's biggest weakness is documentation, which creates a wall in the learning curve to hit right after "I can now build and deploy a test apache on a linode" and "I can build and deploy this in a staging enviroment"
  • I think there should be a learn chef 200 series that goes over;
    • Using a wrapper cookbook, and the different types of abstraction you often see with these.
    • Teaching everything chef-dk adds; bootstrapping, runtests, and automated integration testing.
    • Highlighting several useful patterns for cookbook development.
    • Using more of chef's tools; ex ohai
  • If chef is going to rely on the community for docs maybe it should create a way where they can contribute to the main docbase just like they do code.
  • go here, have your life changed
  • If you are in the Las Vegas, NV area come hang out at #lvdevops on freenode and tell me how I make you feel
  • I'm going to spend another week or two trying diferent ways to structure my cookbooks and see what works.

  1. I believe this is a horrible anti-pattern in documentation. If you believe your power is flexibility then you should highlight that but still outline some predominate patterns for your top two or three use cases. 

  2. I know it's not the best way because they are deprecating chef-solo for chef-zero, which is good but it's a great example about the speed that Chef is changing. 

Tagged