BunnyMan.Info

Building chef-dk on FreeBSD 10

D.O.S. — Tue, 02 Aug 2016 21:23:00 -0700

For those that don’t know I’m a Chef for a living. Not the kind that works with food but one that works with code. What you may not know is I’m a FreeBSD guy, or at least claim to be one. I’ve been building a new FreeBSD workstation and discovered that there is no chef-dk for FreeBSD. Building it isn’t bad, but there is a trick to it.

So without further ado, here is building Chef-DK for FreeBSD 10.3 (and probably most >=10.0)

If this is a fresh box you need pkg, ports, and some base packages. All this assumes run as root.

pkg install pkg
portsnap fetch extract
portsnap fetch update
pkg install sudo

Ok, for the rest of this I am assuming you are running as a user that has sudo rights. If you aren’t then ymmv.

sudo pkg install ruby rubygem-bundler portdowngrade git
cd ~
sudo portdowngrade devel/gecode r345033
cd gecode
sudo make deinstall install clean
sudo sed -ie 's/\(#define GECODE_VERSION_NUMBER\)\s*/\1 300703/' /usr/local/include/gecode/support/config.hpp
cd ~
git clone https://github.com/chef/chef-dk.git
cd chef-dk
USE_SYSTEM_GECODE=1 bundle install --without development

There you go! It’s not going to give you the /opt/chef-dk omnibus but you will have all the chef-dk you need to do your stuff! Maybe later I’ll document how to make a package but this will likely work for me.

UPDATE 2017-01-17: Thanks to Shawn for the GECODE_VERSION_NUMBER update. This was broken several months after this post initally was written

BaaS: Burnout as a Service

D.O.S. — Mon, 02 May 2016 00:00:00 -0700

I wanna take a moment to to address what I like to call Burnout as a Service; how I see burnout as a product of the tech industry and culture. My friends, coworkers, and I have all experienced it in various levels, sometimes to crippling physical side effects. In this article I'm going to use strong language like need, and should, and I won't work with you if you [don't|do] $thing. While I have strong opinions about this subject and feel I have developed a powerful framework for helping avoid burnout it’s bound not to be perfect or complete. I do study this subject at great length in the name of personal development and productivity/energy management so if you have thoughts, opinions, feedback, or insights further into this topic I'd be happy to hear about them in the comments.

I'd like to point to this article, written by an amazing operations expert who has reached the jaded level where all things are approached solely by the perceived possible failure points, causing decision paralysis. Having lived the fear of doing for the failure it may cause down the road it I personally identify with this brand of burnout myself.

In my experience I have come to see burnout as a product of our tech industry's culture more than anything else. There is a lot of different little causes that all stack up on each other; the fast evolution of technology, the constant threat from bad actors, high stakes companies built on investors, product first revenue second business models, fickle customers caused by untested in market products, unheard of before uptime requirements on hastily developed systems, etc. The list could go on listing reasons why the industry itself is a primary source of maximum energy drain.

My mother is a healthcare worker who often asks why I make as much if not more than her working on computers. I often have to remind her that every moment, at work or not, I can be more or less responsible for the shut down of an entire company's revenue stream (possibly permanently) not just through negligence but from lack of anticipating the next failure or attack and guarding against it properly.

Let that sink in for a few moments. As an Operations Engineer or Developer not just my negligence but my lack of constant vigilance, research, and forethought can destroy an entire company. From bugs that open security holes, infrastructure mistakes that allow hidden-until-failure single points of failure, to a seemingly solid design choice that causes massive cascading failures in ways I never expected¹. Let's not even address what happens when someone makes an oops uh-oh and that backup you really need is coming back corrupt.

However all this is the nature of the industry we buy into often with the understanding of that this is how it works. I love having a job where mentally jogging day in and day out to keep pace or even get ahead of developing technology and security vectors. This is my great mental stimulation and when I tire of it I'll go pour drinks at a bar.

However we all need to both expend and recover energy in equal amounts. This is a core biological imperative that we don't often think about. Our desire to always be creating (which expends great amounts of mental energy) drives us all to burnout. An amount of rest and recuperation in equal volume and type to our expenditures needs to be done regularly. So many of us ignore the daily, weekly, and monthly cycles of stress and rest that are necessary to work at peak optimal shape. Instead we push for days, week, or even months to try to reach a constantly moving goalpost with a promise that we will someday maybe take that vacation we need. This is only assisted by our 24/7/365 pagers and systems that take no rest, constantly waiting to fail or fall over from the ever present threat of bad actors or full hard drives.

I have watched pager fatigue alone completely destroy someone mentally. A poorly managed monitoring system that pages over things that aren't absolutely actionable and urgent or doesn't soft notify well enough in advance things that could be resolved before they become a critical issue is psychological abuse when delivered at the right volume. If someone can't disconnect because of the ever present pages that may or may not be actually actionable and critical then they are being slowly tortured, nerves frayed down with the rasp of their own phones.

At an even higher level I think we actively foster burnout amongst our peers and even ourselves with this great rockstar solo act so many of us pull. The concept of the Bastard Operator From Hell (BOFH) is the singlehanded "everything IT" person who has built everything from ground up and maintains everything even in the face of his users "always breaking everything". Because of this they become so jaded that they begin to torment their own end users and customers for mental relief. The worst part of this is we have formed a whole worship culture about being the lone gunman tech asshole to a point where I have seen a lot of people glorifying and emulating it well above and beyond their own time.

If we built up the proper support systems both mentally and technically we would be able to weather the storms the environment and systems rain down on us much better. If we worked together as much as possible instead of competing we would be well armed against the ever-present threat of burnout. It tends to be a lot to do with the personality types we pulled to technology for years; high on technical knowledge, low on social skills, lots of communication through digital means and not a lot of interpersonal interactions. Mix this in with the E/INTJ Type A personalities that are drawn to this higher stakes world of startups and high payoff companies and we develop this culture where we think everyone needs to be a rockstar or a ninja. We slim down staffing and just "hire the most brilliant mind in tech" to not just design but also implement and support these companies ad infinitum or more realistically, until they burn out, quit, or both.

I understand staffing is expensive and money is tight when your product is still only on the verge of success but in so much of operations and development we are paid to think, not to turn cogs. We design, develop, and foster ideas and solutions to problems no one else has solved. These kinds of ideas are not easily grown in a vacuum, but best cultivated through discourse and experimentation. However difficult it is to measure these expenditures or notice when they are getting strained there are ways we can approach them that helps identify issues faster as well as spread the mental load out more safely.

The best thing that ever happened to me in my experience in operations was learning to foster an interpersonal technical rapport with my co-workers and keeping it open. Constantly jogging ideas back and forth, never letting myself, or them stick on a problem and instead kicking it out to jog between us. I've done it twice and now it's a job requirement for me. The ability to "pair up" with operations to constantly foster and develop the most efficiently.

You see a lot of this starting to bud up in the tech world these days actually;

Open floor plans and chat based teams open up as much quick and easy discussion on issues as possible.
Code reviews are becoming the norm not just in development but in several forms of operations².
The whole DevOps movement has large parts about enhancing communication and working together with others to help pool strengths and minimize weaknesses.
Some parts of agile/scrum are all about raising concerns and roadblocks as quickly as possible to put them up to the whole group, not trying to stick a single developer to solve a hard problem.
Pair programming is the next evolution of this, literally putting two minds to a single problem to solve it as efficiently and quickly as possible.

Even with all these trends though I still don't see enough of it, enough brainstorming, enough idea swapping, enough "Hey man I'm trying to do this like this but..." and that's why we are stuck staring at the flaws in every system. We work in our closed loops assuming that Issac Newton really did just sit under a tree staring at apples until he invented the Law of Gravitation so if we stare at our Apples and burn up all our mental energy the best way to handle this new package deploy will come to us. This is really where burnout comes from.

In closing I'd like to talk a little bit more about my mother. She's been a registered nurse her whole professional life. She's started in emergency rooms and for as long as I remember has been a hospice nurse. As you can imagine she's best at dealing with emergencies and the dying. Having dealt with the maimed and dying her whole life she's gone though mandatory on the clock grief therapy all of her professional life as well. In heath care when you deal with extreme emotional situations they regularly put you through therapy to help deal with and recover from what you work with. Whenever I ask someone who works about it it feels obvious to them that someone put though that level of emotional strain regularly would snap and do something horrible if they didn't receive regular emotional therapy. So as someone who goes through regular mental strain what are you doing for mental therapy to prevent your own burnout?

tl;dr

In my opinion burnout is a natural product of the tech industry culture. In order help combat this you need to do all of the following, which is not a panacea but a powerful preventive framework;

Work with others to help solve problems instead of trying to be a rockstar and solve them all yourself.
Create review processes around configuration and software changes so that you have reduced liability and risk.
Work with someone to develop intelligent alerting as well as a support rotation that allows rest and relaxation cycles as frequently as possible to reduce pager fatigue.
if you have critical failures more than once a quarter you need to review your infrastructure & procedures and ask yourself what your company is doing that's more important than having a reliable, usable product.
Rest yourself mentally every day, take breaks when you are locked in to a problem, and take regular vacations.
Bring in as much positivity to your workplace and your life as possible.
Fight negativity by analyzing it down to its root causes instead of superficially dismissing or accepting it.

And I hear you saying it already; "This isn't important right now", "I/we don't have the bandwidth for this at the moment", "I'm doing fine right now", or the worst "I'm not gonna burn out".

But know that when you do burn out I can promise you that you absolutely will not have the bandwidth or mental energy to do these things. By the time you are so far into burnout you actually become self aware of it deadlines will have slipped, everyone's already jaded and unhappy, and maybe you won't even care enough to implement this change.

I have some amazing stories about Sybase databases running on a failover Sun Cluster using Veritas Volume Manager and the catastrophic problem with SCSI-3 reservations preventing disks to randomly not be mountable during a fail over event caused only when the LUNs are provided from newer NetApp hardware. Long story short, the database will start without half it's disks and it is not a pretty sight where it goes from there. ↩
The third best thing I have ever done in my career so far was take all of a companies critical configurations (haproxy, dns, maintenance crons, backup scripts, monitoring, ect) and put them in git, created a code review process, and used jenkins to deploy to production. Call it CI/CD, call it change control, call it whatever you want; I call it almost never again making that one line stupid oops uh oh that breaks an entire system again. ↩

Being Happy

D.O.S. — Wed, 30 Dec 2015 00:00:00 -0800

In my personal experience not everyone wants to be happy. When I say that I'm not talking about that one friend either. I'm talking about me. For a very long time I didn't wanna be happy. It was a very frustrating time. I can't say I knew I wanted to be unhappy, I just was all the time. I was angry, depressed, and off and on apathetic. It was actually my mother who coined the phrase "You just need to be unhappy." somewhere in my early to mid 20s.

Above and beyond my own internal strife one of my problems was that those around me were constantly interested in trying to make me happy. Maybe that doesn't sound like a problem to you but it was aggravating to me. It wasn't that people wanted to help it was this frustrating struggle of people offering what I saw as cookie cutter back pats and weak attempts to stoke a nonexistent ego instead of just facing the truth and seeing the world like I saw it. They didn't understand my problems. I thought about how miserable life is and how horrific this whole world works. I thought all the time about how my mediocre skill set in an over saturated field that shared a job pool with savants & geniuses meant I would going to toil and claw against an over abundant and under paid workforce. I thought about how compared to my peers my nothing-to-offer existence meant I get to writhe away in this world alone with a few cynical friends to cheerlead ourselves along to the grave. I looked at the ugly side of every story and trust me when you stop sugar coating everything you realize that there is millions of fucked up things going on every day and we do our best to gloss over it and put lipstick on this pig so that we can wake up and say today is an awesome day. This whole world is a corrupt Masque of the Red Death; an extravagant gala thrown by the privileged to hide away the social plague destroying everyone else around us.

Over time I learned how to communicate with others about my feelings in ways that helped prevent the constant fawning over my state. Of course I had to be highly selective of who I chose to associate with since I didn't want to have to go through this rigmarole constantly. I wasn't happy at that point but I reached this semi-content equilibrium where I got by with thick sense of dark humor and snark, as much as one possibly could exist in this sickness.

After a long while doing this I reached this weird nirvana where I was just me, and everyone was ok with it and I was ok with it to. I was just ok. Then the weird shit happened.

Through having this small social group, literally six to eight people deep, I was able to find this confidence in myself. Maybe what I could do was shit to anyone else but I could do things that made a difference to my friends, or at least impressed them. Over a period of about ten years give or take this grew, as well as my social circle. I often felt like the imposter in the room but through all these people I started to realize my own potential.

I can't say the exact moment it hit me. I know it was when I was working for Stephens. I had stuck my foot out enough times and somehow not gotten the door slammed on it enough times that I had made it somewhere in the company. The group I was running hadn't completely imploded on itself around me yet and I was a pretend famous DJ. Somewhere around this time it dawned on me that life isn't the Olympics but an Industry and even if I sucked the fact I wasn't going to stop trying made me valuable.

It was around that time I stopped reading fiction and switched to non-fiction. I read a lot of 90s-00s new era "be awesome at life" self help books and started implementing all these systems and tricks I read about.

My personal mantra around then was "It Never Hurts to Help"; a tongue in cheek reference to a cartoon from my youth called Eek the Cat. It was a morbid tale about an anthropomorphic feline whose overly sunny attitude and unflappable willingness to help other constantly ended him up in the hospital. That was his catch phrase, the one he said right before he was mauled by something. I like to say I was using it ironically since in the end I rarely caught fire after saying it but I did end up getting places professional and personally.

Needless to say life started moving really fast when I became truly motivated to help and get things done. I can't say I was happy... but I was really busy.

It was around then the shift really happened. I don't know if my attitude shifted first or those around me but things became nightmarishly disjointed and stressful at work over bad management, the club promoters I was working with went to war with another promotions group, and the community I was dealing with collapsed around the time someone slept with a minor unknowing and then someone else killed themselves. Needless to say these were dark times. However through all this stress I had a mantra and I stuck to it. Suddenly I was a too positive person for those I was around.

I was right back to where I was before in the reverse way. Everyone told me I need to "take it down a notch" and accused me of being disingenuous and sarcastic simply because I'm living my live the way I chose. I can't possibly feel like that, I can't do this, and we can't do that.

That's when my mantra changed to "I Only Have Cans". Just like before I had to adjust who is important in my life since I don't want to surround myself with those who are going to try to slow my progress and scowl at my outlook. This new mantra isn't just a tongue and cheek spite of never giving up and always helping. This one is only having the positive, always being able to do something. I guess that's when I decided to be happy?

I can't say I'm always happy. In fact I'd go so far as to say no one really stops dealing with depression. I still have the eternal funeral procession of self doubt, loathing, paranoia, and ill wishes flickering through my mind like an unending film. However I can decide it doesn't control me and I have way way better things to do with my life than be consumed by my own innate apathy. I can say that I'm in control of my outlook and what is important to me and I can control the world around me enough to decide I'm gonna be happy this day.

My mantra has been changing lately. I didn't have a mantra for over 26 years and now suddenly in eight years I have gone through three of them. Like I said, things started moving fast. It's not final, nothing is, but these days I'm sticking with "Be Fucking Amazing".

Not bad for someone who needed to be unhappy almost his entire life.

For those who actually read this far, I didn't write this to publicly stoke my dick at everyone or at least that wasn't the original intention. I've been thinking a lot lately about those around me who are unhappy now. I get a little sad and want to go make them happy which reminds me of where I stood not that long ago. I'm not going to pretend my story applies to anyone else and this should be shared around facebook by duck lip hotties as some overly winded it gets better back pat. However I'd like to hope that there are plenty of people who struggle with needing to be unhappy that will learn to take control of their own world and be fucking awesome in their own right.

Quick Note on GnuRadio on Pentoo

D.O.S. — Sun, 06 Sep 2015 15:03:00 -0700

Not a big blog, but a quick problem I got solved on IRC that I thought might help others.

I have a Gateway LT4009u with an Atom N2600. It's my "hacker/workshop" laptop. The atom N processors are a bit gimpy so sometimes things don't run right. One thing is GNURadio on Pentoo. Pentoo runs hardened and this pisses off the atom n.

So if you get the following error.

LLVM ERROR: Allocation failed when allocating new memory in the JIT
Can't Allocate RWX Memory: Operation not permitted

Then you need to soft disable hardened with the following command

sudo toggle_hardened

I hope that helps anyone else on the internet.

Thanks to Zero_Chaos in #pentoo on irc.freenode.net for the fix (and pentoo)

Quick Update: This also happens when running in VirtualBox 5 on my 2015 MacBook i7, but the fix is the same

Monitoring Chef runs without Chef

D.O.S. — Fri, 24 Apr 2015 13:52:00 -0700

I, like many sysadmins, really want to monitor all the things I actually care about. Monitoring is in general hard. Not because it’s hard to set up, but it’s hard to get right. It’s really easy to monitor ALL THE THINGS and then just end up with pager fatigue. It’s all about figuring out what you need to know and when you need to know it.

So in this case I really need to know that my machines are staying in compliance with chef.

There was a few ways you can do this. The first thought I had was adding a hook into all of my runs and having them report in on failure. This is mostly because I’m always looking for another way to hack on Chef and work on my ruby. The big problem with this is:

What if the node is offline?
What if the cron doesn’t fire?
What if chef/or ruby is so borked it can’t even fire the app
What if someone disabled chef

I need a better solution

Knife Status

Knife status is just awesome, it has some awesome flags and generally I run it far more than I should. The great part about this query the server approach is that it lets me know;

The server is still happy and spitting out cookbooks to nodes
The status of ALL of my runs from the “source of truth” for runs

Not making my chef test rely on chef

But I’m not going to shell knife status. I’m a damn code snob and something about having the chef test rely on the chef client status didn’t seem right.

Instead I wrote a nagios script that I am not going to share in it’s entirety here because $WORK_CODE¹… insert sad face but I will tell you exactly how I did it.

How to python your chef, or how I stopped worrying and learned to love that I can still use python to do anything.

I’m the most experienced in python and almost all of our internal nagios checks we have written in python. So this is in python.

Step one

Use pynagioscheck and pychef. Seriously. Don’t reinvent the wheel here.

Step two

Create a knife object. have it take all your settings on initialize, then you can create functions for all the different knife commands to recreate them with pychef.

You really only need status for this one. The meat of status is this here, coderanger dropped this on me in IRC

for row in chef.Search('node', '*:*'):
    nodes[row.object['machine name']] = datetime.fromtimestamp(row.object['ohai_time'])

Step three

Now from here I created a TimeChecker object. It takes the dictionary of { server: datetimeObj } on it’s init. For consistency sake I also init self.now = datetime.now(). Then I have a TimeChecker.runs_not_in_the_last() that just takes an int.

The magic of runs_not_in_the_last I will also share with you because I’m proud of this damn script and want to share it with the world

diff = timedelta(hours=hours)
return [k for k in self.runtimes.keys() if self.now - self.runtimes[k] > diff]

Bam!

Step four

Now just extend NagiosCheck with KnifeStatusCheck, make all your options and other goods in your init and then make your check()

In the check you make knife, Make a Timechecker with the status return… then all you have to do is see if you have any runs_not_in_the_last for critical and then warning.

Gotchas and cleanup notes

USE EXCEPTIONS

seriously, this can and will make them so catch them properly and return errors. You will need to catch and handle AT LEAST - URLError - Status - UsageError - ChefError - At least two of your own exceptions

SSL errors

So there is no trusted_certs here. You need to either give your server a working cert, install the snake oil into the nagios server as acceptable or do the dirtiest of monkey patches.

# Dirty Monkeypatch
if sys.version_info >= (2, 7, 9):
    import ssl
    ssl._create_default_https_context = ssl._create_unverified_context

But before you do this think of the children!!!

Weird ass errors with join

I need to maybe open a ticket and patch pynagioscheck but I had the weirdest bug when raising a critical. It would die in the super’s check on “”.join(bt) or something of the ilk.

My work around was to not just pass msg to the Status exception but to make msg a list and put the main message in msg[0] and then put the comma joined list of servers out of compliance in msg[1]. This means the standard error comes up on normal returns but if you run the check with -v it will give you a list of servers out of compliance for troubleshooting or debugging. Not bad.

Handling the pem file

Eeeeehhhh This maybe my one cop out in the whole script. Basically I created a nagios user in chef with a insane never to be used again and promptly lost password and put the nagios.pem file alongside the check script. Then I let the script optionally take a pem name, and it just checks that the pemfile is alongside the check script. I was considering letting you specify a pem script somewhere on the server or in the Nagios’s users home directory but decided to bite that and take the simplest route there.

Don’t destroy your nagios server

Seriously. Did you see this code? Run a search on all nodes and then return an attribute for every node in your nagios server. This is not the worlds fastest check script.

Unless you dedicate some serious power to your solr service on your chef server you should make sure to only check this service once every ten minutes tops. I only check once an hour normally and then follow up with 10 minute checks on fail on my server since I only do converges every four hours so an “out of compliance” warning for me would be at the 12 hour mark and critical at 24 hours².

I don’t yet have any clearance to post or share anything I write for, while, at, or around work. The company owns all that, but we are currently working on getting to the point where we can share some stuff. Especially things not so related to our IP like infrastructure code, cookbook, checks, ect. ↩
The reason I picked these numbers is I don’t want to know the FIRST time a converge fails. I use the omnibus_updater in my runs (Pinned version in attributes of course) so a failed run can be normal. Plus I am deploying something that important I am going to spot check runs and verify everything gets run with knife ssh. I just want to know mostly if a machine is out of the loop for more than a day because that’s a node that needs to get shot. ↩

An Open Year

D.O.S. — Fri, 17 Apr 2015 17:15:00 -0700

It's been about a year since my last post, mostly frustrated with Chef as a beginner. Now I spend most of my day writing cookbooks and recipes. In fact I am even helping the Lead Dev at work learn Chef and got back from Chef conference. There I met a lot of amazing people and even offered to help maintain BSD support in chef.

This post isn't about that so much. It's mostly about a behavior I noticed I picked up. When I worked for Stephens Media I spent a lot of my energy trying to contribute, in posts, open source, pull requests, ect. Then when I moved to Slickdeals.net my time was really sucked up. I drifted from working on Pelican and stopped doing as many pull requests. At some time I set up a personally hosted Stash instance. Then I locked that stash instance off behind a login. Then I started writing in my private confluence instead of here. Now all my projects these days are All Rights I noticed... hmph.

I don't know exactly what triggered this sharephobia but it needs to stop. I almost think it's some weird greed involving my personal time and effort but if I was greedy wouldn't I want people fixing up my code for me? Is there some revolutionary private research in all this that makes me more valuable? I think showing off my abilities and progress makes me more valuable.

I'm just currently working on pulling all my code out of my stash and putting it onto github, with a much better BSD license. I'm remembering what the subtitle of my blog really means.

I've spent a lot of time studying Ruby since I finished my DBA course. There is still a lot of areas where Chef could use improvements and I plan to do a lot about it. We are going to make BSD a first class citizen with Chef and hopefully many of it's tools and cookbooks too.¹

Remember when I used to post monthly? Hahahaha. I don't want to use this as a journal, I already have one of those but I wanted to give a bigger picture life update since I am updating pages and testing my jenkins build trigger with github ;p

I have always preferred UNIX to Linux. My first sysadmin job was a Solaris Admin, a job I did for a long time. With the advent of SystemD I've gone back to my love in the form of BSD. ↩

Chef Frustrations

D.O.S. — Mon, 14 Jul 2014 18:24:00 -0700

I've spent the last week working on implementing chef. The experience is frustrating to say the least. Instead of whining I wanted to take the time to write out some of my pain points and hopefully offer some constructive fixes to what I see as the wall in the learning curve.

Now to be clear up front. Most of my problems aren't with Chef, Ruby, or most of the core product; it's with implementing it. To be more precise I think the failure REALLY is documentation.

Anti-pattern One: Getting Started (into a corner)

Also known as the "Just enough to be dangerous but not useful" anti-pattern

I really liked the new learn chef. I have to give them a ton of credit for all the work but underneath all the new splash and presentation it's still the exact same old Chef 101 it was two years ago; it teaches you the barest of all basics and then drops you off to docs.opscode.com

I know that most would feel that statement isn't fair, since it teaches you all about the design and system behind how chef works, and that it does; but it still feels like not enough to be useful and here is why.

Anti-pattern Two: We Have no Patterns...

Learn Chef teaches you how chef works but not really how to use it at any level of scale; There is no real world usage taught anywhere. It teaches you to set up a Chef Enterprise server and then re-inventing the wheel with a homemade apache or ntp cookbook, and push it all to a vm but you would rarely do this in practice right?

When you leave Chef's documentation you learn about many very important Chef Patterns;

wrapper cookbooks
berkshelf way
one repo per cookbook vs monolithic repo
application cookbooks
service cookbooks

Why doesn't chef teach us these? Is this something we save for consultants to teach us at thousands of dollars an hour? Is it that Chef wants to avoid teaching patterns in order to remain as flexible as possible¹?

It's not just chef either. Go to http://berkshelf.com and tell me how to use this tool assuming you've never done such before. If I was trying to remember a few commands or learn a new trick on top of something this tools docs would be great but it's missing the meat of what this tool is designed for and how to use it. A lot of chef's tools are treated this way.

Anti-pattern Three: ...So please learn everyone else's anti-patterns

This is my biggest frustration, OPD; Other People's Docs. As someone who has been working in Systems for 10+ years I have lived and learned so much from everyone else's blogs, which is why I feel the need to blog all my own lessons and information.

I feel that chef relies too much on OPD though. Especially because chef is such a fast moving target. It's amazing how many people who use chef that I talk to that use it in some odd, bizarre, and or generally 'not correct' way. It's usually because they learned a bad habit from a predecessor or found a bug in a long ago version and found some OPD that convinced them that "oh no you have to run everything chef-solo with your own special bootstraps, that is the ONE TRUE WAY™". I'm not saying that patten doesn't work but I doubt it's the best way for many infrastructures².

I plan on documenting plenty of chef like things myself; in fact I plan on posting as much of my own OPD as possible but with how fast chef evolves as a product and with the large variance of methods for different environments I really hope people take everything with a grain of salt and read the date on the post when consitering my advice.

Here is a great example; where about 2014-07 I went into #chef and asked about some methods for setting things up and was linked to this blog which is treated like a defacto example of how to do things. But read all those updates... and then notice how it's using a lot of deprecated methods. I was linked to an article that could be titled "How to develop some really bad habits, but learn important things while you are at it." It's not Mischa's fault, It doesn't seem like he is a docs writer for Chef. Honestly I feel the best thing that could be done is this document be updated to the latest methodologies and tacked on to the end of learn chef as "One good method to get your enviroment up and going".

As a chef user do you even know about chef-dk? you probably should take a break from what you are doing, read this and then do this. Seriously don't you feel much better? This also should be on the end of learn chef guide. Hell this should probably be the first half of the learn chef guide.

I get that maybe they don't want to declare a "chef way" to do things... but at least give us some better hints.

Next Actions

Just to recap;

I believe chef's biggest weakness is documentation, which creates a wall in the learning curve to hit right after "I can now build and deploy a test apache on a linode" and "I can build and deploy this in a staging enviroment"
I think there should be a learn chef 200 series that goes over;
- Using a wrapper cookbook, and the different types of abstraction you often see with these.
- Teaching everything chef-dk adds; bootstrapping, runtests, and automated integration testing.
- Highlighting several useful patterns for cookbook development.
- Using more of chef's tools; ex ohai
If chef is going to rely on the community for docs maybe it should create a way where they can contribute to the main docbase just like they do code.
go here, have your life changed
If you are in the Las Vegas, NV area come hang out at #lvdevops on freenode and tell me how I make you feel
I'm going to spend another week or two trying diferent ways to structure my cookbooks and see what works.

I believe this is a horrible anti-pattern in documentation. If you believe your power is flexibility then you should highlight that but still outline some predominate patterns for your top two or three use cases. ↩
I know it's not the best way because they are deprecating chef-solo for chef-zero, which is good but it's a great example about the speed that Chef is changing. ↩

Where have you been?

D.O.S. — Sun, 13 Jul 2014 00:00:00 -0700

Actually I should have called this where have I been. This seemed catchier though.

In short the answer is I have been at my new job at http://slickdeals.net They have keep me as busy as can be. Because of this major shift I changed a lot of habits; I stopped writing here as much, I stopped actively contributing to Pelican¹, and I also stopped posting to github.

Be not too afraid. I still write plenty of code and fille around with everything. I just stopped posting most of it to github. I mostly use my own private stash instance. There is some publically accessable code for those who are interested. The reason is that stash is SIGNIFICANTLY cheaper for my private repos so I just put most of my stuff there. I have been rethinking that lately since I miss a lot of the github community style.

I haven't been blogging much since I have just been focusing most of my documentation and writings to work, and got in the odd habit of putting eveything in my private wiki. For the sake of sharing and hashing things out I am going to refocus on using this blog to document out useful things. Hopefully this means you cna excpect floods of useful things. Maybe some smaller posts too.

On the other side I have been busy with my girlfriend, cycling, dba courseware, studying Japnaese, and my unstoppable anime habit. I have my own house now and a workbench set back up so I may blog about those projects as well...

Anyways. No one loves a vanity post but I wanted to at least put an update out

I'm really sad about this but my personal time dropped sharply between a new job and new girlfriend. ↩

Coming back to vim

D.O.S. — Thu, 11 Apr 2013 21:53:00 -0700

It's time for my monthly or so post! I wanted to go through and post about my OpenBSD firewall I built but that's not 100%. Also I'm not ready to go on about anything amazing with puppet because without my lab being done puppet isn't useful so lets go back to talking about my dev environment!

I know Justin has been asking for this for a little while.

Preface: Going "back" to vim

As a sysadmin at work I use vi a lot. Not even vim; vi. We have lots of unix boxes that default to vi as the installed editor and we don't just go installing vim on everything. Personally I use vim a good amount on my machine since I spend a lot of command line time anyways. I know more than just a few of the commands but I really only consider myself a second or maybe third year vim user¹ since I never used it full time to write code. I live the motion and use things like ci[ and C-v 5j x but I still fail to use multiple registers, buffers, or tabs… or even the leader commands.

I'm an amateur software developer at best; I have serious aspirations about seeing if I have the chops to go pro but right now I'm honing edges. Irregardless of how developer or not I may be, I am developer lazy so I spend money on tools that make my life easier. I've been using PyCharm to help me write utilities and my mini apps and it's just the best. Sometimes I worry about leaning on IDE tools stunting my abilities so I took some time a month back to stand up and step back from PyCharm and instead just use vim…

This is the setup.

Part One: My keyboard

I use a slick trick on my Mac so I have no access to Caps and my CapsLock key acts as BOTH a Ctrl and Escape. If I tap the Caps it's esc, if I chord it with anything else it registers Ctrl… I pinkie reach for nothing. Here is the the instructions, 10.8 approved so YMMV for other versions.

Go to System Preferences -> Keyboard -> Modifier Keys. Set Caps Lock to ^ Control.
Install KeyRemap4MacBook - Software for OS X.
In KeyRemap4MacBook, enable Control_L to Control_L (+ when you type Control_L only, send Escape. Search will help.
Reboot and enjoy.

Part Two: The Development Server.

You didn't think I was just going to vim run.py and take off did you? No.

If I'm going to work from the shell I want to make it so I can work from anywhere while I am at it. I have a Mac OS X server that would love to be my dev box so away I go. Open some ports, SSH keys, virtualenv, python3 from brew… tada! But it's not ready yet.

OpenSSH is my best friend. I keep keys close at all times and use cools scripts on my laptop to help manage them. However SSH is not enough and this is where Mosh: the mobile shell comes in. Mosh isn't a total end to end transport solution but it's high speed udp style and local echo features make it supreme when then connection starts lagging and you don't want it to slow down your code. Best yet? brew install mobile-shell on both boxes… done…

If only we had a windows client already.

If you want some portable keys help check out the following;

Just remember to not make these your ONLY keys, all posable keys should be password encoded and easily revokable so keep a backup and list of your emergency to revoke when it gets lost.

Part Three: The Terminal

I need a sweet terminal so I use zsh with oh-my-zsh and a while bunch of personal mods. Remember the whole lazy part? Yes. Here is the highlights of my zsh configs;

I have my .zsh run workon at the end to list all my virtenvs. It's a nice nag
I use ^p and ^n for fuzzy history find and it's amazing dotfiles/zsh/keybindings.zsh at master · onlyhavecans/dotfiles · GitHub
vi-mode!!!!!

The second major part is tmux. Whatever you are using now… drop it and use tmux. I remapped all my common tmux commands to vi-mode style and C-a for my leader because now ctrl and a are touching. For the full list of my configs which I won't get too deep into check out tmux.conf at master · onlyhavecans/dotfiles · GitHub.

My main tmux window generally looks like this

     -----------------------
    |    Chat   |           |
    |     or    |           |
    |extra shell|    VIM    |
    |-----------|           |
    |           |           |
    |    IRC    |-----------|
    |           | MiniShell |
    =========================

Chat is my flux buffer that gets changed between a personal chat and second work buffer. IRC is my ever present wee-chat connection. The mini shell is a little shell I keep in the same dir as vim so I can quick run python -m unittest discover module over and over or whatever. When I'm playing with Flask it's running there. Depending on where my focus is the vertical split is usually about 65% weighted to the work to squish distractions without cutting them all out or I am on the 11' MacBook Air screen instead of a 20+' external display.

I often have a second window but the latest version added C-a z for window zoom and that's GREAT when I really wanna focus on something or blow up the mini-shell while I am debugging something.

Part Four: Into VIM

First and foremost I keep an 8.5x11 copy of Beautiful Vim Cheat-Sheet Poster & Printable Downloads on my desk. It's a nice way to keep reminding me of all the features I NEED to be using and if you don't want to give someone 10USD for it there is a free link right on the page for a low res.

tl;dr the configs

dotfiles/vim at master · onlyhavecans/dotfiles · GitHub

A lot of my inital vim config like most was stolen from somewhere but over time I have stripped out everything I didn't adapt in. I started with YADR and then seriously hacked it to death. In the end there is still yadr references but you shouldn't take anything that claims being from yadr in there still is. I'm just lazy about renaming files for scuz.

Highlights

Setting toggle line numbers to F2 & toggle NERDTree to F3 is heavenly.
I use Solarized - Ethan Schoonover and the plugin is great
I use a lot of markdown but if you use plasticboy/vim-markdown · GitHub you need to set an extra setting to kill that damn folding
klen/python-mode · GitHub is just amazing and I only use a little of it
I really miss snippits but I am not yet ready to dive into $SNIPMANAGER-X
I'm just now REALLY getting used to the power of scrooloose/nerdtree · GitHub
I just got into using Lokaltog/powerline · GitHub. it's really slick looking but I'm really not sold on a second status line
I'd love to use joonty/vdebug · GitHub more but I haven't gotten it to work yet

There you go! I know it feels like I'm skimming the VIM part of the vim writeup but there is really only so much you can do TO VIM itself. It's the development environment you put around it and what you put out with it. Hopefully I will be putting out great things once I learn how to use tags and rope and all that other stuff to get back to ultra fast code sifting and editing.

Part Five: Wishlist

Auto-running tests

Just something that PyCharm and Komono before that spoiled me on. :w running my tests since I very often TDD would save a lot of window jumping

Snippits

I'll never get deep code intelligence with vim and that's kinda the point but PyCharm saved somewhere around a billion keystrokes when you learned when to hit the auto complete right.

Part Six: Warnings

This allows me to do some awesome stuff and so far I am happy with with it outside of a few small caveats.

It's damn fiddly. So much and learn and fiddle with distracts from the work.
Sometimes cruising around inside of vim, inside of tmux can make for some finger dancing that I don't care for; C-a l C-a k C-w l… until I trip over my own keystrokes. C-a ; is really useful when popping between panes when it comes to mind

Part Seven: Going forward

I can connect in technically from anything I trust enough to plug my key jump-drive into. I currently have bought a YubiKey and am seriously considering switching right over to two-factor OTP which makes me LESS afraid of plugging in the key into something.

Another area I am considering going forward with is I technically won't even need my computer to work. I could just work with my iPad and a keyboard! I'm really sure these articles had just a little bit to do with my idea of moving over to all command line vim. I don't know if I am there but it is tempting;

Was this more in depth than you expected? Do you want more? Lemme know.

See Just Use Sublime Text - Andrew Ray's Github Blog for details on what I mean by that. ↩

SSH Keys on a USB jump drive on Mac OS X Part 2

D.O.S. — Thu, 07 Mar 2013 07:50:00 -0800

All right! You read my post SSH Keys on a USB jump drive on Mac OS X (Mountain Lion Edition) and want to take this to the next level huh? Maybe having your config posting back to a jump drive and having to have it plugged in every time you want to log into something is sooooo lame! You often just leave it there, plugged in when you walk away from your work station. There has to be a better way…

I got you bro

What you need to be doing is adding your keys to ssh-agent on demand then pulling your key except for when you REALLY need it. But how do you do that easily?

I got you bro

Windows

Lets start with the basics. PuTTY Download Page - Pageant

Download it.
Install it.
Yes you have to use their crappy ppk so set that up.
Start Pageant then load your ppk version of your key.
Eject drive.
open new session, don't bother manual adding key.
Party time
Excellent

The cool part is that Pageant remembers your keys so if you pin it to the start menu then it's highlight, over enter, password… you get the point.

The shitty part is it keeps that key loaded until you reboot. That's ULTRA fucking weak. Sorry windows. If anyone knows better give me a shout out.

Linux

Linux is easier and way better. Lets assume you do everything in the command line so.

alias loadkey=ssh-add -t 30m /mnt/whatever/ssh/$1

then all you have to do is

loadkey keyname

then POW!!! for 30 minutes you have you key loaded. See the 30m in the command? Change that for maximum moddage of commands. Are you a lazy brogrammer slash skriptkiddie and need the time format table?

Bro… got… you are… by me

<none>  seconds
s | S   seconds
m | M   minutes
h | H   hours
d | D   days
w | W   weeks

Time format examples:
600     600 seconds (10 minutes)
10m     10 minutes
1h30m   1 hour 30 minutes (90 minutes)

Damn! Yea….

Mac

Ok, this is where the gold is. I slaved over a hot mess of applescript to you guys this so feel the love bro. Since it's my Mac and I wanted key exchange to be as easy as possible I pulled out all my scripting to invoke MAXIMUM lazy.

Maximum lazy engage!

Ok. To start there is no ssh-askpass on Mac OS X Mountain Lion which is fairly un-bro of apple but whatever. I replaced it with an applescript I stole and slightly tweaked to run better.

Take this, it's dangerous to go alone. onlyhavecans/mac-ssh-askpass · GitHub

Just put it in ~/Applications like me or /usr/local/bin or where ever makes you warmest and fuzziest inside. Either way don't forget where you put our makeshift ssh-askpass

Now break out the applescript editor and take this puppy for a ride.

(*
This key adding mini program indexes the keyfiles on your jump drive
then prompts you for what one you want to load into ssh-agent

You need mac-ssh-addpass for this to work as well
https://github.com/onlyhavecans/mac-ssh-askpass
*)

property keyFolder : "KEYDRIVE:ssh"
property askPass : "$HOME/Applications/ssh-askpass"
property keyTime : "1h"

tell application "System Events"
  set theList to the name of every item of folder keyFolder
  set theKeys to {}
  repeat with i from 1 to the count of theList
    set theFile to {item i of theList}
    if (theFile as string) ends with "pem" or (theFile as string) ends with "_dsa" or (theFile as string) ends with "_rsa" then
      set end of theKeys to theFile
    end if
  end repeat

  set frontApp to short name of first process whose frontmost is true
  tell application frontApp
    activate
    set theKey to choose from list theKeys with title "Choose your Destiny" with prompt "What Key do you want to activate for " & keyTime default items {first item of theKeys} without empty selection allowed and multiple selections allowed
  end tell

  do shell script "SSH_ASKPASS=" & askPass & " /usr/bin/ssh-add -t " & keyTime & " \"" & {POSIX path of folder keyFolder} & "/" & theKey & "\""
end tell

Now pay attention or this will hurt.

The keyFolder property is in Apple's format because it just worked better that way. It needs to point to the folder that has all your key files. The format is drive:folder:subfolder:youget:theidea. Now make sure to set that and the location of our new ask-pass. Also because I was a lazy scripter the filter is hardcoded. If your key files end in anything other than .pem, _rsa, or _dsa you are perfectly legit but I hate you anyways and you will need to edit the nasty (theFile as string) ends with blah line.

Now... toss this applescript in your scripts folder and set the hotkey ctrl-opt-k in FastScripts and if you don't have it then shut up and go buy it in thanks from having this awesomeness rained down upon you… or put it in Alfred App... whatever bro; the point is you should be running all your shell and applescripts from the keyboard so do it.

Now hit that key command and watch the awesome of the menu box you can arrow through! Select your key of the minute and hit enter! Be amazed at the applescript password prompt.

Feel amazing bro. Feel amazing.

Note for those who are curious.

The applescripts are hooking off the frontmost application. IE it's telling whatever app is in front to activate¹ and then display the dialogs for the scripts. So don't bug out because the icon displayed is some "random seeming" app on your system. I do this so that the box is always to front so it can be keyboard operated and drops you right back into your front most app without fuzzing off the focus elsewhere or in some truly random app of my choosing like Finder.

I'll probably put this in my GitHub later, or forget and leave it as a blog exclusive so enjoy.

I had some weird edge case focus issues and making sure to activate before springing the next dialog box smoothed it all out. Sorry if that has any weirds, it really shouldn't ↩

Update to my Transmission Tools

D.O.S. — Wed, 27 Feb 2013 07:10:00 -0800

Nope, not something to help unmount and flush your transmissions, it's some tools for Transmission, specifically MY tools.

I just though I would share that I redid this repository completely and cleaned up the Move & Stop script to cover some possible bugs and be a bit more python modern. If you had any problems with it not moving single files or just wanted some slightly cleaner code go grab the new version.

I should really break out TransmissionRequestWrapper into it's own package and distribute that since it's so sweet. Please feel free to use it with some credit.

onlyhavecans/Transmission_Tools · GitHub

SSH Keys on a USB jump drive on Mac OS X (Mountain Lion Edition)

D.O.S. — Mon, 25 Feb 2013 10:39:00 -0800

Here I address the eternal struggle.

I want to store all my private keys on my jump drive I wear around everywhere. I use Win, Linux, but primarily Macs to do to my work so it needs to be some FAT variant formatted. I want to use the absolutely least hacky way.

Windows and Linux were easy to overcome. In short for windows you use putty to make a putty key and in linux you do something shockingly similar to what is below… but I get ahead of myself.

All the searches in duck duck go and google came up with "Nope! Can't do it! Give up!" for the Mac. All these people are horrible liars and don't want you to succeed. After figuring this out it was REALLY EASY as long as you are terminal competent. If you aren't terminal competent I'm not sure why you read my blog, I'd assume most of my stuff is really flying over your head.

As always YMMV and don't copy paste things I write here into the terminal.

tl;dr jump point

Find your uid with "id -u". 99% of the time it's 501.
Double check your drive name with "ls -ln /Volumes".

with sudo add a new line to /etc/fstab¹

#Drive called iamaKey and user 501
LABEL=iamaKey none msdos -u=501,-m=700

There is no step 4; eject & replug your key to enjoy ssh keys used directly from drive.

Considerations

You have to do this on every mac you use. The drive name is always the same but there is a chance the user id is different.
If you aren't sure if it worked or are having trouble give another "ls -ln /Volumes" which should look shockingly similar to this if you did it right:
```
lrwxr-xr-x  1 0    80     1 Feb 25 07:01 Macintosh HD -> /
drwx------  1 501  20  8192 Feb 25 10:17 iamaKey
```
You should REALLY make sure this drive name is unique to this drive for your machine. Any drive with the same name will get grabbed up by this now and if it's not msdos formatted or you don't want this havoc can be had.
In a multi-user environment this could likely cause weird edge case problems unless you are using very unique drive names.
I'm really sure putting spaces in your drive name will only wreck havoc. However you would assume that having lowercase letters in it would also fail hard but as you can see it works fine for me.
if you wanna be really cool then set up your config file on the jump drive as well and then alias keyssh to 'ssh -F /path/to/usb/config' Then that can ALL follow you around, except for on linux since it would mount on media or whatever instead of /Volumes breaking all your paths in the config but then you can just create a symlink anyways and it will work again… or something like that.
While not the point of this article yet your OpenPGP, GnuPG, GPG whateverPG, etc keys should also be WAY happier now placed on the jump drive. I believe they also hate being world readable.
This may not work in 10.9 or 11 or whatever comes next. The file /etc/fstab.hd says this is on the deprecation list. If Apple does break this down the road and you are reading this from the future the trick is to go looking wherever diskarbitrationd or it's successor is getting it's settings.
Final note, if you are going to do all this PLEASE don't use password-less keys. Just don't do it. Especially if you put your config file on the drive too. Then you are just telling people what systems they now have access too. Each key on the drive should have a strong & long passphrase, have it's comment clearly marked as being the jump dive key (on the machines they key is loaded), and should be unique to that jump drive only so it can be easily purged and reissued in case of loss.

On a new 10.8 machine /etc/fstab does not exist. You must create a fresh file as root. ↩

Puppet Configuration Checks with Jenkins

D.O.S. — Tue, 19 Feb 2013 19:32:00 -0800

Ok, so we have all our Nagios configs being sanitized and checked by Jenkins, why not Puppet: IT Automation Software for System Administrators? WHY NOT PUPPET!?

A lot of this is going to be rehash of the PRIOR article but I wanted to document this out for later anyways since it's slightly different.

Step One: Assumptions and Layout

I'm going to stop and assume we are well past the Jenkins setup phase. Please see the prior article for that… or better yet? make puppet do it for you. That's what it is for.

Now Puppet is a system that hold dynamically updated configurations of your everything and Jenkins is a system that automatically runs random code summated to it to validate it. I think needless to say I'm not even going to consider these two pieces of software running on the same machine. I'm serious the answer is NO.

Ok, now lets start by assuming the following;

You are not using anything crazy like githubbing puppet configs nor can you afford github enterprise.
Puppet Master (puppet.domain.ex) runs in DMZ and is locked down tight.
/etc/puppet is your config locations on puppet.domain.ex and you have a user that can write to it other than root.
Jenkins is hidden away in your network where it belongs, also fairly secured.
Your central git server is on the same lan as your Jenkins, if not on the same box for some reason.
You solemly swear not to do anything stupid and copy paste anything from this document directly into a command prompt. I may toss a bad char in there just to keep you honest.

We have a few caveats to overcome here but it's not impossible.

Step Two: Prepare your Puppet box

Puppet server should be locked down. So for my it's a box with all the screws down tight as they can be, puppet's web port and ssh passworded key to one user only is enabled. In order to allow pushing changes through git we will set up our friend the hub repo.

# Make puppet config directory a git repo
cd /etc/puppet
git init
git add .
git commit -m"Inital Commit"
# Make a hub repo in home
cd ~
mkdir puppet_configs.git
^mkdir^cd^
git --bare init
# Link and push to our hub
cd /etc/puppet
git remote add hub ~/puppet_configs.git
git push hub master

There! That wasn't hard at all. In fact it was a short rehash of what I did last yesterday. However, lets add the post-merge hook to THIS hub.

cat __EOF__
#!/bin/sh
cd /etc/puppet
unset GIT_DIR
/usr/bin/git pull hub master
__EOF__ >  ~/puppet_configs.git/hooks/post-merge
chmod 755  ~/puppet_configs.git/hooks/post-merge

So now when the hub gets pushed to, puppet gets a fresh load of configs and it does whatever it needs to do from then on out! Yea? Awesome

Now get out of your puppet box and STAY OUT (until it breaks, you did set up Nagios on it right?)

Step Three: Jenkins

In case you haven't already Jenkins is going to need a few upgrades. Hit it up with the following plugins.

Warnings Plug-in
Jenkins GIT plugin

I have been told you can set up this to work with the RVM plugin to deploy this and that but I took the cheap route since puppet is installed on this machine anyways. For this setup we will need one gem though. If this is for a buisness network I'd go the extra hours to learn RVM but it's not, this is my home lab.

sudo gem install puppet-lint

Now this is where I start to steal heavily from Continuous Deployment with Jenkins and Puppet.

Now starts the puppet configurations!

Name: Puppet
Source Code Management: Git
- Repositories: Your central git hub, not the one on puppet¹
Trigger Builds Remotely: check
- Authentication Token: pick something simple but unique here like 'stopHackingRoot'
Poll SCM: check
- Schedule: H/15 * * * *²

On your git repo for puppet you need to add a 'post-update' hook. to trigger builds remotely. Don't forget to sub out YOUR_TOKEN with the token you picked above and set the kenkins server proper. The quick and dirty is;

cd puppet_configs.git/hooks
cat __EOL__
#!/bin/sh
echo "Sending build command to Jenkins"
curl -sSL 'http://mycooljenkins:8080/job/Nagios_Config/build?token=YOUR_TOKEN' >> /dev/null
exec git update-server-info
__EOL__ >> post-update
chmod 755 post-update

Now lets add two execute shell actions.

Build Execute Shell Number one:

for file in $(find . -iname '*.pp’)
do
  puppet parser validate \
    --render-as s \
    --modulepath=modules \
    "$file" || exit 1;
done

Build Execute Shell Number two:

find . -iname *.pp -exec puppet-lint --log-format "%{path}:%{linenumber}:%{check}:%{KIND}:%{message}" {}  \;

The second one shouldn't error out but it will toss up style warnings and possible errors so lets check for those with our warning plugin.

Post-Build Actions
Scan for compiler warnings
- Parser: Puppet-Lint

Step four: The Danger Zone

Now this is where you might expect me to tell you to set up a Post-Build Action using Git Publisher or some script or another to automate your verified go-live puppet, maybe with a cool mcollector call to speed up your push...

That recomendation is NOT forthcoming

The simple fact of the matter is that if you are automatically pushing configs to puppet from Jenkins you have;

A passwordless key laying around on a CI box (and any boxes Jenkins spawns) to your puppet config (or worse)
Much much bigger balls than me
Hopefully a whole enterprise security team all over this shit like a hawk.

You can set up an automated push and hope that everything you or anyone else pushes in master is gold but my recommendation is to test in branches, tag, and then have a human eyes verification on the configs before your push to puppet. When you have a bad push to nagios monitoring glitches out. When pupped fucks up... well you are running down a bad road at full speed.

Seriously, if you don't have a central git server set up a central git repo SOMEWHERE safe, even on your jenkins box if need be (like we did for nagios) Just remember that your going to end up with something sensitive in there someday. ↩
You still want to regularly poll the git repo regularly just in case a commit script fails, it's no good to have it falling through the cracks. ↩

BunMailPot Alpha…

D.O.S. — Tue, 06 Nov 2012 15:16:00 -0800

Ok… so that is not the best name I've every come up with. Sorry, but whatever. This is a quick but fun one.

I have been collecting malware with my BunnyPot for a while and have been finding some diminishing returns coming to me. I started thinking that I can set up more of these low interaction server honeypots but how can I get even MORE goods? The ability to grab random files pushed to any server seems like a limited way here…

Then it hit me, actually dinged me… my email dinged that is.

Malware comes through the mail too, duh.

The setup

This is hella… well you will see. It needs some cleaning up and maybe not to be relying on Apple's Mail.App… but seriously it's a super quick and dirty hack. It's not even ON the server, it's on my laptop at the moment.

The mail rule

Ok, I have mail… lots of it, all my work and personal and trash mail all in one app with nice aggressive mail filters… so lets get the malware out!

Just make a mail rule that is all of the following;

Message is junk Mail
Any Attachment name Does not contain .htm

Now perform the following action;

Run Applescript ~/Library/Scripts/Applications/Mail/MalwareSaveRule.scpt

WHAT'S IN THE SCRIPT?! WHAT'S IN THE SCRIPT!!!!!!!

on perform_mail_action(ruleData)

  -- The folder to save the attachments in (must already exist)
  set attachmentsFolder to ((path to home folder as text) & "Downloads") as text

  -- Save in a sub-folder based on the name of the rule in Mail
  set subFolder to name of |Rule| of ruleData as text
  tell application "Finder"
    if not (exists folder subFolder of folder attachmentsFolder) then
      make new folder at attachmentsFolder with properties {name:subFolder}
    end if
  end tell

  -- Get incoming messages that match the rule
  tell application "Mail"
    set selectedMessages to |SelectedMessages| of ruleData
    repeat with theMessage in selectedMessages

      -- Get the date the message was sent
      set {year:y, month:m, day:d, hours:h, minutes:min} to theMessage's date sent
      set timeStamp to ("" & y & "-" & my pad(m as integer) & "-" & my pad(d) & "-" & my pad(h) & "-" & my pad(min))

      -- Save the attachment
      repeat with theAttachment in theMessage's mail attachments
        set originalName to name of theAttachment
        set savePath to attachmentsFolder & ":" & subFolder & ":" & timeStamp & " " & originalName
        try
          save theAttachment in savePath
        end try
      end repeat
    end repeat
  end tell

end perform_mail_action

-- Adds leading zeros to date components
on pad(n)
  return text -2 thru -1 of ("00" & n)
end pad

Simple ya? Nothing fancy, quick and kinda diiiirty. When this runs it creates a folder in your downloads that matches the name of the mail rule you created. I went with the hell creative Malware From Junk

Processing the malware

Just like every other this is how I process files thing I have written up this part is all in Hazel. Seriously I can't praise this app enough. Go pay this guy money. Stop reading this, go buy, use, love, live.

We add our folder which you remember is named after your mail rule to be watched by hazel… now lets start with the goods

Oh and disable Throwing away duplicate files, it might accidentally toss a variant… md5 will hash it down to what is and isn't the same.

Uncompress

This is too easy

Kind is Archive

Uncompress

Rename exe's to md5 and send to repo

Still really easy. The only trick is I set the rename and the upload to two separate scripts and actions so the whole thing can be halted easily in case something fails.

Extension is exe

Run Shell script embedded script

newname=$(md5 -q "$1")
mv "$1" "$newname"

Run Shell script embedded script

curl -sS -F upfile=@"$1" http://your.home.server:9000/cgi-bin/dionaea.py

Display Growl Notification "New MailMal (file)"

Notification

Ok, I'm not going to teach you how to set up growl to notify your phone for when you move this to your mac mini server at home, but needless to say Prowl - iOS Push Notifications is your friend. If you sent up the receiver I talked about in the last bunnypot article then you should already have that dionaea.py already to go!

Next steps

Next I need to find out how to get an email or even an entire domain onto as many spam lists as possible…

Feel free to send donations or suggestions to free_mal_pleez@bunnyman.info

simpliFiRE.IDAscope API browsing on a Mac

D.O.S. — Tue, 30 Oct 2012 19:29:00 -0700

I know this isn't the most revolutionary tip ever but my google searches pulled up nada when I tried to figure this out. I'm an IDA Pro MAC user. Bastard child of IDA right? Not really, except for plugins usually which is the point of this short tips post.

If you are using simpliFiRE.IDAscope to help you dig through malware you are collecting with your bunnypot (and gods why aren't you?) you will bang against the WinAPI browser whining about the following error while searching for native windows calls¹

Well, something has gone wrong here. Try again with some proper API name.
Exception: [Errno 2] No such file or directory: 'C:/WinAPI/fs/createfile.htm'

While not directly documented the fix is really easy. Just tweak your config.json file in the IDAscope folder.

"winapi": {
  "search_hotkey": "ctrl+y",
  "load_keyword_database": false,
  "online_enabled": true
}

After you do this don't even restart IDA, just close the IDAscope tab and rerun the script. This trick comes with another fringe benefit; IDAscope will load in a flash compared to the normal 30 second running, wait, almost... there... OK. The only disadvantage is that each search is now a web lookup but I don't think that's a horrible situation.

Now if I can only get the hotkey working right…

Happy hunting!

Of course I am assuming you are using version 1.0b (Released Oct 2012) or later, Otherwise you don't even have the ability. ↩

Git-Fu Advice

D.O.S. — Tue, 11 Sep 2012 15:05:00 -0700

Now who would have thought, I start blogging about git and people have advice. This post isn't just my personal learning but also some advice I received from others!

git reset HEAD^

From Alexis: Something I'm doing a lot is when I mess up with git, I sometimes need to uncommit something but keep the changes I had just before the commit.

git add -i

From Alexis: "Use this…" Brief but powerful advice. I never thought of using the interactive mode personally but if you are doing a complex commit or want to double over your work git's interactive mode is fairly robust.

cd my_git_repo
echo 'git push' > .git/hooks/post-commit
chmod 755 .git/hooks/post-commit

I mentioned this in an earlier post. This hook trick is for the lazy at heart. This script runs a push after every commit. If you always have access to your origin repo when you are coding, ie like you are a cubicle worker, this might not be too bad. This might get annoying of you are the type of coder that likes to write a lot of little commits on the road and then push in bundles.

    git diff --cached [--ext-diff]

As mentioned in my last post I'm big for double-checking my commits before go. This long command (that deserves an alias) pops open a diff of everything in your index ready to commit. Always a quick check before proceed!

Fun with Git, Jenkins, & Nagios

D.O.S. — Fri, 07 Sep 2012 11:04:00 -0700

Welcome to another edition on how to automate the hell out of your workflow.

Preface

One thing I have been addicted to since I learned it was source control. I don't understand how some developers work without it... and I really don't understand how any syadmins live without it. I have actually found it more useful as a sysadmin as a programmer, but only because at my day job I have used it in most of our major configs. Putting our 400+ file bind setup in subversion and using hooks to test and deploy our changes was not only a massive time saver but tail saver as well.

Another system that gets plenty of additions or tweaks is our Nagios configuration. Server gets deployed? Nagios. New services? Nagios. Something gets moved? Nagios. The list goes on.

The problem is that at work the Nagios commits aren't so automated. You push to the SVN server, then go to the nagios box and then checkout to test your changes. This results in extra commits to fix breaks and just work. There doesn't need to be work!

Getting to the point

The goal is to set up Nagios to have all it's configurations in git. Then I want Jenkins CI to test my commits automatically and push to production if they are solid! This was shocking easy to do with just a few gotcha's. I put this entire setup together in about a few hours, so expect possible areas of improvement.

Step one: Get your install on.

Install Nagios and git. I'm not even gonna get into this, also if you haven't already throw java on the box. I'm going to demonstrate how to do everything on one box but it should be easy to break everything out onto multiple systems where needed.

Jenkins might seem intimidating because it's java but it's not, really. I don't even bother with package installers, just grab the .war file from the website and java --jar jenkins.war¹. You might want to set up an init script and play with all the features but I'm not going to cover that, it's all well documented on Welcome to Jenkins CI!. If you are setting this up on a mac server, I tend to steal my LaunchAgent plists from Homebrew

For simplicities sake, or to mimic me, set up jenkins running as the same user as nagios. In my case I'm going to give nagios it's own dedicated jenkins that can only be accessed by private IPs.

Step two: Git-fu

This was the tricky part but it will seem easy when we are done.

Start by initing your nagios configs;

cd /usr/local/etc/nagios
git init
git add .
git commit -m"initial import of Nagios Configs"

Now you will want to set up the "hub" repo. For sakes of simplicity I'm setting it up in Nagios' home directory but as long as Jenkins and you can reach it then you are solid.

cd ~
mkdir nagios_configs.git
^mkdir^cd^
git --bare init

Now we clone over our working data to the "hub";

cd /usr/local/etc/nagios
git remote add hub ~/nagios_configs.git
git push hub master

While we are in the nagios config directory lets script a hook so that when this repo pulls and updates its configs from the hub it automatically reloads nagios.

cd .git/hooks
vim post-merge
#!/bin/sh
echo Running  'killall -HUP nagios' to reload settings
exec killall -HUP nagios
:wq
chmod 755 post-merge

Now leave this repo/directory and NEVER RETURN unless you break the hell out of the repo.

Now we have enough so that you can clone the hub repo and work on your nagios configs on your local workstation git style before pushing them back to your git hub... HA! See what I did there? Sorry...

Now what about testing?! and automation!? How does the data get from hub back to your nagios configs? Well this is where Jenkins comes in.

Step Three: Putting Jenkins to work

First you need the Jenkins git plugin. Jenkin's plugins are quick and easy. I'll shortcut you to it just for completion;

Go to http://nagiosbox:8080/
Click on Manage Jenkins
Click Manage Plugins
Click the Availible tab
Search for and select the Jenkins GIT plugin
Click Download now and install after restart

Jenkins; It's just that easy. Now lets set up our tests and deploy!

From the dashboard click New Job.
Project Name: Nagios_config²
Description: Whatever, not really important
Check Discard old builds
Set Max # of builds to keep to something reasonable like 5 or 20
Under Source Code Management check Git
Repository URL: is $HOME/nagios_configs.git or wherever you put this³
Set Branches to build to **
Under Build Triggers you want to check Trigger builds remotely
You'll need to pick a token; I recommend keeping it simple but not guessable like caronLovesBronies
I like to also set Poll SCM and set something like */30 * * * * just in case something doesn't get triggered. I don't think it's necessary though.
Under Build click the Add Build Step drop down and select Execute shell
Now we have our box to type our Nagios config Test, /usr/local/bin/nagios -v nagios.cfg

Technically that is all the Jenkins config we need to have it automatically clone the hub to it's own private repo and then nagios -v to test the config. All typed out it seems like a decent amount of steps but by the time you set up your third Jenkin's test you realize most all of that is boilerplate. Most of the time you spend with Jenkin's is checking to see why your build failed and sometimes tweaking tests for the environment. It's really a get set up and get out of your way kinda tool.

Now BEFORE we move on let's make Jenkin's do a little extra work and deploy from the hub to the live configs if the tests pass... oh crap you clicked save already didn't you... If you did just go back to the project and hit Configure.

Now Under Build click Add Build Step and add a second Execute Shell. Then put the following unto it⁴

cd /usr/local/etc/nagios
unset GIT_DIR
/usr/local/bin/git pull hub master

Now you can hit save

To finish off our super automation let's make it so that when anyone pushes to the hub it instantly triggers a build, test, and if the tests passes deploy. Let's just jump back to the command line to add a hook to our hub repo now

cd ~/nagios_configs.git/hooks
vim post-update
#!/bin/sh
echo "Sending build command to Jenkins"
curl -sSL 'http://localhost:8080/job/Nagios_Config/build?token=YOUR_TOKEN' >> /dev/null
exec git update-server-info
:wq
chmod 755 post-update

Note that with the curl line, if you enable authentication on jenkins you will need to create a user that has "build" level permissions and put it into that line. Also replace my example token with yours.

All right! Now we have it so that using ssh you can clone the hub repo and work on it. When you push back to hub it triggers Jenkins to build the tests. Then jenkins will take a copy, run nagios -v to test it, and if it passes it will tell the live config to pull the new updates... and once that is done the live config -HUPs nagios for us.

AMAZING!!! But not done yet.

Step Four: Final Boss^WConfigs

Ok. This might seem just about perfect but there is a catch. The default Nagios config uses absolute pathing to all of it's files. This means we need to modify some of Nagios' configs to properly allow Jenkin's testing to read all the proper files.

This process should be as easy as taking all the links at the top of the nagios.cfg file and just making them relative to the main config. For example, here is the head of my config minus comments;

$ egrep '^[^#]' nagios.cfg | head
log_file=/usr/local/var/lib/nagios/nagios.log
cfg_file=objects/commands.cfg
cfg_file=objects/contacts.cfg
cfg_file=objects/timeperiods.cfg
cfg_file=objects/templates.cfg
cfg_dir=systems
object_cache_file=/usr/local/var/lib/nagios/objects.cache
precached_object_file=/usr/local/var/lib/nagios/objects.precache
resource_file=/usr/local/etc/nagios/resource.cfg
status_file=/usr/local/var/lib/nagios/status.dat

Notice the gotcha in there!!! This one stuck me up for about two hours. I was unable to get Nagios to accept any relative path for the resource.cfg file. This introduces it's own gotcha but most people don't need to edit this too frequently.⁵

To explain a few other lines;

cfg_file=objects/commands.cfg
cfg_file=objects/contacts.cfg
cfg_file=objects/timeperiods.cfg
cfg_file=objects/templates.cfg
cfg_dir=systems

This is referring to /usr/local/etc/nagios/objects. I store all the default config files in there, commands, contacts, time periods, ect.. However I choose to put all my actual system, switch, & device configs in systems. I store personal templates and copies of all the defaults for reference in templates and then when I want to add a new group of systems I just copy a template to systems and fill it out. No need to edit the nagios.cfg every time.

Here is the layout of my nagios config directory;

$ tree nagios_configs 
nagios_configs
├── cgi.cfg
├── htpasswd.users
├── nagios.cfg
├── objects
│   ├── commands.cfg
│   ├── contacts.cfg
│   ├── templates.cfg
│   └── timeperiods.cfg
├── resource.cfg
├── systems
│   ├── chunkhost.cfg
│   ├── lazylopranch.cfg
│   └── shells.cfg
└── templates
    ├── printer.cfg
    ├── switch.cfg
    └── windows.cfg

The End?

So there you go! It's not perfect because of the listed caveats below. Someone malicious or even a sysop who is ignorant of the fragilities of the system can break it fairly easily trying to be tricky or cool. You could get around most of the git merge conflicts problems with hooks though.

The cool part though is this system is FAST. On a good build I push a change, by the time I can get to the web page to check the build it's already deployed to production.

There will probably be a follow up when I figure out how to make the setup a bit more solid, this really was a few hours of hack.

If you think I've missed anything feel free to drop me a comment.

Bonus points

Got someone who doesn't get git and can't remember to push? Hate the extra command?

Go into your personal repo and make git a little more SVN, for better or worse.

cd my_nagios_configs
echo 'git push' > .git/hooks/post-commit
chmod 755 .git/hooks/post-commit

How do you feel about circular dependencies?

If they are your thing then use the Nagios Jenkins Plugin to make Nagios check Jenkins and throw up alerts when your Nagios configs fail their tests!

Caveats

Never use git push --force

If you ever ever EVER EVER do anything that requires a git push --force on hub like an --amend then may god have mercy on your soul. You know how they tell you that push --force is bad and breaks things when you first learned git? This is that exact use case where it ruins everything. Just don't do it, the point of nifty automation is to make your life easier. Suck it up and let your tree be nice and linear, have mistakes, and be "bloated"; Git has some great storage ratios across lots of minor commits.

The test links to the prod version of resource.cfg

You are always testing against the live version of resource.cfg, which will usually be the HEAD^ version but if you break the build it could be even farther off. If you break the build on the resource.cfg file, the bad version will push, nagios will fail to restart properly, and then your NEXT build will fail in Jenkins and refuse to push so you will have to go into the server, test by hand, and then pull from hub manually. DAMN IT. I consider this to be a fairly major flaw that prevents me from wanting to deploy this in a professional environment.

You'll want to set up authentication, general security, and maybe even want to restrict access by firewall to jenkins in the long run. Jenkins is a well tested system but left unsecured and open on the internet this system can be invoked to execute arbitrary code in a snap. ↩
I've noticed that if you put a space in your Jenkins project name it puts a space in the path to the "workspace" that Jenkins uses to test and deploy from. This can break things from time to time so to error for safe sides don't do this. ↩
Don't use a tilde in Jenkins paths. It doesn't like them even a little bit. It probably has to do with it's cross system compatibility. ↩
You might be wondering why we have Jenkins go into the live config and run a pull from hub instead of pushing the configs to the live repo but that's just a nuance of git and most version controls that I have used. They don't like having things 'pushed' into them if they have a working copy. Only pulls will properly grab the changes and merge them into the working copy. That's why the hub we have uses the --bare flag. If You go to check out the hub repo you will see that it instead has the normal contents of .git just lying around instead of the normal files. ↩
Yes I consider this be a major flaw, read the CAVEATS section. ↩

Honing my Git-Fu Part 1

D.O.S. — Wed, 29 Aug 2012 21:26:00 -0700

Backstory

My git-fu sucks. I have to use an awesome git tool called SourceTree to do the git wizardry that I do. It's totally free and for the Mac so if you want to just jump into git and have expert features clicks away go download this. I bought it back when it cost money but now you can have it for free. I'll wait…

Anyways, I've been rolling around in the lap of GIT/SourceTree luxury these past months; clicking away and using features I only wished SVN could ever touch. However when jumping around between machines and VMs it would be faster to just use the command line. Now a days I'm now on the development team of a well sized open source project and having to fumble around git & github while testing submissions and making patches to help other people test is just NOT COOL. I think it all came to a head when the main project maintainer started flaunting some of his git-fu when submitting and fixing patches… well honestly since I just love cramming as much into my head as possible I thought I would hone my git foo.

Now back when I bought the McCullough and Berglund on Mastering Git - O'Reilly Media video's while they were on stupid sale and decided to double up with the newly released version of Version Control with Git, 2nd Edition - O'Reilly Media. Time to get my learn on.

What? Studying DISASM, Developing software, working a full time job AND now deep studying a VCS is too much? PISH, I'm single and have the time.

The point

SnowLprd wanted me to get some documenting on and take notes on what I find useful. Over the next month or so I am going to litter this blog with some posts on the "next steps" for git. While I organize my thoughts and really get my git-fu on it may be a bit all over the place. Sorry to anyone who finds this too rudimentary at points but I am going to try to focus on skimming the core concepts while posting lots of gotcha real world commands and why you would use them.

Starter Commands

I'm not gonna put up how to clone a repo here. SRSLY if you haven't gotten past 101 I'm not going to be useful. This is supposed to be 105b.

git remote set-url origin git@github.com:onlyhavecans/pelican.git
git remote add upstream git://github.com/getpelican/pelican.git
git remote add MyExBF git://github.com/justinmayer/pelican.git

Ok, so this one could be obvious but double check to make sure you have everything linked properly. You want the the R/W link to any repos you are pushing to. You might want to change your auth or connect style if you've say… been made a repo maintainer. For sakes of safety/neatness you should still treat your upstream as R/O and make all your changes in push requests and patches. Also add the R/O of your major contributors and co-developers so you can test their patches easier.

git pull upstream master && git push origin

There really should be a shortcut in this for git. Maybe I'll discover it later. I don't do my work on getpelican/pelican, I have the onlyhavecans/pelican fork! Each time I want to work on something I branch, code, commit, push, pull-request. It's an endless cycle. I never really touch my master, I just want to keep it up to date with the upstream. I don't know an amazing shortcut for that so the above micro shell script does the trick.

git push origin --delete <branchname>

I branch for every single patch to maintain tree neatness… however this becomes ugly fast on a good code month. This quickly dumps a branch and deletes it from github. Don't do this before it's been merged into master though, you are asking for pain the first time you make that mistake. Try to remember, branches aren't too expensive disk-wise so don't g too crazy on deleting them.

git add -p <file||.>

Don't add whole files to your staging area! Jeez! Who does that anymore! Be 37337 and review every change you make as you add to staging by committing at the patch level instead of at the file level by using the -p flag

git log HEAD^^ -p

This one took me a minute to figure out. So git show shows the patch for the last commit with patch if applicable but what happens when I want to spew the patches from the last three commits on someone else's branch to get a quick idea of what they are fuffng up? This one! Don't blindly follow my ^^ syntax either, learn about it below.

git commit --amend -c HEAD && git push --force

Ok, for bigger things --fixup or --squash might be better but when you open up a PR and instantly realize you are a total moron and everyone is gonna see how dumb your are because of your misspelling/typo/misscommited line… this squished one liner will recommit your quickly staged fix (letting you tweak the commit message in case that's your problem too) and replace your commit in the Pull Request with this much more awesome one.

Concepts to think of

Revision bad4dad

All revisions are named with SHA-1's of the actual commit contents. Then it uses something called treeish to sparse that down to the first 5-8 unique (to the repo) hex. That's why your commit's name is bad1dea. Obviously since repo's are distributed there is no way to create linear counters so all repo's are actually a linked list of commits. Everything from branches, to tags, to HEAD is just pointers to SHA-1.

HEAD^

Now then what's the ^? Well everyone knows HEAD is an alias to the most recent commit. Well you can add the ^ to any repo name and it's now the previous one. So bad1dea^ means the previous bad idea, while HEAD^ literally just means the previous commit. What's better is it's stackable HEAD^^ is two back. Now you probably don't wanna stack 10 ^'s so just use the shorthand HEAD~10 for ten revisions back.

Ahhh? History is a Linked list! so this traverses the linked list and returns all the revisions! Good for putting together a change log. To just see the last two commits git log HEAD^..HEAD Yea? Thats the cool stuff.

BunnyPot Alpha

D.O.S. — Sat, 04 Aug 2012 15:20:00 -0700

Since the beginning of the year I have been mildly obsessed with reversing and malware. Studying ASM and low level debugging is put an awesome edge on my understanding of computer science, programming, and hacking in general.

A week or so ago I started reading the Malware Analyst's Cookbook and DVD: Tools and Techniques for Fighting Malicious Code where they recommend you build up a honeypot to collect malware. Of course I can't a simple suggestions and set up a normal honeypot; Being a SysAdmin at heart I have to automate the hell out of everything.

If you aren't interested in the scripts or the setup just scroll down for the iPhone pic money shot.

Thus is born BunnyPot 0.5

I started with a Ubuntu VPS, patched but lightly hardened. Obviously being a honeypot the system has to be quasi-disposable.

Setting up dionaea was the hardest part!! Seriously. The install directions for it are fucked and the whole thing just feels like a one man hack job, however it has the cool features that I want to play with. I wrote up a little batch script that SHOULD one-shot install it. Just like the original install instructions some of the versions might disappear/break so some of the fetch instructions may break and meed to be tweaked. While this script is dirty it's smart enough to exit with error if it fails to download or decompress a file. However if you run it outside of sudo it will fail in ugly ways. Standard disclaimer applies; never run random scripts off the internet without vetting them yourself.

#!/bin/sh

aptitude install libudns-dev libglib2.0-dev libssl-dev libcurl4-openssl-dev \
libreadline-dev libsqlite3-dev python-dev \
libtool automake autoconf build-essential \
subversion git-core \
flex bison \
pkg-config p0f

mkdir /opt/dionaea


# liblcfg (all)

git clone git://git.carnivore.it/liblcfg.git liblcfg
cd liblcfg/code || exit 1
autoreconf -vi
./configure --prefix=/opt/dionaea
make install
cd ..
cd ..


#libemu (all)

git clone git://git.carnivore.it/libemu.git libemu
cd libemu || exit 1
autoreconf -vi
./configure --prefix=/opt/dionaea
make install
cd ..


#libnl (linux && optional)

git clone git://github.com/tgraf/libnl.git
cd libnl || exit 1
autoreconf -vi
export LDFLAGS=-Wl,-rpath,/opt/dionaea/lib
./configure --prefix=/opt/dionaea
make
make install
cd ..


#libev (all)

wget http://dist.schmorp.de/libev/libev-4.11.tar.gz
tar xfz libev-4.11.tar.gz
cd libev-4.11  || exit 1
./configure --prefix=/opt/dionaea
make install
cd ..


#Python3

wget http://www.python.org/ftp/python/3.2.3/Python-3.2.3.tgz
tar xfz Python-3.2.3.tgz
cd Python-3.2.3/ || exit 1
./configure --enable-shared --prefix=/opt/dionaea --with-computed-gotos \
      --enable-ipv6 LDFLAGS="-Wl,-rpath=/opt/dionaea/lib/ -L/usr/lib/x86_64-linux-gnu/"

make
make install
cd ..

#Cython (all)

#We have to use cython >= 0.15 as previous releases do not support Python3.2 __hash__'s Py_Hash_type for x86.
wget http://cython.org/release/Cython-0.16.tar.gz
tar xfz Cython-0.16.tar.gz
cd Cython-0.16 || exit 1
/opt/dionaea/bin/python3 setup.py install
cd ..


#udns (!ubuntu)
#udns does not use autotools to build.
wget http://www.corpit.ru/mjt/udns/old/udns_0.0.9.tar.gz
tar xfz udns_0.0.9.tar.gz
cd udns-0.0.9/ || exit 1
./configure
make shared
#There is no make install, so we copy the header to our include directory.
cp udns.h /opt/dionaea/include/
#and the lib to our library directory.
cp *.so* /opt/dionaea/lib/
cd /opt/dionaea/lib
ln -s libudns.so.0 libudns.so
cd -
cd ..


#libpcap (most)

wget http://www.tcpdump.org/release/libpcap-1.1.1.tar.gz
tar xfz libpcap-1.1.1.tar.gz
cd libpcap-1.1.1 || exit 1
./configure --prefix=/opt/dionaea
make
make install
cd ..


# FINALLY!!!!

git clone git://git.carnivore.it/dionaea.git dionaea
cd dionaea || exit 1
autoreconf -vi
./configure --with-lcfg-include=/opt/dionaea/include/ \
      --with-lcfg-lib=/opt/dionaea/lib/ \
      --with-python=/opt/dionaea/bin/python3.2 \
      --with-cython-dir=/opt/dionaea/bin \
      --with-udns-include=/opt/dionaea/include/ \
      --with-udns-lib=/opt/dionaea/lib/ \
      --with-emu-include=/opt/dionaea/include/ \
      --with-emu-lib=/opt/dionaea/lib/ \
      --with-gc-include=/usr/include/gc \
      --with-ev-include=/opt/dionaea/include \
      --with-ev-lib=/opt/dionaea/lib \
      --with-nl-include=/opt/dionaea/include \
      --with-nl-lib=/opt/dionaea/lib/ \
      --with-curl-config=/usr/bin/ \
      --with-pcap-include=/opt/dionaea/include \
      --with-pcap-lib=/opt/dionaea/lib/ 
make
make install

#Fix some permissions
chown -R nobody:nogroup /opt/dionaea/var/dionaea
exit 0

WHEW! That's a big one huh? That took me several hours to suss out so I figured it's best shared! I also wrote an update script for the system and all the git based packages, which MOST were.

#!/bin/bash

aptitude update
aptitude -y safe-upgrade


# liblcfg (all)

cd liblcfg || exit 1
git clean
git pull
cd code
autoreconf -vi
./configure --prefix=/opt/dionaea
make install
cd ..
cd ..


#libemu (all)

cd libemu || exit 1
git clean
git pull
autoreconf -vi
./configure --prefix=/opt/dionaea
make install
cd ..


#libnl (linux && optional)

cd libnl || exit 1
git clean
git pull
autoreconf -vi
export LDFLAGS=-Wl,-rpath,/opt/dionaea/lib
./configure --prefix=/opt/dionaea
make
make install
cd ..


# FINALLY!!!!

cd dionaea || exit 1
git clean
git pull
autoreconf -vi
./configure --with-lcfg-include=/opt/dionaea/include/ \
      --with-lcfg-lib=/opt/dionaea/lib/ \
      --with-python=/opt/dionaea/bin/python3.2 \
      --with-cython-dir=/opt/dionaea/bin \
      --with-udns-include=/opt/dionaea/include/ \
      --with-udns-lib=/opt/dionaea/lib/ \
      --with-emu-include=/opt/dionaea/include/ \
      --with-emu-lib=/opt/dionaea/lib/ \
      --with-gc-include=/usr/include/gc \
      --with-ev-include=/opt/dionaea/include \
      --with-ev-lib=/opt/dionaea/lib \
      --with-nl-include=/opt/dionaea/include \
      --with-nl-lib=/opt/dionaea/lib/ \
      --with-curl-config=/usr/bin/ \
      --with-pcap-include=/opt/dionaea/include \
      --with-pcap-lib=/opt/dionaea/lib/ 
make
make install

#Fix some permissions
chown -R nobody:nogroup /opt/dionaea/var/dionaea
exit 0

If you are following along at home I really recommend you put both of those scripts into their own dio-build directory since they make a mess of any directory you run them in and you want to keep all the installers about.

All you have to do after that is start it all up

  sudo p0f -i any -u root -Q /tmp/p0f.sock -q -l -d -o /dev/null -c 1024 && sudo chown nobody /tmp/p0f.sock
  sudo /opt/dionaea/bin/dionaea -u nobody -g nogroup -p /opt/dionaea/var/dionaea.pid -D

If it doesn't self destruct then you are up and going. If course I recommend you set this up with something like runit but this will get your honeypot working.

Where the cool automation comes in

dionaea doesn't seem to have the cool IRC features of other honeypots but it still has web submissions. In fact it's designed to submit to several standard honeypots. I'm greedy so I deleted all of those and put in my Mac OS X server as one. For a quick and dirty python receiving server check through MNIN Security | Malware Analyst's Cookbook or buy the book.

Having these md5 named malware files on my OS X server means… what? They get backed up? Whoopty dooo! I guess that means I have easier access to them but so what. Lets make my system do all the pre work for me!

Using the power of the almighty Noodlesoft Hazel, Homebrew, and a few python packages the system watches the incoming binary directory for new files and then goes to work¹. It runs a series of scans on the files and chimes my phone with the summary on my new gift from the internet!

Sweet huh? The first time I was sitting around and my phone chimed, delivering info on a fresh piece of malware, I did a dance.

Logging into the box and running a single check against the name nets me a larger summary;

➜  malware  ./pescanner.py binaries/12fb7332920a7797c2d02df29b57c640
################################################################################
Record 0
################################################################################

Meta-data
================================================================================
File:    binaries/12fb7332920a7797c2d02df29b57c640
Size:    57344 bytes
Type:    PE32 executable (GUI) Intel 80386, for MS Windows
MD5:     12fb7332920a7797c2d02df29b57c640
SHA1:    47707d46e3324be11cde22cdfe2be7d17193a5a2
ssdeep:
Date:    0x4AD5C802 [Wed Oct 14 12:45:54 2009 UTC]
EP:      0x40369e .text 0/4
CRC:     Claimed: 0x0, Actual: 0x151d9 [SUSPICIOUS]

Signature scans
================================================================================
Clamav: binaries/12fb7332920a7797c2d02df29b57c640: Trojan.Spy-78857 FOUND

Resource entries
================================================================================
Name               RVA      Size     Lang         Sublang                  Type
--------------------------------------------------------------------------------
RT_RCDATA          0xa0a0   0x4621   LANG_KOREAN  SUBLANG_KOREAN           data
RT_RCDATA          0xe6c8   0x400    LANG_KOREAN  SUBLANG_KOREAN           PE32
 executable (GUI) Intel 80386, for MS Windows, Petite compressed

Suspicious IAT alerts
================================================================================
CreateProcessA
StartServiceA
CreateServiceA

Sections
================================================================================
Name       VirtAddr     VirtSize     RawSize      Entropy
--------------------------------------------------------------------------------
.text      0x1000       0x5196       0x6000       5.959918
.rdata     0x7000       0xc76        0x1000       4.498997
.data      0x8000       0x1cdc       0x1000       3.521438
.rsrc      0xa000       0x4ac8       0x5000       7.283870    [SUSPICIOUS]

The next piece of this will involve setting up parallels on the Mac Mini server and getting remote malware deployment and testing going. dionaea also records tcp streams so that it can capture shell codes, something I also want to start indexing and stealing up.

Where is this awesome collaboration of scripts you ask? Well I'm not giving up all the goods yet, especially since it's incomplete. Eventually I will probably post the whole deal. However most everything I set up is in Chapter 3 of the book I have been talking about the whole time. If you are super curious or want your own bunnypot now I can be bribed with beer and liquor like most hackers. ↩

More Pelican, or how I learned to stop worrying and start developing with others.

D.O.S. — Tue, 17 Jul 2012 14:19:00 -0700

Pelican 3.0 is almost out the door now. It's been great working on a "real" project with other developers for what is, in retrospect, the first time ever.

Some programmers I know would rather fork a project and trudge off into the snow with the code base, then fork off their own leg and write their way back before working on someone else's code; I think at some point I caught that same fever from someone close to me. I think the phrase tossed around was; "The only thing that is worse than someone else's code is working with someone else's code." The crazy part is, after actually working with other people on a project I wish I had done it earlier.

After fitting in to someone else's product and pitching my ideas and work into it instead of trying to run in it I almost don't want to go back. Not everything I've suggested flew, some things were downright rejected and other things had to be rethought to fit it all... but it was a great experience to have a driven goal and to contribute to it. Luckily the Pelican team is also small, awesome, and very supportive so I've found it downright pleasant to work with them... I realize that not every team is as friendly and constructive as the Pelican project has been but the concept of check and balances and the feedback alone is invaluable.

Yes, there is a part of me that wants to be the solo super programmer hacker that gets an idea and sits down to write the next program that becomes the next golden standard but we can't all be Linus and honestly, who wants to maintain that project into the ground? I'm not an amazing programmer, maybe someday I will be after lots of training. My programs work but without constant learning and challenges they aren't going to be written well.

While cleaning out little bugs and documenting things over several times may not be everyone's best use of time; I feel getting used to have to test everything, double think my commits, write tests for everything is important. Normally I can be a little lazy with my commit, get it in and test it after that... but I can't just do that with a bigger project. On a personal project you can quickly develop and learn bad habits. Working with a team forces good habits, or at least as good habits as the team itself.

For big professional life time programmers I am probably preaching to the choir. Term development, code review, and focused controlled projects with deadlines are vital to putting out great projects. For all those that have been playing the part of the antisocial developer, like I was, I really recommend finding a smaller project that you can help grow, getting in on the channel, and getting your team coding on... you will be shocked how great it can be to work on a project together once ego gets set aside.

The project has sidetracked several other of my side projects, but in one of the best ways possible considering the amount of joy and constructive knowledge and experience I think I am getting from it. I still plan on learning more languages, namely lisp… but right now I am juggling working on this project and studying reversing, one of my biggest interests for the future.

As a whole sidetrack to this whole post, I think that infosec is where I am going to steer my career and reversing and malware is my current big "interest". Even if I can't grok it enough to become a professional malware analyzer or software analyzer, learning some of the lowest levels of computing and programming will help me understand all the other facets of security and infrastructure that much better. Not that infrastructure management isn't a great career, but I don't feel it's really focusing me where I want to be long term. I have days where I really regret stepping up from Sr. Systems Admin.

Now with less dynamics

D.O.S. — Fri, 06 Jul 2012 19:04:00 -0700

I've been stalling a lot on writing this post. I guess I wanted everything to be perfect by the time I actually wrote about it however my willpower to finish these test cases is weak.

The site has now changed, yet again. I decided not to wait 5 years between refreshes this time.

The biggest change people should notice is at first glance you shouldn't notice change. When you click around it should be easy to realize that the site isn't WordPress based anymore though. I've switched over to Pelican. The project still feels very young and small, but it's done in python and the developers are very active and open to submissions so I jumped in.

It's took some wiggling around to get into it, and lots of bashing on my old chunk theme to get it to look perfect in Pelican but in the end I really like the fact I did a lot of back end work to the system. I submitted my theme to the official Theme's repo and am currently working on adding in several features of my own to the project... and by several I mean I have a to-do list longer than most any of my own software by far.

A little thank you goes a long way

D.O.S. — Wed, 27 Jun 2012 17:37:00 -0700

Things like this happen sometimes

[4:27 PM] <SnowLprd> tBunnyMan: Nice work on #389. I can see how that will come in handy. :^)
[4:27 PM] <tBunnyMan> Thanks!
[4:28 PM] <tBunnyMan> I really just wanted a 404 and 50x error that matched my theme... but making it more extensible helps everyone
[4:28 PM] <tBunnyMan> I just need to finish this test case for it when work stops being annoying
[4:29 PM] <SnowLprd> Nice of you to generalize it for everyone, despite only needing it yourself for 404/50x errors.
[4:29 PM] <SnowLprd> And the test will also be most welcome! \o/
[4:32 PM] <tBunnyMan> Pelican is fairly awesome IMO, I hope I can help tweak it into epic levels.
[4:34 PM] <SnowLprd> With contributions like yours, those levels will be here in short order. :D
[4:34 PM] <tBunnyMan> haha. I'm not that good ;p Thanks for the kind words
[4:35 PM] <tBunnyMan> Catch you around. It's time to travel
[4:35 PM] <SnowLprd> Every little bit counts!
[4:35 PM] <SnowLprd> Sounds good. Cheers!
[4:36 PM] <tBunnyMan> It does. It's why I love contributing little things to big projects.

All I really was trying to do is add a very quick and simple feature I needed. This was the response I got for it! The thing is, you would be shocked how infrequently I see something like this... It's this type of additude and behavior amongst developers that makes people WANT to work with eachother and help really grow a product.

I just wanted to post a little good will and show that some people rule. Expecially SnowLprd, kylef, bbinet, and doubly so alexis!

Small Update

D.O.S. — Tue, 19 Jun 2012 00:00:00 -0700

I haven’t posted in a while because I am working on redoing this site. Once I am done with the backend I will start updating regularly.

My Contribution to Calibre

D.O.S. — Mon, 30 Apr 2012 00:00:00 -0700

Backstory

When I bought my NOOK Simple Touch™ in January of this year I rediscovered calibre E-book management. While software always felt clunky in Mac OS X you could never deny it's power and sheer amazing once you got past the UI.

Being a big fan of ReadItLater I immediately tried to have the program pump my massive reading list into my Nook. To my dismay I discovered that the plugin was hardly complete. It piped my entire ~500 article¹ reading list into a several megabytes large ebook and ordered articles from newest to oldest. Running the plugin a second time… produced the same results. It hadn't even the courtesy to mark articles as read.

After digging about I found a few "close enough but incomplete" and outdated solutions along side plenty of complaints. You can't blame Calibre, the massive project's maintainers likely don't use the service like I do of at all.

Solution

A python based open source project with something that I want improved? A bunny like me only can do one thing in a situation like this; reach back to grab one of my many 'cans', learn the API Documentation for recipes, and get my money out of PyCharm.

After a few days and learning more than just Calibre api², I put out what I called ReadItLater V3 on to my github and reposted it to Calibre Recipes MobileRead Forums. Other people picked it up and I ended up adding a few extra features per requests

All was good with the world.

Reward

A few weeks ago Read It Later officially rebranded to Pocket. This prompted me to pull up Calibre and check my plugin was working… all clear. Then, since I like to cover my bases, I pulled up Mobile Reads and dug around… and that's when I found it.

Calibre merged my version into the official trunk³.

This invigorated me to put my all into the project. However, all I really managed was the branding, some code cleanup, and docs. Pocket wasn't ready to give out their new API to small beans like me. At one point during this Calibre devs merged in someone else's but quickly pushed my latest version once I completed it.

I find myself flattered to have my code and name, even my pseudonym, in such a prestigious python project. Some of my friends felt my accomplishment was mediocre at best but I reject the haters. I've actively contributed to widely used open source. I don't get a ton of warm fuzzies from my wins but this was one of those times. I'm still stoked as hell and hope that I can do it again.

I should get my reading list under control. My situation with real books looks exactly the same way, stacks and stacks of the latest up read interests. I read plenty but the information overload fills the world; the challenge of sorting fluff from gold increases daily. ↩
mechanize Browser and Beautiful Soup are now favorites in my toolbox of handling and scraping web content, something I do more than any bunny should. ↩
While this link only shows one part of the merge I chose it because it showed when they merged in my name to the top. Eventually they replaced the whole thing with my version. ↩

The search for the perfect wallet

D.O.S. — Wed, 21 Mar 2012 00:00:00 -0700

Through my whole life I have always struggled to find the perfect tool for the job. No matter what I am doing I from the big to the little I question methods and tools I am using. Sometimes I find it, sometimes I struggle on and on. This is a story about how I think I found perfection.

I have always had a fascination with wallets. They are where we keep the summary of our life and our valuables. Your wallet is like a woman's purse, the thing that curious people want to pick through to find out more about you and the thing more introspective people think about how it reflects on them. Any time you pay for something the seller glances down and takes a quick look not just at your payment but at what it is coming out of. Your wallet is a statement of you in one of those subtle ways.

In my search I have gone through piles of wallets. I still have a small box in my closet that I have been loathe to throw away with then. I have gone from sleek leather, to big woven bulletproof nylon with lots of features, to slim money clips with stamped leather card slots, to the famous minimal Slimmy. Even with the Slimmy I went through two separate versions, one with window and one without.

My goal was to find a wallet that held everything I need to carry with me while still being small and slim. I have been on the largest minimal kick so the Slimmy and money clips were used for the last three years but they were too minimal for my needs. In the end the fact remains that any wallet I found that is small enough to make me happy didn't hold enough and any wallet with enough space for everything was just too damn big.

The answer to my problem came to me several weeks ago while I was hanging out at a bar with my younger brother. We were about to tip up and I heard a familiar sound, the sound of rubber against the hard plastic of a credit card before snapping into place.

As a child my father never carried a wallet. He always just took his little pocket address book, pressed all of his cards up against it, folded up some money, and then wrapped it up in a pair of rubber bands. I asked him several times about why he never carried a wallet like everyone else and he told me it was because he never found a wallet he liked. Also, since he received the newspaper which is wrapped in a rubber band, it was free and replaceable weekly.

Like father like son I guess. When I asked my brother about it he said he "got it from dad" and "It was inevitable, you will succumb to it too." It was and I did.

In the top most picture I didn't use a normal rubber band but I quickly found that any fancy band I used was too much. Through a few months of testing I found that a traditional rubber band was just enough. In the picture you can easily see that I took the contents of the Slimmy, added my notebook and with the band it is still thinner than the EMPTY Slimmy. Even if I were to add more cards, which I have done, the wallet still manages to be no bigger than the standard four cards and a small amount of cash in a Slimmy that they recommend.¹

My rubber band wallet currently contains a Moleskine extra small notebook², A collapsable pen to write with, a small set of cash, and every card and ID I could need on a day. Somehow it is still smaller and more useful than every wallet I have owned before!

As you can see with the pictures just above the new wallet is just a hair thicker than a AA battery and holds a mass of cards. In fact it actually contains more cards than I actually need. I found if there was too few cards the pen will roll over the cards and wiggle about. I now keep every gift card I have, my Fresh & Easy card, and a random hotel keycard from FurtherConfusion with art on it. Minimalism is amazing but the fact that I never have to go "Oh crap, I forgot card X" is nice.

As for looks? What does this say about me? While as a kid I thought it tacky to have a rubber band for a wallet most people tend to comment on how utilitarian and useful that is. How it holds a notebook and pen in the same space as their own bloated wallets.

No matter how many times I think about the process I think I found the perfect one for the job.

My problem with the slimy is its design. It is only meant to hold about four cards and a small amount of cash. On any day I need to carry my ID, Work ID, Debit Card, PCard. This means ALL extra credit cards, Costco memberships, health cards, gift cards, and even business cards are out. Yes I can live without them but having to drive home and grab my Costco card or applicable gift card every time I want to grab some shopping on the way home is ANNOYING. ↩
I use a BRIGHT PINK notebook too. It gets comments a lot, sometimes snide, sometimes funny… but the fact of the matter is when my "wallet" is on the table it's hard to miss and that in the end is the point; High visibility. ↩

Nothing happens for a reason

D.O.S. — Wed, 14 Mar 2012 00:00:00 -0700

"Everything happens for a reason." I read this normally innocuous phrase on a friend's blog recently and wanted to toss my coffee mug. The notion of the phrase makes me want to jump out of my chair and flip my keyboard. I'm not going to rag on them for saying or even feeling it, it's simply not my place to tell people how to live their lives, however it bothers me that people who I care about in the world think like that.

I think the mentality of saying not only goes against my beliefs but undermines people's ability to excel. The phrase is a sheep-like mindset that tells us to embrace our losses instead of trying to overcome them. The nature¹ of the phrase is predestination, which inherently is talking about divinity, though most often when I see people use it the meaning is less of a religious mentality² and more an extension of sour grapes. When we over-rationalize our painful decisions or situations we become placated or fight the reoccurrence of the opportunity because it wasn't meant to be. We need some level of coping when we have perceived large losses but too much drags us down.

I quickly jumped up on the train of thought that this year's Burning Man was going to be a bomb when I was denied access to STEP³. However I never rationalized my loss to some point of divinity or told myself that there might be something better I need to make. There is no meant to be in my eyes; only what is and what is not. To this end I penciled in three furry conventions to fill my summer and replace the burn. Just a few days ago some things fell in to place and now I have access to tickets. Had I chose the meant to be mentality would I still want these tickets? Was there a reason for me to go through the extra month and a half of mental strife? No. It happened because mistakes were made. I received an opportunity to get tickets this month because of the people I choose to associate with and the impression I put on them. There was no luck in the preceding.

I am unwilling to ague the validity of anyone's religious or spiritual beliefs but I believe that the moment you take the mentality that anything but you controls and shapes your life you give up the ability to reach your full potential. Removing the accountability for your problems not only dulls the pain of the problems but also the need to own and fix the them.

I'm not going to get into the literal meaning of the phrase, because technically everything does happen for a reason. Laws of nature, inertia, influence, chemical reactions, etc. I am taking about what people mean when they say everything happens for a reason. I was unable to find the etymology in searches but according to my results it seems that the phrase most likely stems from religion. ↩
Though my mother often uses the phrase followed with "God has a plan", most of my friends don't seem to subscribe to the hyper religious mentality. ↩
I didn't enter the initial lotto, instead choosing to hold out for the full price second sales. Only people who entered the lotto for cheaper tickets where given access to STEP, the second chance ticket (re)sale system. ↩

Things that are not happening for 100, Alex

D.O.S. — Thu, 08 Mar 2012 00:00:00 -0800

A: Importing this content into the new site was attempted only for a few moments before a review of it's nature and amount had it canceled.

Q: What is, BunnyMan's old LJ?

Correct!

Wow. No, not happening. I still have it all backed up but I think all that old content will stay back there. I am currently trying to set this up so I can post my mostly private journal here as well as posting public news and information. LJ will not be getting 100% of the content I post, but it still will get some updates. The dawn of handpicked RSS content and feeds has consumed community and "friend based" feeds and for the most part I think we are better for it. I'm getting on the ship.

Wow!

D.O.S. — Thu, 08 Mar 2012 00:00:00 -0800

Another new site, another clean slate ready to be filled with so much information!

I will try to make this site as little about the drivel, porn, rage, and trolls that the rest of the internet seems to amuse itself about. This site is going to be all about the things that really interest me. Currently those things are Fiction, Writing, Technology, writing fiction, and Productivity… There also may be stuff about glitchy painful music and podcasts.

My old site is archived safe and while most of the old links may be gone feel free to check it out. It highlights one of the busiest and most externally exciting period of my life where photography and music production was every facet of my life. You can also feel free to ask questions if you really care. I don't expect this site to get too much traffic.