Tagged with git

Git-Fu Advice

Now who would have thought, I start blogging about git and people have advice. This post isn't just my personal learning but also some advice I received from others!

git reset HEAD^

From Alexis: Something I'm doing a lot is when I mess up with git, I sometimes need to uncommit something but keep the changes I had just before the commit.

git add -i

From Alexis: "Use this…" Brief but powerful advice. I never thought of using the interactive mode personally but if you are doing a complex commit or want to double over your work git's interactive mode is fairly robust.

cd my_git_repo
echo 'git push' > .git/hooks/post-commit
chmod 755 .git/hooks/post-commit

I mentioned this in an earlier post. This hook trick is for the lazy at heart. This script runs a push after every commit. If you always have access to your origin repo when you are coding, ie like you are a cubicle worker, this might not be too bad. This might get annoying of you are the type of coder that likes to write a lot of little commits on the road and then push in bundles.

    git diff --cached [--ext-diff]

As mentioned in my last post I'm big for double-checking my commits before go. This long command (that deserves an alias) pops open a diff of everything in your index ready to commit. Always a quick check before proceed!

Tagged

Fun with Git, Jenkins, & Nagios

Welcome to another edition on how to automate the hell out of your workflow.

Preface

One thing I have been addicted to since I learned it was source control. I don't understand how some developers work without it... and I really don't understand how any syadmins live without it. I have actually found it more useful as a sysadmin as a programmer, but only because at my day job I have used it in most of our major configs. Putting our 400+ file bind setup in subversion and using hooks to test and deploy our changes was not only a massive time saver but tail saver as well.

Another system that gets plenty of additions or tweaks is our Nagios configuration. Server gets deployed? Nagios. New services? Nagios. Something gets moved? Nagios. The list goes on.

The problem is that at work the Nagios commits aren't so automated. You push to the SVN server, then go to the nagios box and then checkout to test your changes. This results in extra commits to fix breaks and just work. There doesn't need to be work!

Getting to the point

The goal is to set up Nagios to have all it's configurations in git. Then I want Jenkins CI to test my commits automatically and push to production if they are solid! This was shocking easy to do with just a few gotcha's. I put this entire setup together in about a few hours, so expect possible areas of improvement.

Step one: Get your install on.

Install Nagios and git. I'm not even gonna get into this, also if you haven't already throw java on the box. I'm going to demonstrate how to do everything on one box but it should be easy to break everything out onto multiple systems where needed.

Jenkins might seem intimidating because it's java but it's not, really. I don't even bother with package installers, just grab the .war file from the website and java --jar jenkins.war1. You might want to set up an init script and play with all the features but I'm not going to cover that, it's all well documented on Welcome to Jenkins CI!. If you are setting this up on a mac server, I tend to steal my LaunchAgent plists from Homebrew

For simplicities sake, or to mimic me, set up jenkins running as the same user as nagios. In my case I'm going to give nagios it's own dedicated jenkins that can only be accessed by private IPs.

Step two: Git-fu

This was the tricky part but it will seem easy when we are done.

Start by initing your nagios configs;

cd /usr/local/etc/nagios
git init
git add .
git commit -m"initial import of Nagios Configs"

Now you will want to set up the "hub" repo. For sakes of simplicity I'm setting it up in Nagios' home directory but as long as Jenkins and you can reach it then you are solid.

cd ~
mkdir nagios_configs.git
^mkdir^cd^
git --bare init

Now we clone over our working data to the "hub";

cd /usr/local/etc/nagios
git remote add hub ~/nagios_configs.git
git push hub master

While we are in the nagios config directory lets script a hook so that when this repo pulls and updates its configs from the hub it automatically reloads nagios.

cd .git/hooks
vim post-merge
#!/bin/sh
echo Running  'killall -HUP nagios' to reload settings
exec killall -HUP nagios
:wq
chmod 755 post-merge

Now leave this repo/directory and NEVER RETURN unless you break the hell out of the repo.

Now we have enough so that you can clone the hub repo and work on your nagios configs on your local workstation git style before pushing them back to your git hub... HA! See what I did there? Sorry...

Now what about testing?! and automation!? How does the data get from hub back to your nagios configs? Well this is where Jenkins comes in.

Step Three: Putting Jenkins to work

First you need the Jenkins git plugin. Jenkin's plugins are quick and easy. I'll shortcut you to it just for completion;

  • Go to http://nagiosbox:8080/
  • Click on Manage Jenkins
  • Click Manage Plugins
  • Click the Availible tab
  • Search for and select the Jenkins GIT plugin
  • Click Download now and install after restart

Jenkins; It's just that easy. Now lets set up our tests and deploy!

  • From the dashboard click New Job.
  • Project Name: Nagios_config2
  • Description: Whatever, not really important
  • Check Discard old builds
  • Set Max # of builds to keep to something reasonable like 5 or 20
  • Under Source Code Management check Git
  • Repository URL: is $HOME/nagios_configs.git or wherever you put this3
  • Set Branches to build to **
  • Under Build Triggers you want to check Trigger builds remotely
  • You'll need to pick a token; I recommend keeping it simple but not guessable like caronLovesBronies
  • I like to also set Poll SCM and set something like */30 * * * * just in case something doesn't get triggered. I don't think it's necessary though.
  • Under Build click the Add Build Step drop down and select Execute shell
  • Now we have our box to type our Nagios config Test, /usr/local/bin/nagios -v nagios.cfg

Technically that is all the Jenkins config we need to have it automatically clone the hub to it's own private repo and then nagios -v to test the config. All typed out it seems like a decent amount of steps but by the time you set up your third Jenkin's test you realize most all of that is boilerplate. Most of the time you spend with Jenkin's is checking to see why your build failed and sometimes tweaking tests for the environment. It's really a get set up and get out of your way kinda tool.

Now BEFORE we move on let's make Jenkin's do a little extra work and deploy from the hub to the live configs if the tests pass... oh crap you clicked save already didn't you... If you did just go back to the project and hit Configure.

Now Under Build click Add Build Step and add a second Execute Shell. Then put the following unto it4

cd /usr/local/etc/nagios
unset GIT_DIR
/usr/local/bin/git pull hub master

Now you can hit save

To finish off our super automation let's make it so that when anyone pushes to the hub it instantly triggers a build, test, and if the tests passes deploy. Let's just jump back to the command line to add a hook to our hub repo now

cd ~/nagios_configs.git/hooks
vim post-update
#!/bin/sh
echo "Sending build command to Jenkins"
curl -sSL 'http://localhost:8080/job/Nagios_Config/build?token=YOUR_TOKEN' >> /dev/null
exec git update-server-info
:wq
chmod 755 post-update

Note that with the curl line, if you enable authentication on jenkins you will need to create a user that has "build" level permissions and put it into that line. Also replace my example token with yours.

All right! Now we have it so that using ssh you can clone the hub repo and work on it. When you push back to hub it triggers Jenkins to build the tests. Then jenkins will take a copy, run nagios -v to test it, and if it passes it will tell the live config to pull the new updates... and once that is done the live config -HUPs nagios for us.

AMAZING!!! But not done yet.

Step Four: Final Boss^WConfigs

Ok. This might seem just about perfect but there is a catch. The default Nagios config uses absolute pathing to all of it's files. This means we need to modify some of Nagios' configs to properly allow Jenkin's testing to read all the proper files.

This process should be as easy as taking all the links at the top of the nagios.cfg file and just making them relative to the main config. For example, here is the head of my config minus comments;

$ egrep '^[^#]' nagios.cfg | head
log_file=/usr/local/var/lib/nagios/nagios.log
cfg_file=objects/commands.cfg
cfg_file=objects/contacts.cfg
cfg_file=objects/timeperiods.cfg
cfg_file=objects/templates.cfg
cfg_dir=systems
object_cache_file=/usr/local/var/lib/nagios/objects.cache
precached_object_file=/usr/local/var/lib/nagios/objects.precache
resource_file=/usr/local/etc/nagios/resource.cfg
status_file=/usr/local/var/lib/nagios/status.dat

Notice the gotcha in there!!! This one stuck me up for about two hours. I was unable to get Nagios to accept any relative path for the resource.cfg file. This introduces it's own gotcha but most people don't need to edit this too frequently.5

To explain a few other lines;

cfg_file=objects/commands.cfg
cfg_file=objects/contacts.cfg
cfg_file=objects/timeperiods.cfg
cfg_file=objects/templates.cfg
cfg_dir=systems

This is referring to /usr/local/etc/nagios/objects. I store all the default config files in there, commands, contacts, time periods, ect.. However I choose to put all my actual system, switch, & device configs in systems. I store personal templates and copies of all the defaults for reference in templates and then when I want to add a new group of systems I just copy a template to systems and fill it out. No need to edit the nagios.cfg every time.

Here is the layout of my nagios config directory;

$ tree nagios_configs 
nagios_configs
├── cgi.cfg
├── htpasswd.users
├── nagios.cfg
├── objects
│   ├── commands.cfg
│   ├── contacts.cfg
│   ├── templates.cfg
│   └── timeperiods.cfg
├── resource.cfg
├── systems
│   ├── chunkhost.cfg
│   ├── lazylopranch.cfg
│   └── shells.cfg
└── templates
    ├── printer.cfg
    ├── switch.cfg
    └── windows.cfg

The End?

So there you go! It's not perfect because of the listed caveats below. Someone malicious or even a sysop who is ignorant of the fragilities of the system can break it fairly easily trying to be tricky or cool. You could get around most of the git merge conflicts problems with hooks though.

The cool part though is this system is FAST. On a good build I push a change, by the time I can get to the web page to check the build it's already deployed to production.

There will probably be a follow up when I figure out how to make the setup a bit more solid, this really was a few hours of hack.

If you think I've missed anything feel free to drop me a comment.

Bonus points

Got someone who doesn't get git and can't remember to push? Hate the extra command?

Go into your personal repo and make git a little more SVN, for better or worse.

cd my_nagios_configs
echo 'git push' > .git/hooks/post-commit
chmod 755 .git/hooks/post-commit

How do you feel about circular dependencies?

If they are your thing then use the Nagios Jenkins Plugin to make Nagios check Jenkins and throw up alerts when your Nagios configs fail their tests!

Caveats

Never use git push --force

If you ever ever EVER EVER do anything that requires a git push --force on hub like an --amend then may god have mercy on your soul. You know how they tell you that push --force is bad and breaks things when you first learned git? This is that exact use case where it ruins everything. Just don't do it, the point of nifty automation is to make your life easier. Suck it up and let your tree be nice and linear, have mistakes, and be "bloated"; Git has some great storage ratios across lots of minor commits.

The test links to the prod version of resource.cfg

You are always testing against the live version of resource.cfg, which will usually be the HEAD^ version but if you break the build it could be even farther off. If you break the build on the resource.cfg file, the bad version will push, nagios will fail to restart properly, and then your NEXT build will fail in Jenkins and refuse to push so you will have to go into the server, test by hand, and then pull from hub manually. DAMN IT. I consider this to be a fairly major flaw that prevents me from wanting to deploy this in a professional environment.


  1. You'll want to set up authentication, general security, and maybe even want to restrict access by firewall to jenkins in the long run. Jenkins is a well tested system but left unsecured and open on the internet this system can be invoked to execute arbitrary code in a snap. 

  2. I've noticed that if you put a space in your Jenkins project name it puts a space in the path to the "workspace" that Jenkins uses to test and deploy from. This can break things from time to time so to error for safe sides don't do this. 

  3. Don't use a tilde in Jenkins paths. It doesn't like them even a little bit. It probably has to do with it's cross system compatibility. 

  4. You might be wondering why we have Jenkins go into the live config and run a pull from hub instead of pushing the configs to the live repo but that's just a nuance of git and most version controls that I have used. They don't like having things 'pushed' into them if they have a working copy. Only pulls will properly grab the changes and merge them into the working copy. That's why the hub we have uses the --bare flag. If You go to check out the hub repo you will see that it instead has the normal contents of .git just lying around instead of the normal files. 

  5. Yes I consider this be a major flaw, read the CAVEATS section. 

Tagged ,

Honing my Git-Fu Part 1

Backstory

My git-fu sucks. I have to use an awesome git tool called SourceTree to do the git wizardry that I do. It's totally free and for the Mac so if you want to just jump into git and have expert features clicks away go download this. I bought it back when it cost money but now you can have it for free. I'll wait…

Anyways, I've been rolling around in the lap of GIT/SourceTree luxury these past months; clicking away and using features I only wished SVN could ever touch. However when jumping around between machines and VMs it would be faster to just use the command line. Now a days I'm now on the development team of a well sized open source project and having to fumble around git & github while testing submissions and making patches to help other people test is just NOT COOL. I think it all came to a head when the main project maintainer started flaunting some of his git-fu when submitting and fixing patches… well honestly since I just love cramming as much into my head as possible I thought I would hone my git foo.

Now back when I bought the McCullough and Berglund on Mastering Git - O'Reilly Media video's while they were on stupid sale and decided to double up with the newly released version of Version Control with Git, 2nd Edition - O'Reilly Media. Time to get my learn on.

What? Studying DISASM, Developing software, working a full time job AND now deep studying a VCS is too much? PISH, I'm single and have the time.

The point

SnowLprd wanted me to get some documenting on and take notes on what I find useful. Over the next month or so I am going to litter this blog with some posts on the "next steps" for git. While I organize my thoughts and really get my git-fu on it may be a bit all over the place. Sorry to anyone who finds this too rudimentary at points but I am going to try to focus on skimming the core concepts while posting lots of gotcha real world commands and why you would use them.

Starter Commands

I'm not gonna put up how to clone a repo here. SRSLY if you haven't gotten past 101 I'm not going to be useful. This is supposed to be 105b.

git remote set-url origin git@github.com:onlyhavecans/pelican.git
git remote add upstream git://github.com/getpelican/pelican.git
git remote add MyExBF git://github.com/justinmayer/pelican.git

Ok, so this one could be obvious but double check to make sure you have everything linked properly. You want the the R/W link to any repos you are pushing to. You might want to change your auth or connect style if you've say… been made a repo maintainer. For sakes of safety/neatness you should still treat your upstream as R/O and make all your changes in push requests and patches. Also add the R/O of your major contributors and co-developers so you can test their patches easier.

git pull upstream master && git push origin

There really should be a shortcut in this for git. Maybe I'll discover it later. I don't do my work on getpelican/pelican, I have the onlyhavecans/pelican fork! Each time I want to work on something I branch, code, commit, push, pull-request. It's an endless cycle. I never really touch my master, I just want to keep it up to date with the upstream. I don't know an amazing shortcut for that so the above micro shell script does the trick.

git push origin --delete <branchname>

I branch for every single patch to maintain tree neatness… however this becomes ugly fast on a good code month. This quickly dumps a branch and deletes it from github. Don't do this before it's been merged into master though, you are asking for pain the first time you make that mistake. Try to remember, branches aren't too expensive disk-wise so don't g too crazy on deleting them.

git add -p <file||.>

Don't add whole files to your staging area! Jeez! Who does that anymore! Be 37337 and review every change you make as you add to staging by committing at the patch level instead of at the file level by using the -p flag

git log HEAD^^ -p

This one took me a minute to figure out. So git show shows the patch for the last commit with patch if applicable but what happens when I want to spew the patches from the last three commits on someone else's branch to get a quick idea of what they are fuffng up? This one! Don't blindly follow my ^^ syntax either, learn about it below.

git commit --amend -c HEAD && git push --force

Ok, for bigger things --fixup or --squash might be better but when you open up a PR and instantly realize you are a total moron and everyone is gonna see how dumb your are because of your misspelling/typo/misscommited line… this squished one liner will recommit your quickly staged fix (letting you tweak the commit message in case that's your problem too) and replace your commit in the Pull Request with this much more awesome one.

Concepts to think of

Revision bad4dad

All revisions are named with SHA-1's of the actual commit contents. Then it uses something called treeish to sparse that down to the first 5-8 unique (to the repo) hex. That's why your commit's name is bad1dea. Obviously since repo's are distributed there is no way to create linear counters so all repo's are actually a linked list of commits. Everything from branches, to tags, to HEAD is just pointers to SHA-1.

HEAD^

Now then what's the ^? Well everyone knows HEAD is an alias to the most recent commit. Well you can add the ^ to any repo name and it's now the previous one. So bad1dea^ means the previous bad idea, while HEAD^ literally just means the previous commit. What's better is it's stackable HEAD^^ is two back. Now you probably don't wanna stack 10 ^'s so just use the shorthand HEAD~10 for ten revisions back.

Ahhh? History is a Linked list! so this traverses the linked list and returns all the revisions! Good for putting together a change log. To just see the last two commits git log HEAD^..HEAD Yea? Thats the cool stuff.

Tagged