Organizing Puppet Code

I feel like every team I talk to, at some point decides they need to blow up their Puppet code base and apply all of the things they've learned to their new awesome codebase. Well, we're at that point in my shop and there's a small debate going on about how to organize our Puppet modules.

This is really not meant to be a mind-blowing blog post, but more of a catalog of thoughts for me as I make my argument for separate repositories for each Puppet module. A few background items.

  • We'll be using the Roles/Profiles pattern. What I'm calling "modules" are the generic implementations of technologies. These are the modules I'm suggesting go into separate repositories. I'm OK with profiles and roles co-existing in a single repository.
  • We're coming from a semi-structured world where all modules lived in a single SVN Repository. Our current deployment method for Puppet code is a svn up on the Puppet Master.
  • We'll be migrating to Git (specifically Stash)
  • We'll have multiple contributors to the code base in 2 different geographic locations. (Chicago and New York for now) The 2 groups are new to each other and haven't been working together long.

I think that's all the housekeeping bits. My reasons for keeping separate Git repositories per modules are not at all revolutionary. It's some of the same arguments people have been writing about on the web for awhile now.

Separate Version History

As development on modules move forward, the commits for these items will be interspersed between commit messages like "Updating AJP port for Tomcat in Hiera". I know tools like Fisheye (which we use) can help eliminate some of the drudgery of flipping through commit messages, but you know what else would help? Having a separate repo where I can just look at the revision history for the module.

Easier Module Versioning

With separate repositories, we can leverage version numbers for each release of the module. This allows us to freeze particular profiles that leverage those modules on a specific version number until they can be addressed and updated. With two disparate teams, this allows them to continue forward with potentially disruptive development, while other profiles have time to update to whatever breakage is occurring.

Access to Tools

Tools like Librarian-Puppet and R10K are built around the assumption that you are keeping your Puppet modules in separate repositories. I haven't done a deep dive on the tools yet, but from what I can tell, using them with a single monolithic repository is probably going to be a bit of a hurdle.

Easier Merging/Rebasing and Testing

The Puppet code base is primarily supported by the Systems team. The world of VCS is still relatively new to the Systems discipline. As we get more comfortable with these tools, we tend to make some of the same mistakes developers make in their early years. The thing that comes to mind is commit size and waiting too long to merge upstream. (Or REBASE if that's your thing) Keeping the modules in separate repositories tightens the problem space your coding for. If you need create a new parameter for a Tomcat module, a systems guy will probably

  1. Create the feature branch
  2. Modify the Tomcat module to accept parameters
  3. Modify the profiles to use the parameters
  4. Test the new profiles
  5. Make more tweaks to the tomcat module
  6. Make more tweaks to the profiles
  7. Test
  8. Commit
  9. Merge

With separate modules, the problem space gets shrunken to "Allow Tomcat to accept SHUTDOWN PORT as a parameter". We've removed the profile out of the problem scope for now and have just focused on Tomcat accepting the parameter. Which also means you need a new, independent way to test this functionality, which means tests local to the module. (rspec-puppet anyone?) This doesn't even include the potential for merge hell that could occur when this finally does get committed and pushed to master. Now I'm not naive. I know that this could theoretically happen even with separate modules, but I'm hoping the extra work involved would serve as a deterrent. Not to mention that gut feeling you get when you're approaching a problem the wrong way.

In favor of a Single Repository

I don't want to discount that there could be some value in managing all your code as a single repository. Here's the arguments I've heard so far.

It complicates deployments of Puppet Code

True that. Nothing is easier than having to execute an svn up command......except for running a deploy_puppet command. Sure you'd have to spend cycles writing a deployment script of some sort, but if that's a valid reason then we're just being lazy. I might be being terribly optimistic but it doesn't seem like a hard problem to solve.

In addition, I've always preferred the idea of delivering Puppet code (or any code for that matter) as some sort of artifact. Maybe we have a build process that delivers an RPM package that is your Puppet code. A simple rpm -iv or yum update and we've got the magic.

It complicates module development

Sometimes when people are developing modules, their modules depend on other modules. I extremely dislike this approach, but it is a reality. You would now have to check out two separate modules and all of their dependencies in order to develop effectively.

Truth is, this sounds like bad workflow. A single module shouldn't reach into other modules. In the rare event that it does (say your module leverages the concat module) then these dependencies shouldn't just be inferred by an include statement. They should be managed by some tool like librarian-puppet, because in reality, that's all it is. Every other language has an approach to solving dependency management (pip, gradle, bundler and now librarian)

What's Next?

With my thought process laid out with pros and cons, I still feel pretty strongly about separate repositories for each module. Another solution might be to create some scripts that manage the modules in a way that can fool the users that care into thinking that they're dealing with a single repository. But this tends to defeat some of the subliminal messaging I hope to gain from separate modules. (Even though that's probably a pipe dream)

I'll be sure to post back what becomes of all this and if the team has any other objections.

Video from Opening the Clubhouse Doors C2E2 Panel

I'm a little late in posting this, but better late than never. The panel that I moderated for C2E2 finally has video footage up. It was a great discussion with a bunch of really great people. Scott Snyder was so incredibly humble and gracious. He's lucky I hadn't started reading Batman Eternal before the panel. I'm so in love with that book I'm not sure I could have prevented myself from bear-hugging him. Anyways, I digress. Check out this great talk.

Rethinking Stability - Part 1 Review of In Search of Certainty

Stumbling through the web, I found this book club called ReadOps. It's an amazing idea and our first book to read is In Search of Certainty by Mark Burgess, a seriously smart man. We're reading the book in sections and with a bit of effort, I was able to get through part 1. Below is the writeup I did for ReadOps. If you're in the IT Field, ReadOps might be worth checking out.


Part 1 of this book was rough, but I promise that it gets better in the later chapters. The principle issue I have is the amount of depth that Burgess goes into to setup his arguments. There are significant correlations between his work in physics and its history, but I don't find it useful beyond the 2nd or 3rd paragraph. Now with that being said there are a few points that I find absolutely stellar.

How We Measure Stability

The sections on stability really challenged my thinking on how we measure stability. In general, I've always measured stability throgh the ITSM Incident/Problem Management) processes. But Burgess struck home for me when he says that what we're actually measuring in these processes are catastrophes, not stability.

If I had the right models for stability, I'd recognize the erratic memory usage patterns of the Java Virtual Machine (JVM). Those swings/fluctuations would give me an idea on the stability of the JVM. (I'm also mixing a concept he talked about in regards to scale, but that's a whole different thing) Instead, I don't pay attention to the small perturbations that lead up to the eventual OOM or long pause garbage collection. Instead the OOM triggers an incident ticket, which then gets tracked and mapped against our stability, when truth be told, stability was challenged much earlier in the lifecycle.

I'm not sure if ITSM has controls to deal with these types of situations or not. In my specific case above, an incident ticket might not have been warranted due to the fact that memory usage can be self-healing through garbage collection. But without defined thresholds and historical data to trend against, it would be easy to see miss an incident or situation where memory usage whent from 40% -> 75% -> 44%. Sure memory usage dropped significantly, but it's still up 4% from where we started. What do I do with this information? I guess that's where clearly defined thresholds come into play.

A Stability Measurement

With all that being said, I wonder if there's the possibility to measure stability into some abstract value or number. (Maybe this has already been done and I'm late to the party?) I think a lot about Apdex and how much I love it as an idea. But for me, as a Systems guy, its at the wrong scale and it introduces components that I have no control over. (Namely the client browser and everything that happens inside it) What would be incredibly useful though was some sort of metric for ServerDex. I'm imagining taking a range of values, deciding which ones fall outside the desired thresholds and applying some sort of weighted decaying average to it. (I'm literally just spitballing here) That could give Systems folk some sort of value to track against. It be nice to be able to take several of these measurements and combine them for an approximation of the stability of our system as a whole.

Ok, time for me to hit the hay.

C2E2 is Almost Here

Wow, I can't believe that C2E2 is almost here! The con has definitely become one of my favorite events of the year and I'm glad that CNSC got the opportunity to work with them again.

This year I'm also excited to be moderating a panel entitled "Opening the Clubhouse Doors: Creating More Inclusive Geek Communities". If you're going to be at C2E2, for selfish reasons, I highly recommend you check it out.

Date: Friday April 25

Time: 6:30pm - 7:30pm

Location: S401CD

On the panel will be

  • Michi Trota, blogger, essayist, fire spinner and general bad ass.
  • Mary Robinette Kowal , Hugo Award-Winning Sci-Fi and Fantasy author.
  • Mary Anne Mohanraj founded the World Fantasy Award-winning and Hugo-nominated magazine, Strange Horizons.
  • Karlyn Meyer is an attorney who uses her vacation days to work at PAX and game development conferences. She studied intellectual property and the law’s application to gender and sexuality
  • Scott Snyder is a comic book writer working for the likes of DC and Marvel as well as creator owned titles. You can see Scott's do his thing on the pages of Batman and The Wake.

As you can tell, this is going to be an awesome panel. Some of our panelists were together last year at C2E2 for the Excorcising the Spectre of the Fake Geek Girl panel. Video link below.

Brew is Failing with Crazy Error

If you're on Mavericks and you're using Homebrew, you may have experienced a weird error message.

/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/rubygems/coreext/kernelrequire.rb:45:in `require': /usr/local/Library/Homebrew/download_strategy.rb:88: invalid multibyte escape: /^\037\213/ (SyntaxError)

This seems to be caused by an update to Ruby version 2.0 as part of Mavericks. All you need to do is make sure that Homebrew points specifically to the 1.8 version of Ruby.

Edit /usr/local/bin/brew

Change the first line from

#!/usr/bin/ruby

to

 #!/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby

Run a brew update afterwards. If brew update works, you're good to go. If it fails, you might need to go nuclear on this bitch. Understand this will probably trash any custom formulas you have.

cd /usr/local
git reset --hard origin/master
git clean -df

Hopefully that solves your problem.

Postgres, Django and that Damn Clang Error

I'm migrating to PostGres for one of my Django projects. (From MySQL) I'm writing this more as a note for myself, but if someone else finds it useful, go for it. If you've done this on a Mac, you may have seen the following errors.

Error: pg_config executable not found.


If you've found that error, then you may not have Postgres installed. If you do have Postgres installed then make sure the installs bin directory is in your path. If you don't have Postgress installed, the easiest way is Postgres.app. After installing Postgres drop to a terminal and add a new value to your PATH.

export PATH=$PATH:/Applications/Postgres.app/Contents/Versions/9.3/bin
pip install psycopg2

Don't feel discouraged when this fails again. Because it probably will. The message is extremely helpful if you've got your Ph.D.

clang: error: unknown argument: '-mno-fused-madd' [-Wunused-command-line-argument-hard-error-in-future]
clang: note: this will be a hard error (cannot be downgraded to a warning) in the future
error: command 'cc' failed with exit status 1


It may not be downgraded in the future, but today is not tomorrow. So lets hack this bad boy.

Things you need:

  1. If you don't have it already, download Xcode and install it.

1a. Drop to a terminal and install the command line tools with

xcode-select --install

  1. If you already had Xcode installed and it's version 4.2 or earlier, skip ahead to step X. If you downloaded Xcode in step 2, you'll need to install some additional compiler tools that were removed from Xcode. The best way is to use Homebrew. (You are using Homebrew RIGHT?)

    brew install apple-gcc42

  2. Once that's complete. If this works, you're done. If not (and it probably won't) move on to step4.

    pip install psycopg2

  3. Set an environment flag to skip the BS compiler flags being used.

    export ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future
    pip install psycopg2


With any luck, that will result in a successful install.

IT and the Empathy Deficit

This post is REALLY late, but I think the topic is still relevant, even if the trigger events have faded in our memory

The Information Technology field is completely devoid of any ability at self-reflection. The whole damn thing, from companies to board of directors, to developers, to system admins. How easily and quickly we can wag our finger when someone else fails, yet when we ourselves fall down, there’s a “perfectly logical explanation”.

In case you were under a rock on last Friday, many of Google’s services went down for an extended outage. I know for our fast paced world of hyper-connectivity, 25 minutes without email or documents is the end of the world. There’s the entrepreneur who finally got his chance to pitch in front of a venture capital firm, but couldn’t get to his presentation. The college kid that was trying to print his assignment before making a mad dash to beat the deadline. I get it, these services impact our lives in major ways.

But it’s alarming to see how the people who should understand most, are the first to pile on. Yahoo just couldn’t help themselves and tweeted about the issue multiple times. They have since apologized but honestly,at this point who cares.

But as the Twitterverse collectively freaked out everyone in my office was calm as a cucumber. Sure we couldn’t access email, but we knew Google would fix the problem and be back up as soon as possible. How did we know?

Because it’s what we would do.

News flash. Sometimes people make mistakes. Sometimes process fails. Sometimes gaps we didn’t know about are found. Sometimes test cases are missed. As a developer, tester or system admin, have you never made a mistake? Have you never let a bug slip in to production? Have you never under-estimated the impact of a change? If you’re perfect, then this message isn’t for you. But if you’re like the other 99.999% (see what I did there?) of people in our field, I’m sure we can agree on a few things.

  • Google’s uptime is pretty damn good.
  • Google is run by some pretty smart people.
  • Even smart people can be fallible.
  • Downtime is a human tragedy. We should treat it with respect.

That last one sounds crazy, but seriously. For someone on that Site Reliability Team, the outage wasn’t a laughing matter. It probably doesn’t feel good to know that the Internet is collectively dismayed and disgusted by a mistake you made, even though 50% of people wouldn’t understand the mistake if you explained it to them. Instead of ridicule, we should encourage open dialogue about how mistakes like this are made, so everyone, not just Google can learn from them.

Outages are learning opportunities for everyone. Why did it happen? Was it a tools failure? I’m sure others would like to know if it’s a tool they use as well. Was it a process failure? Open dialogue about the failures of traditional IT Operations shops and their failures had a huge hand in forming the DevOps movement. Was it human error? Why did that person think the action they took was the right one? If it made sense to them, it will make sense to someone else, which means you might have a documentation or a training issue.

All of these problems are correctable but only if we feel comfortable talking about our failures. This constant ridicule and cynicism our industry has when someone fails threatens the dialogue necessary.

Google has shared some details about the outage, and I’m happy to say it seems to be a growing trend among companies, but what about at a lower more personal level?

I challenge those in our field.

Be fallible
Be open with your failures
Get to the heart of why the failure happened. Don’t just call it a derp moment and move on.
Recognize when someone is trying to do these things and encourage it.

Taking Back My Wasted Time

I was on the train today and the worst possible thing on the planet that could happen to me happened. My phone died. Normally I have a contingency plan for that, but all of them fell through and I was forced to ride the train in total silence, with nothing but my imagination to pass the time. This is the perfect example of wasted time. And despite all the iPhone’s ways of keeping me connected to the world, the single greatest accomplishment of the mobile era, is helping me reclaim those wasted moments in my life.

I’m a productivity nut. But how I categorize productivity might be different than the usual definition. Things I categorize as being productive that might be surprising are:

  • Reading books
  • Watching television
  • Playing video games

Why are these productive? Because living a good life also requires having some fun. And in this new, always on world, it can be easy to lose site of that. So in order for me to live my life balanced, I actually have to put these fun things in my to-do app because if I don’t, I may not make the time for them. So watching 30 minutes of the Daily Show during a train ride is a huge win for me. I’ve now made that wasted time, productive and without the added guilt of thinking “there are 40 other things I could be doing right now.”

In the same light, there are some things that I categorize as unproductive yet they still need to be done for various reasons.

  • Cleaning the house
  • Grocery shopping
  • Cooking dinner

These are things that HAVE to be done in everyone’s life. But they’re unproductive to me because

  1. I can find other ways to get the same result. (Take out?)
  2. The end result doesn’t phase or impact me in a really meaningful way. (So there’s dishes in the sink. I don’t mind, I’ll wash the dish I need when I need it)

Fortunately I have a wife that very much cares about these things. She saves me from my own slothfulness. But I make these tasks bearable by combining them with mobile devices to also make them productive. I’ll wash dishes, cook dinner and grocery shop all in the same day if I can also listen to my podcasts or audiobooks. Because those activities are deemed productive and therefore have saved me from what would be wasted time. (Although that time is greatly appreciated by the Mrs.)

I try my best to guard against this wasted time. It’s so bad that I carry 3 devices just to make sure it doesn’t happen.

  • iPhone 5 - My goto device for reclaiming wasted time. Works in just about any scenario. Sometimes excessive use prevents you from having all the juice you need when you need it. So in case my iPhone dies, I have as a backup ….
  • iPad 3rd Generation - Next best thing to the iPhone. I don’t have the 4G model, but if my phone has enough juice I can tether. This allows me to do some work if I need to or just catch up on some digital magazines (Wired, Newsweek, Time) or read my RSS Feed at a more comfortable scale. But the 3rd Gen iPad is heavy (First world problem) and can be a pain to use if I don’t have a seat on the train. Not to mention the frigid Chicago Winters can make operating the touch screen a challenge on some mornings. For those days I have ….
  • Amazon Kindle E-Reader (2nd Gen) - The Kindle is a great lightweight device. Battery life is great and with buttons I can operate it with gloves on in the cold. It’s also incredibly easy to use one handed.

Despite these gadgets, the perfect storm occurred today and I was forced to stand in silence, watching others be productive. I mentally created a mind map for this post, which made me feel a little better, (Especially since I’m actually writing/posting it) but all-in-all it was a frustrating experience.

Good or bad, the mobile revolution is pushing us to be busy bodies both in our work and our leisure. The conversations about mobile are always around being connected or disconnected from the world around. For me, it’s just about getting shit done.

Parsing the Contents of Mailq

Every so often you come across that task that you think is going to be insanely easy. But alas, you roll up your sleeves, get into the nitty-gritty and discover a twisted twisted web of minor problems. That's been my experience with the Postfix mailq.

I wrote a mailq log parser in Python. The mail queue is where Postfix and Sendmail dump emails that can't be sent or processed for various reasons. It's a running log of entries that can be dumped out in plaintext via the mailq command. I thought this task would be simple, but a few hurdles I ran into.

1. The log file is a variable line length.

A record could span multiple lines. That means you can't simply iterate through the file line by line. You need to figure out where a record beings and ends. So far from my testing it appears the line length is based on the reason the mail is in the queue. Which leads me to item #2.

2. Figuring out why the record is in the queue

This was trivial, but it was an odd design choice. An item could be in the queue for various reasons. There is a special character at the end of the queue ID that tells you the reason it's in the queue. * means one thing, ? means another and the lack of a special character means yet another. Once you've parsed the queue id out, it's trivial to check the last character, but why not just make it a separate field?

3. Different versions of Postfix have slightly different outputs

As soon as I ran a test in our QA environment I learned that different Postfix versions have slight modifications to the output of the mailq command. The annoying part is that the updates aren't substantive at all, but just change for change sake as far as I can tell. Now the email address is blanketed by <> characters. The count of the number of items in the queue are in the beginning of the file instead of the end. And the text describing that number changes its wording just a tad. "Total Requests" instead of "Requests in Queue". Not very useful.

4. The date doesn't include a year

I mean...really? And the date isn't formatted in a MM-DD-YYYY format. It's more like "Fri Jan 3 17:30". So now you're converting text in order to find the appropriate date. This post is timely too because the beginning of the New Year is where this is really a pain. The fix I'm using so far is to assume it's the current year and then test the derived date to see if it's in the future. If it is, fall back to the previous year. This assumes you're processing the queue regularly. It's an auditor's nightmare.

None of it was terribly difficult, but more more difficult than it needed to be. It's as if they wrote the mail queue to only be parsed by humans. I'll be working on my MailQReader for a little bit because I have a need at work.

Is DevOps a Position?

I’ve heard a lot about the term DevOps. Mostly from employers or from technical recruiters looking to fill these roles. When I hear people talk about DevOps, they’re largely talking about Chef, Puppet , CFEngine or just more general configuration management. While I believe the whole Infrastructure as Code movement is extremely helpful, that’s not the end-all be-all of DevOps.

The more I learn about the DevOps movement, the more I realize it’s already falling victim to the bandwagon types in our industry. People are scrambling to build DevOps teams in addition to their development and operations teams, which defeats the purpose of DevOps entirely. DevOps is not a position, but a mindset. It’s an organizational structure. It’s a methodology. The idea of taking developers and operations staff and destroying the barriers and silos that exist between them is key to the DevOps movement.

People are finding different ways of doing this, but I think one of the best solutions might be embedding operations staff members into stream teams. This will go a long way to ensuring that the needs of the Operations staff are being considered during the development cycle and maximizing collaboration across the organization. It also pleases the separation of duties concerns of auditors, regulators and general compliance wonks.

Building a new DevOps team doesn’t work for a few reasons. First, it simply replicates the issues that currently exist, which is silos of staff members who often have disconnected goals and incentives. The goal of Operations is to keep a stable system. What better way to stabilize a system then to thwart change? The goal of developers is to deliver new features and functionality to the end user. You can’t do “new” without “change”, so these two goals are instantly at odds with one another.

If you add a DevOps silo to this picture, it will naturally land right between these two tribes. The DevOps team’s existence lends itself to the idea that it’s supposed to bridge the divide, but in reality it will become a dumping ground. Developers don’t need to worry about integration or its impact on the system, because that’s DevOps’ job. Operations doesn’t need to worry about how new deployments will impact production, because that should be vetted out by DevOps in the QA phases. Before long DevOps is reduced to simply serving as a QA Operations team and Release Engineering. This doesn’t solve the problem though. The poison still runs through the veins of the organization. You’ve just taken the antidote and diluted it.

Like a lot of problems in the world, the issue boils down to incentives. Not just monetary incentives, but the actual goals of the team and the company. The silos you build, the more you fragment the overall goals of the organization. Silos lead to blinders on to the goals of others. Imagine a team of 4 people building a car. The goal is to make a car that is “fast and cheap”. Then you split the group into 2 teams and give one group the responsibility of “fast” and the other group the responsibility of “cheap”. You can imagine the outcome.

Whether you agree with my approach to DevOps or not, what is undeniable is that DevOps is not a job title. To implement DevOps in your organization, it does require a very specific skill-set. You need developers who understand systems and the impact design and coding choices have on the servers. You need administrators that understand code at a deeper level than simple if/then/else constructs. But rebranding your people under the DevOps moniker, or worse, creating a new DevOps team is no solution.

Invalid Syntax When Using or Installing S3Cmd Tools

I installed the s3 tools on a new machine that was spun up for a testing environment. I was quickly greeted with an ugly error message.

 utc = datetime.datetime(dateS3toPython(s3timestamp[0:6],tzinfo=pytz.timezone(‘UTC’))  ^
SyntaxError: invalid syntax

Not a very helpful error if you're new to Python. I'm not 100% sure what the problem is here, but it appears that Python version 2.5.2 (and older) have a problem with list expansion combined with a keyword argument.

Here's some quick dummy code I put together to illustrate.

Python 2.5.2

def test(args, **kwargs):
print "Received Args"
print "Received Kwargs"

>>> arg=(1,2,3,4)

test(arg, test='bar')
File "", line 1
test(
arg, test='bar')
^
SyntaxError: invalid syntax
test(1,2,3,4,test='bar')
Received Args
Received Kwargs

But running this same code on Python 2.6.8 (which is in the Redhat Repos) doesn't produce the problem at all.

So the easy fix is to upgrade your version of Python. I've reported the bug to the S3cmd team to address. My guess is they'll just require a newer version of Python. Their current version test only looks for 2.4 or better. Which is probably out date.

Protect Production Environments from Test Environments with IPTables

Thanks to the flexibility of virtual machines, you've probably found yourself with a clone of a production machine being deployed to a test environment. There are a variety of reasons to do this. Maybe you're preparing for an application upgrade, tracking down a particularly nasty bug or building a clone of your production environment for QA.

The fear is always "How do I prevent the clone from acting on production?" It's a very real fear, because it's easy to miss a configuration file. In an ideal scenario you'd have the test environment on a different network segment that has no connectivity to the production environment. But if you're not that lucky, then there's iptables to the rescue!

With iptables you can use the below command to prevent your test host from connecting to production.

iptables -A OUTPUT -m state --state NEW -d  -j DROP

This command will prevent any new connections from being initiated FROM your test server to the server specified by . This is handy because it still allows you to make a connection FROM the banned production box to the target test box. So when you need to copy those extra config files you forgot about, it won't be a problem. But when you fire up the application in the test environment and it tries to connect to prod, iptables will put the kibosh on it.

If you're super paranoid, you can get execute

iptables -A OUTPUT -d  -j DROP

This will prevent any outbound packets at all from going to .

Don't forget that the iptables command doesn't persist by default, so a reboot will clear added entries. To save the entries and have them persist, execute:

service iptables save

Good luck!

mysqldump Command Not Working in Shell Scripts

I was working on a quick and dirty mysql backup script today. It was nothing complex, yet for some reason I could not for the life of me get it to execute. Here's the script in its entirety, minus the obvious redactions.


!/bin/bash

DB=${1}
USER="USER"
PASS="PASSWORD"
BACKUP_DIR="/tmp/sqlbackups"
CMD="/usr/bin/mysqldump"
DATE=`date +%Y%m%d`
BACKUP_COMMAND="${CMD} -u${USER} -p'${PASS}' ${DB}"

echo "`date` -- Backing up database ${DB}"

${BACKUP_COMMAND} > ${BACKUP_DIR}/${DATE}.${DB}.sqldmp
rc=$?

if [ $rc -eq 0 ];
then
echo "`date` -- Zipping up DB"
/bin/gzip ${BACKUP_DIR}/${DATE}.${DB}.sqldmp

echo "`date` -- Deleting old backups"
/bin/find ${BACKUP_DIR} -name "*${DB}.sqldmp.gz" -mtime +5 -exec rm {} \;
else
echo "`date` -- Backup Failed! Aborting"
fi

I continuously received an "Access Denied" error message for the user. Just so I knew I wasn't crazy, I echo'd the command that was being executed, copied and pasted it and voila. Backups. WUT!?!

The current password was pretty long and contained spaces, so I figured, maybe the spaces were causing problems. I created a brand new user on sql.


grant select, lock tables on 'databasename'.* TO 'databaseuser'@'localhost' IDENTIFIED BY 'password with no spaces'

Same results. Works when I copy and paste the command, but doesn't execute through the script.

So to debug the thing, I removed the -p from my command line so that I'd be prompted for a password. DISCO! It worked. WUT!?!?

Now that I switched the password to be a password with no spaces, I decided for shits and giggles to remove the single quotes. Suddenly I'm in backup heaven.

I don't claim to be an expert on the evils of string variables in bash, but my understanding was that quoting the string, inside of a double quote would produce a literal single quote. Based on the output when I echo'd the command line variable, that's EXACTLY what was happening. But for some reason mysqldump just didn't care for that.

Odd, but solved. I saw a bunch of people reporting the same problem on the web with no answers, so I thought I'd post my experiences.

Pandas and Django

I've been playing around with the pandas library recently. For those that don't know, pandas is a powerful data analysis library written in Python. I've never used R before, but from what I hear, Pandas is heavily influenced by R.

My goal is to get some of my data from a Django project into Pandas so I can leverage some of its awesome features. Pandas has a few options for getting data converted to a data frame, one of pandas data types. Pandas can read tabular data, csv files and even accept a RAW SQL query.

Considering I've already got the data I want in my database, the RAW SQL query seems the way to go. There are a few projects out there to help include support for your pandas in your Django project. I haven't used it yet but the django-pandas project looks interesting.

For my purposes though, I just needed something quick and dirty. Writing SQL queries by hand seemed silly considering all of these pretty models I have laying around, why not use them? So here's how I leverage the Django ORM to make life a little bit easier.

First I'll need to update the necessary libraries

import MySQLdb
import pandas.io.sql as sql
from football.models import FantasyPoints

MySQLdb is the connector library I'm using for my database. pandas.io.sql is a parser for converting query results to the pandas data frame data type. The last line is the model from my project.

records = FantasyPoints.objects.filter(statcardseasonyear=2012)
db = MySQLdb.connect('','',' ','')

First I'm going to grab a query set object from the database. We'll use the query set object to produce the SQL for us. Also remember that you're query set isn't evaluated until you iterate it, so we haven't hit the database yet. The second line is setting up our DB connection object.

Now we can use the sql object to fetch our data and convert it to a data frame.

df = sql.read_frame(str(records.query), db)

Our QuerySet object has a property called 'query'. It holds the internals for query construction, but if you wrap it in a str() function call, you'll get the raw SQL. With that, also pass the database connection object and voila, you have a data frame object from your database.

Caveats

There are a few gotchas here though. For starters, the query property is not part of the Public API, so things could change. The documentation says it's safe for pickling, so it's probably not going away, but who knows how it might change. If you're going to use this approach in production code, I HIGHLY recommend you add a layer of abstraction to it. That way if it changes, yo've only got to make your modifications in a single class/function whatever.

Another gotcha is that the str() representation doesn't quote WHERE clauses that filter on strings appropriately. So lets say your query has a filter like
WHERE position = 'RB'

The query property will output the query with no quotes around RB, making the SQL syntax invalid. I haven't spent much time digging into this because I haven't had to filter on a string. With the use of Pandas, my strategy has been to pull filter as little as possible in SQL and do my slicing and dicing in Pandas. (And caching the data frame for later reuse)

With these two big gotchas, your mileage may vary. I happen to do a lot of work in the IPython Notebook, which makes this approach very quick and simple for me.

Oh and speaking of IPython and Django projects, if you are using notebook, you'll want to check out the Django Extensions module. It allows you to launch a notebook with all of your Django Project variables in tact. Any hoot, I've babbled long enough.

Happy Hacking

Why Are There So Many Pythons? From Bytecode to JIT | Toptal

Found a great blog post on the many implementations of Python today. If you only take one thing away from the article, this would be it.

"The first thing to realize is that ‘Python’ is an interface. There’s a specification of what Python should do and how it should behave (as with any interface). And there are multiple implementations (as with any interface)."


How and why tmp_table_size and max_heap_table_size are bounded

tl;dr -- If your query exceeds the lowest of these 2 settings in size, it will write the temp table to disk. These 2 parameters need to be updated together.

How and why tmp_table_size and max_heap_table_size are bounded: ""

(Via .)

Structuring Selenium Test Code

I had the opportunity to use the Selenium browser automation suite for a project at my, now previous, job. Our testing needs weren't anything spectacular, especially considering we had absolutely zero in the way of automated testing for the project I was working on. I figured anything now would just be icing on the cake.

Playing around with Selenium, the very first thing I struggled with was how to structure my test code. I did a bit of research and a bit of my own architecture to produce a solution that worked well for me. Your mileage may vary.

Page Objects

The first major revelation for me was the idea of structuring each web page as an object. Each of these page objects should return another page object during an operation. Sometimes it might be the same page, other times it will be the resulting page of a particular action. This allows you to handle the WebDriver API inside the Page classes and removes the need of handling drivers in your actual test code.

Sample Login Page in Python

class LoginPage(object):

definit(self, driver=None):
self.url = ''
self.driver = driver or webdriver.Chrome()
self.driver.get(self.url)

def typeusername(self, username):
self.driver.find
elementbyid('txtUserName').sendkeys(username)
return self

def typepassword(self, password):
self.driver.findelementbyid('txtPassword').sendkeys(password)
return self

def submitlogin(self):
self.driver.find
elementbyid('btnlogin').click()
return HomePage(self.driver)

def main():
loginpage = LoginPage()
loginpage.typeusername('jeff')
loginpage.type
password('password')
newpage = loginpage.submitlogin()

This quick set of code kind of illustrates the structure of the code. From the client perspective, it masks all of the nasty element searching and driver passing and makes the actual test code (the main routine) pretty clean and legible. Since a lot of testers aren't full blooded programmers, the ease of reading the main routine is probably going to be very appreciated.

Composite Objects and Inheritance

In most situations, a page has several common components. For example, on this blog, every page will have a navigation bar at the top, with menu items. The way I handled this scenario is by making the Navigation bar a composite object of the page. So the navigation is handled by this NavBarComponent. So in my example, I navigate menus by passing a list object that has the name of the navigation tree. So if the menu navigation would be Server->Inventory->Details, I'd pass a list of ['Server', 'Inventory', 'Details'].

class NavBarComponent:

def navigatemenu(self, driver, *args):
for arg in enumerate(args):
hover = None
element = driver.findelementbypartiallinktext(arg[1])
hover = ActionChains(driver).move
toelement(element)
hover.perform()
click = ActionChains(driver).click(element)
click.perform()
page
class = self.getfinalpageobject(arg[1])
return page
class(driver)

The getfinalpageobject method is just a method that I created so that I can dynamically return a Page Object. That's not important however for our discussion here. So now, navigation can be handled by this subclass.

class HomePage(object):

def _init_(self, driver):
self.navbar = NavBarComponent()
self.driver = driver

def navigatemenu(self, *args):
return self.navbar.navigatemenu(self.driver, *args)

Now we have our HomePage object expose the navigatemenu method but simply passes that call to its composite NavBarComponent object. This allowed me to keep my code pretty focused on specific tasks. It also has the added benefit of being able to either update the NavBarComponent independently in the event it's ever changed or we can substitute the NavBarComponent for some specialty Navigation Bar in the event a page behaves differently than others.

I extended this same approach to things like DataGrids. Typically a DataGrid object will be reused from page to page. Sure simple things like column names may change, but just by parameterizing those small changes, you can easily use the Datagrid Object across multiple pages.

Inheritance

Lastly, we use inheritance to eliminate more code. In my project, we had a Server Inventory page, a Storage Inventory Page and a Network Device Inventory page. But all these pages had most of the same components and logic. That allowed me to place all the common actions into an InventoryPage class, which handled the nitty gritty things like, sorting by column name or grouping by column name. These are functional steps that would exist on each page. So by encapsulating all of those items into the InventoryPage base class, I can quickly add functionality to a bunch of different pages at once.

Here's a quick UML diagram of what this might look like.

UML Diagram of InventoryPages

This is by no means the absolute best way to do things. I'm sure there are plenty with more experience in Selenium that have other approaches. But this approach has been easy to maintain, easy to follow and easy to understand. I think that's what most of us are looking for in our Test Suites. Your mileage may vary.

Chicago Nerd Social Club

800

799