Rails Machine - Views and Actions

Passenger 4.0.1 Is Out!

Mon, 06 May 2013 00:00:00 +0000

Passenger 4.0.1 is out and that means we need to add support to Moonshine for it! If you update Moonshine, you'll have everything you need to deploy Passenger 4.0.1 - I highly recommend it. Why? Out of band work (garbage collection)! The Phusion guys blogged about it during one of the betas, but their instructions for setting it up don't work very well with the final release. I think this is my favorite new feature in Passenger and could end up being extremely useful for folks running larger apps that struggle with garbage collection. I got it working this morning on one of our internal apps and decided I would save you the trouble of figuring it out yourself.

First, you need to add passenger to your Gemfile. I don't normally put it in mine since it lives outside of the Rails stack, but you'll need it for this:

gem 'passenger', '4.0.1'

Now we need to add some cool stuff to your config/application.rb. Near the top, after you require rails, add this:

if ['staging','production'].include?(Rails.env)
  require 'phusion_passenger/rack/out_of_band_gc'
  require 'phusion_passenger/public_api'
end

And inside the module where you have all your other configuration, this:

if ['staging','production'].include?(Rails.env)
  config.middleware.use PhusionPassenger::Rack::OutOfBandGc, 5

  PhusionPassenger.on_event(:oob_work) do
    # Phusion Passenger has told us that we're ready to perform OOB work.
    t0 = Time.now
    GC.start
    Rails.logger.info "Out-Of-Bound GC finished in #{Time.now - t0} sec"
  end

end

And one last chunk. In app/controllers/application_controller.rb, you need to add a before_filter to add the header telling Passenger to do its thing:

before_filter :add_out_of_band_header

private

def add_out_of_band_header
  response.headers["X-Passenger-Request-OOB-Work"] = "true"
end

Now, after you deploy that (to staging first, I hope), you should be able to tail the log and see the app running GC every 5 requests. If your app is super busy, you may want to change the setting to more requests, but 5 is probably a good place to start.

I can foresee a lot of other uses for out of band work, almost anything that the user shouldn't have to wait for but that needs to done while you have access to the request (assuming that's something out of band work can do - I haven't checked yet).

Watch That First Request, It's a Doozy

Fri, 08 Mar 2013 00:00:00 +0000

I have a love/hate relationship with the asset pipeline. Ok, it's mostly hate. I applaud what it tries to do, but it takes something relatively simple - javascript and CSS - and makes it painful, mostly by making deploys take forever (even with turbo-sprockets-rails3).

For example, I was working on something yesterday and doing deploys to the staging environment. I noticed that it took forever to be able to start serving requests. Passenger-status showed all three instances were running, and were all stuck serving their first request. A little digging revealed that there were a bunch of node.js processes running compiling assets! What the?! I thought I was precompiling everything! Nope, apparently not.

After a little digging, I found this setting:

config.assets.compile = true

Set that to false and redeployed, and all of a sudden, I was seeing 500 responses for assets that weren't precompiled! I ended up spending about 30 minutes adding a bunch of new javascript and CSS files to the config.assets.precompile list and eventually the app started up fine and I got rid of the unpleasant "first request takes forever" symptom in Passenger.

I think this is most likely going to bite you if you use a gem that provides its own assets and doesn't add them to the precompile list on its own (which it should).

I hope that helps someone avoid some downtime / slow requests in the future, because it took me a while to figure out!

30 Minutes in the Life of Troubleshooting Weird Rails Stuff

Thu, 21 Feb 2013 00:00:00 +0000

Managing hundreds of Rails apps gives us the opportunity to see all of the amazing ways that Ruby and Rails can misbehave. I had an experience so weird last night that I think it deserves its own blog post.

An app that hadn't been deployed in two weeks all of a sudden returned a 500 error for the URL we use to monitor it. Looking at the error, Passenger couldn't find the Rails gem. This is a 2.3 app, and doesn't use Bundler yet, so... this was weird. I've been working with Rails for almost 8 years now and have never seen this before in an app that hadn't been deployed to recently. I could understand if you changed the RAILS_GEM_VERSION line in environment.rb but hadn't actually installed that version of Rails, but that didn't happen in this case.

What did I do? First, I tried the console to see if that could find the rails gem. It worked fine.

Having proved to myself that the rails gem was installed and at least working for one part of the app, I concentrated on Passenger. I uninstalled all the old versions of the gem, then reinstalled the version the app was using, re-ran passenger-install-apache2-module and restarted apache. No luck.

Then I decided to remove rails from the equation and ran rake rails:freeze:gems, which moves all the rails gems to vendor/gems. After doing that, I got the real error. Somehow, rack 1.5.2 had gotten installed, which apparently hates old Rails apps. I removed it, restarted apache (just for luck) and ta-da, the app was alive again and I could go back to sleep.

Other than it happening at 3 this morning, this is why I love working at Rails Machine. If it can go wrong with Rails, we have a front row seat for it. Wait, no, that's wrong. We're on the stage, fixing it and making sure apps stay up and running.

Moonshine, Now 92.6% More Enterprisey

Thu, 06 Dec 2012 00:00:00 +0000

Last week, we added support for Passenger Enterprise to Moonshine, allowing anyone to easily deploy apps with Passenger Enterprise and easily making sure that it’s compiled correctly, licensed and configured to give your app all the shiny new features (OMG, rolling restarts!!).

Upgrading is straightforward. After you purchase your license, download the passenger-enterprise-server gem from your dashboard and throw it in vendor/gems. Then, download your license and put it in app/manifests/templates. Now, in your moonshine.yml, it should look something like this:

:passenger:
  :enterprise: true
  :gemfile: vendor/gems/passenger-enterprise-server-3.0.18.gem
  :version: 3.0.17
  :rolling_restarts: true

We’ve also added support for the new Enterprise-only features:

:rolling_restarts: Defaults to false, but you should really set it to true.
:max_request_time: Defaults to 0. Be careful with this one because there appears to be a bug that kills of passenger instances as they start if they take longer to spin up than the value of this setting. I’d recommend setting it to something extreme like 120 for now (hopefully your app doesn’t take 2 minutes to start up).
:memory_limit: Defaults to 0. We haven’t set this one in production yet, but I’d set it the absolute maximum you’re comfortable running your app (like 500M).
:resist_deployment_errors: Defaults to false.
:debug_log_file: This is a great feature. It moves all the apache errors related to startup or stopped connections to a separate log file. We’ve been setting this to the /srv/APPNAME/current/log/debug.log.

All of these are on the shiny new Default Configuration page on the Moonshine wiki along with all the other settings you can tweak with Moonshine.

We’ve had several Rails Machine customers roll out Passenger Enterprise recently and we’ve heard glowing reports from everyone, especially for the rolling restarts.

Are you using Passenger Enterprise? What do you think?

Heyo From the New Guy!

Fri, 16 Nov 2012 00:00:00 +0000

Hi! My name’s Rit Breisler , and I’m the newish guy at Rails Machine . I’ve got about six weeks under my belt, and the tl;dr of it is that this is a seriously remarkable shop to work in, and I couldn’t have asked for a better bunch of folks to work with.

The slightly longer take is that I came to Rails Machine because I was looking for particular qualities in a shop; I was seeking a place of great challenge where the people were real thinkers, and just as importantly, real doers. I was looking to be in a state of motion as much as possible, and advancing my art toward what I believe is the most important goal for Ops life .

I come from an Ops background, and (as you probably have gathered from my about page on the site) I strongly believe that everyone who does Web Ops (be you Dev, be you Ops) must strive to bridge the gap between what used to be considered very separate disciplines. My role at Rails Machine is allowing me to pursue that goal, and at a rate which is frankly terrifying, but thrilling, all while being surrounded by people who are thoughtful, expert in their disciplines, and most importantly, give a damn. That last one might cause a double-take in some of you, but it’s actually my number one desired quality in a place of employment, and something that I’ve found to be quite rare. Rails Machine has it.

Another brand new thing is that I’m remote for the first time (I call Boston home, and mostly work from there; Rails Machine HQ is in Savannah). If you are considering taking on that challenge as well, I’ve learned a thing or two so far which I’ll share:

GNU Screen is an extremely powerful tool for managing shells on remote hosts, supporting on-the-go ops, but it also happens to be an outstanding remote pairing too. Learn all the screen-fu you can deal with.
Use Campfire for team chat. I cannot tell people enough how vastly superior group chat is to what I refer to as “silo chat” (individuals all having essentially invisible ad-hoc conversations from which no one else on the team can benefit).
Engage in ChatOps. Get a bot going for your Campfire, like HUBOT (see our CTO Josh talk about it , or Github’s Jesse Newland as well , for some persuasive arguments).
Escalate to video chat when you need it, because greater than 80 percent of communication is nonverbal. No, I’m serious. Text is great for pretty much everything, but sometimes you really just need to be face to face. FaceTime, Skype, Google Hangouts, whatever you prefer, but do these (very yes).

Keeping sharp and staying able to ingest all of my new various -Fu isn’t easy, but I’ve been keeping up with the usual suspects (I’ve been reading Web Operations again), practicing, and of course trying to keep balance with all of the normal things (sleep plenty, eat your green vegetables – y’know, stuff we learn from Spider-Man). Still, I’ve learned more in six weeks than I have in the past year. It’s absurd. And it’s great – exactly what I was seeking.

Every day I get to work on exciting software and hardware, learn all kinds of rad things, establish relationships with and deliver value for our customers, and I feel privileged to be here. I was seeking challenge, caring, and geeky camaraderie; I’ve found that in Rails Machine, and I’m delighted to be on board!

A Web Operations View of Election Day

Thu, 15 Nov 2012 00:00:00 +0000

Yes, I know the election was last week, but it’s taken us that long to recover from the sheer awesomeness of how well it went for our customers. No, not the outcome of the election, but their availability and performance throughout.

We have two customers we worked closely with to make sure they were ready for the election rush: Mobile Commons and NationBuilder. Each one will probably end up being their own blog post, but this time, I wanted to talk about how we worked with NationBuilder to make sure things went smoothly last week.

For NationBuilder, last week was about both making sure the main product was ready for everything their customers would throw at it in the days leading up to Election Day, but also planning, provisioning and deploying everything needed to launch their Election Center product – a first-of-its-kind national voter registration database open to everyone.

We’re really proud to be involved in our little way with both products, and I want to break down a little of what I think went into making last week so successful (as in, no one got woken up in the middle of the night and things were actually faster than the week before).

I think all of these are applicable to most relationships between Development and Operations teams – it just happens that in this case, the operations team is a separate company:

Teamwork & Emphasis on Performance: We spent the summer and early part of the fall working with NationBuilder to add capacity, and on making sure their performance was everything it could be. We spent a lot of time looking at Scout and NewRelic to see what could be improved and then worked together to make it happen. By election day, by working together to attack hot spots, we were able to cut NationBuilder’s average response time by 33%.
Capacity Planning: With both customers, they were very clear about their goals, and we were able to measure our progress against those goals. We made sure they had the right pieces in place to support the traffic they expected – and the traffic no one could predict.
Frequent Checkpoints: We met weekly with NationBuilder to make sure we were all on track with not only deployment tasks, but performance and feature development. The NationBuilder team did an amazing job fixing performance issues when we pointed them out – to the point that when I pulled the transaction report from NewRelic, almost all the requests were new to the report.
Fix the Problem: The NationBuilder folks are great under duress. It doesn’t happen often, but when things go wrong, we work together to fix it in the moment, and save the root cause analysis for after.
Move Quickly but Deliberately: NationBuilder launched an entirely new product right before the election, The NationBuilder Election Center, and wanted it to be able to handle a lot of load right away. This started as a prototype and ended up being a fairly complex app with a lot of dependencies and a fairly large hardware footprint. By relying on our existing good communication, we were able to scale it up on all new hardware and launch with a lot of press attention and use.

And it warms our nerdy little hearts when we get feedback like this from NationBuilder’s CEO:

I just wanted to say thanks to everyone at Rails Machine for your efforts, particularly the last couple of weeks as we’ve barreled toward election night. The last 5 days were so immensely critical to us. We are working really hard to establish trust in our company, and political tech is notorious for crashing on election day. We didn’t even have a blip, so we earned a lot of cred. You helped make that happen in a big way. Thank you.

Like I said, we’re really proud to have played a small part in the democratic process and in helping NationBuilder and Mobile Commons meet their goals for performance and availability. Now it’s time to get ready for the holiday season!

Red(is) Alert!

Mon, 22 Oct 2012 00:00:00 +0000

We have a lot of customers who use Redis as a part of their infrastructure. Most of them start off using it with Resque to process background jobs. Then, they figure they’ve got keys and values they want to keep somewhere, and well, Redis is sitting there looking all friendly and you’ve already got it configured for Resque, so why not use it for storing things?

And that’s when things get interesting. You see, Redis keeps everything in memory and saves to disk periodically based on configuration. Unfortunately, Redis kicks off a new process during a save, which uses up as much memory as the main process. So, in essence, you always have to use less than 50% of the available memory on your server when using Redis.

Why? Well, as soon as your Redis process grows large enough to hit the magical 50% memory limit, it can no longer save. When this happens, the only thing you can do is foreground save, which leaves Redis unavailable until the save is finished (which on large installations can be more than a minute) or you delete keys until things can save again.

The neckbeards in the audience may be feeling superior at this point, thinking to themselves, “It’s your own fault. You didn’t allocate memory correctly in the first place” or “that’s what monitoring is for.” And you’d be partially correct. The problem is that web apps grow over time, and sometimes get popular very quickly. That means you may end up with more jobs than you have workers to promptly handle – which means the queue grows – which means Redis grows – which means someone’s waking up in the middle of the night to babysit a sick Redis that can’t save because it ate too much. Or, true story, someone (let’s say Resque) decides to log stack traces to Redis and you somehow trigger the world’s biggest stack trace over and over again, causing Redis to explode, leaving little bits of stack traces dripping off the ceiling.

There are things you can do to avoid these disasters, and most of them are not hard (although they may be expensive), and some things to watch out for when monitoring Redis to make sure you’re looking at the right things (because, on top of a tendency to be a naughty little imp when it gets full, it also lies).

Over-Provision Everything

The argument from Redis fans is that if you only configured your servers correctly to begin with, then you wouldn’t have any problems with it. And in exactly one way, they’re right. So go ahead and provision your Redis server like your future in-laws told you to buy diamonds and houses – always get the one twice as big as you think you’ll need because it will eventually feel small.

Get more memory than you think you’ll need, more disk, and have more workers available than you think you’ll need. Just do it.

Always Have a Backup Plan

One way to get around foreground saving on the primary server is to have a standby that you can foreground save on without affecting workers or users. This allows you to babysit a sad Redis without making any other part of your infrastructure sad. Yes, you’re spending for another server that’s the same size as the primary (that part is really important – it needs to be able to take over in case the primary goes away), but your peace of mind is worth it, right?

Also, if you need to restart Redis for any reason, and you have a lot of data, restarting can take a long time. So, you break the slave relationship, have your app talk to the slave Redis, restart the master and then make it a slave of the old slave. You can then reverse that process if you need to restart the former slave.

The other backup plan you need to have is to know what you can delete without causing irreparable harm to your app. Know how to get those keys and delete them (like Resque stack traces). When you hit that 50% memory mark, this should be the first thing you do to try to get back to your happy place.

Monitoring the Right Things at the Right Time

The INFO command returns a lot of information, and a lot of is actually useful. Some of it, though, is wrong. For example, never trust the used_memory_human entry, because I’ve seen it be off by more than 500mb. Always look at used_memory for monitoring and do the conversion yourself if you need to.

You need to set up alerts for yourself (using Scout maybe) that trigger before Redis gets to 50% of available memory, because once Redis passes that mark, things are going to be very sad.

You should also monitor last_save_time and alert if it hasn’t saved in a reasonable amount of time, where reasonable is defined by you. Another option to get this info is to monitor the Redis log file for failed saves (we use the Log Watcher plugin in Scout for this for “Can’t save in background”)

If you’re using Redis for Resque (and you probably are), you need to closely watch the size of your queues, know your average velocity on completing jobs, and do some math to figure out how efficient your workers are. Why? Because you need to know when your workers will no longer be able to keep up with the pace of incoming jobs with enough time to spin up more workers if you need them. You don’t want to get into a state where you have queues on fire (putting Redis into a bad state) and not enough workers to put out that fire.

Look Into Append-Only File

Redis has an alternate persistence strategy, called AOF that eases some of the pain of forking the process to save a snapshot, but not all of it, since the folks at Redis suggest using both. If you’d like to read more on Redis persistence, antirez wrote an epic blog post about it.

Separate Functions

Don’t run your workers on your Redis server. Workers use up RAM that Redis needs to be able to save things. Workers should have their own servers, just like your app server is a separate thing from your Redis server (please tell me it’s separate).

If you’re happily using Redis for Resque, and then want to use Redis for some other nefarious purpose, get yourself another Redis server. Don’t combine them. The two uses are completely different and grow differently.

Have an Expiration Strategy

Redis is in a weird place between “hot” caching like memcached where things fall out of the bottom of the cache when memory is needed and disk-based data stores where you don’t need to store everything in memory. So, the storage strategy in Redis is a little hard to get your head around. You have it because it’s persistent, which is good. But, you also have to have enough memory to keep everything in memory all the time. So, you can’t keep everything in it forever, because it’s also not horizontally scalable (yet).

You need to come up with a expiration plan for how you’re going to expire things out of Redis and either forget them forever or move them somewhere less volatile and less “expensive” (disk is cheaper than RAM). You could do it by activity, by time, whatever, just have a plan and have it built into the code so it can be run if you need to free up memory to make Redis happy again.

Because Redis will get sad. It just will.

In Conclusion

Redis is a great in-memory database. It’s extremely flexible, developer-friendly and easy to get started with. The key with any piece of software is to use it correctly, know what it looks like when it’s about to fail, and how to keep it well-fed and happy. I could (and probably will) have written this blog post about any of the software our customers use, because we’ve had problems with all of them. I hope this post saves you some time and makes you a happier and healthier Redis user!

New Relic + Rails Machine partnership brings free monitoring to customers

Thu, 18 Oct 2012 00:00:00 +0000

It’s with great excitement that we get to announce our partnership with New Relic! This will allow us to provide New Relic Standard for free, forever, as long as you’re a Rails Machine customer. We want your applications to kick ass and run well. New Relic allows us to peer inside the applications we manage and make sure they’re doing just that.

“New Relic, Inc. is the all-in-one web application performance management provider for the cloud and the datacenter. Its SaaS solution combines real user monitoring, server monitoring, application monitoring, and availability monitoring in a single solution built from the ground up. It changes the way organizations manage web application performance in real-time, enabling developers and operations teams to quickly and cost effectively monitor, troubleshoot, and tune application performance.”

And the best part is that it’s free for all customers! To get started, head on over to http://newrelic.com/railsmachine and claim your free New Relic Standard account today.

Official Press Release: New Relic Press Release

Adobe's Source Code Pro

Tue, 25 Sep 2012 00:00:00 +0000

Source Code Pro, A new monospaced font to help cure those tired eyes.

Yesterday, Paul D. Hunt, announced that Adobe was releasing the open sourced monospace font, Source Code Pro. (Adobe announced a lot of new, exciting things yesterday.) And today, the entire Rails Machine team updated their text editor preferences to take full advantage of the pretty.

Source Code Pro seeks to remove the “monotonous rhythm” that seems to pervade many monospaced fonts by relying on the clarity of Adobe’s Source Sans Pro. It ships as a family with six different weights, we used regular in our text editors, and really makes a difference when used in Terminal. Read about the insight that went into Adobe’s first open source font family and then head on over to Source Forge to get started. I put it to use on the Rails Machine site via Google Web Fonts earlier this morning and it’s looking great:

Source Code Pro in the wild.

If you’ve never installed a font before, I’ll step you through the easy process using Font Book on a Mac:

Download Source Code Pro via Source Forge
Open Font Book
Hit ‘command + o’ to locate the font
Find the Source Code Pro folder in your downloads folder (you may need to unzip the folder first)
‘command click’ all the OTF font files and click ‘Open’
Update the fonts in your favorite editor

Enjoy and thanks to Paul and Adobe for their hardwork! You can find Paul on Twitter via @pauldhunt

MongoDB Manipulation, Mastery and Monkey Business

Tue, 28 Aug 2012 00:00:00 +0000

I’m a big fan of MongoDB. I used it for a product at my last company and found it to be easy to manage and deploy and fun to develop with. We have a couple of customers here at Rails Machine that are also big fans of MongoDB and we helped one of them upgrade their replica set today from two nodes with an arbiter to three gigantic servers. You’d think that completely replacing the hardware that runs your database would be painful… but with Mongo, it’s really not. Here’s how we did it.

Since these were new servers, and our customer has a staging environment, we did a lot of testing of the new servers by adding them to the staging replica set. It was really easy to move things around using the Moonshine MongoDB plugin. Some of the tests we ran:

How long does it take for a new node to become a fully functioning secondary?
- Depends on the amount of data, but not as long as you’d expect.
What happens when a secondary unexpectedly drops out?
- Not much. If there was an election, it was so quick we didn’t notice.
What happens when the primary unexpectedly drops out?
- There’s an election and things “flap” for a few seconds. We did this a few times, and elections took as little as 2 seconds and as long as 10.
Does it take longer to sync two new secondaries than one?
- As long as there are as many secondaries as there are new nodes, then no.

After we were happy that the servers were ready, we removed them from the staging replica set and deleted their data. With Moonshine, removing them from staging and adding them to the production deployment was just moving a few lines of configuration around.

I’ve changed the names of the servers and ip addresses for this example, so, let’s pretend that things look like this:

Existing replica set:
- arbiter: 10.0.0.1
- donald: 10.0.0.2
- daisy: 10.0.0.3 – the current primary (pretty much force to be primary because we set its priority to 2, which will come in later)
New ‘super’ mongo servers:
- huey: 10.0.0.4
- dewey: 10.0.0.5
- louie: 10.0.0.6

We did this over two days to make sure the new servers were “happy” and ready to take over all the traffic, but here’s what we did:

We deployed to all the servers to make sure the correct iptables rules were in place so that everything that needed to talk to the new Mongos could, and also to add the three new nodes to the app config (they should be in the config after the two current nodes).
Confirmed that the new servers could connect to, and be connected to by, the old ones (you’ll get an error if things are wrong, otherwise you’ll connect and can run queries.):
- from huey:
  - mongo 10.0.0.2
  - mongo 10.0.0.3
- from daisy:
  - mongo 10.0.0.4
  - mongo 10.0.0.5
  - mongo 10.0.0.6
Now you need to connect to the current primary (daisy for this story) and reconfigure things. Since we always wanted there to be a quorum of “up” nodes in the replica set to keep things from possibly going south (as in, not enough functioning nodes to elect a primary, which I’ve seen before and is unpleasant). We need to first remove the arbiter and add one of the new nodes:
1. config = rs.conf()
2. huey = {_id:4,host:'10.0.0.4:27017',hidden:true,priority:0}
3. config.members.push(huey)
4. config.members.splice(INDEX,1) where INDEX is the index of the arbiter in the members array.
5. config.version++
6. Now, before we commit this, we need to look at the config variable and make sure things make sense:
  - Are all the nodes you want to be in the list of members?
  - Do they all have their host field set to “IP:PORT”?
  - Are the new nodes set to hidden?
  - Do the new nodes all have unique _id fields?
  - Do they all have priority set to 0?
7. rs.reconfig(config)
Now you get to obsessively run rs.status() over and over again until huey is all synced up and a full-fledged member of the set (which will happen when it’s no longer RECOVERING and says SECONDARY). You may see a few things that look like error messages while this is happening:
- “errmsg” : “initial sync need a member to be primary or secondary to do our initial sync” – This almost always means the election is taking place. Wait a few seconds and run rs.status() again.
- “errmsg” : “initial sync cloning db: DBNAME” – This is good! That means the sync is happening.
- “errmsg” : “syncThread: 10278 dbclient error communicating with server: 10.0.0.5:27017” – I saw this one a couple times right after I triggered an election. I think this is the normal “I’ve just triggered an election and am switching connections” message.
- If you see any other errors, google them, because I didn’t see them.
Once the new node is a SECONDARY, you can add the other two (because there are now three healthy nodes, adding two won’t cause an imbalance):
1. config = rs.conf()
2. dewey = {_id:5,host:'10.0.0.5:27017',hidden:true,priority:0}
3. louie = {_id:6,host:'10.0.0.6:27017',hidden:true,priority:0}
4. config.members.push(dewey)
5. config.members.push(louie)
6. config.version++
7. Before we commit this, go through the checklist we went through the first time we did this and make sure we don’t have any typos or other mistakes. If things are cool:
8. rs.reconfig(config)
9. Again, obsessively run rs.status() until everything’s happy.
Once the new nodes are all listed as SECONDARY in rs.status(), you’ve successfully added the new nodes to the replica set. This is where we stopped on day one so we could watch things to make sure everything was fine with the new nodes. But, once the new nodes are secondaries, you can continue with dropping the old nodes:
1. We created a pull request that had the updated app config that removes the two old nodes.
2. We also stopped all the resque workers at this point to make sure we didn’t cause any jobs to fail during the election.
3. Open up the mongo console on the current primary and get rolling!
4. config = rs.conf()
5. For each of the hidden nodes in the members list:
  - config.members[x].hidden = false
  - config.members[x].priority = 1
6. Now we need to give one of the new nodes a priority higher than the current primary’s, so:
  - config.members[3].priority = 3
7. config.version++
8. Look at the config variable again and make sure it’s got all the right stuff in it and then:
9. rs.reconfig(config)
10. Do the rs.status() dance until the new primary is elected. There may be a considerable amount of “flapping” during the election. I’ve seen it take as little as 2 seconds and as long as 10 for a new primary to be elected. Just keep checking rs.status() until things calm down.
After the new primary is elected, we merged the pull request that removes the old nodes from the app config and deployed.
Once the deploy was done and the app was up and running talking to the new primary, it was time to remove the old nodes from the replica set.
1. Connect to the new primary’s mongo console.
2. config = rs.conf()
3. And now we need to splice out the old nodes. For each of the old nodes:
  - config.members.splice(INDEX,1) (where INDEX is the indexes of the old node we’re removing)
4. config.version++
5. Go through the checklist again, and this time, make sure the old nodes are no longer in the members array.
6. rs.reconfig(config)
7. There might be some more flapping here as it disconnects the old nodes. We definitely saw a few seconds of “weirdness” when we did it.
That’s pretty much it!

Overall, the entire process went very smoothly. The only issue we had was when we removed the old nodes, the apps lost their connection to MongoDB entirely and refused to connect. An apache restart fixed that issue. We think it was a “failed” restart during the deploy that didn’t restart all of the passenger instances. That was the only real downtime during the entire migration and it lasted for only a couple of minutes while we restarted Apache.

Having done this process a few times now, and having done this with other database systems in the past, I’m really impressed with how easy it is to manipulate replica sets with MongoDB. It’s a lot easier than I originally thought it would be and while running three instances for a replica set is more expensive than the regular master/slave setup you see with traditional databases, it makes a lot of sense and works really well in the “real world”.

I’d love to hear how other folks have done this kind of thing with MongoDB!

And as a congratulations for getting to the bottom of this post, here’s a photo of a corgi:

A New Machine

Tue, 14 Aug 2012 00:00:00 +0000

It’s that time of year again. Kids are being shuffled back to school after what was hopefully a long and eventful Summer break. Parents are adjusting their morning routines by packing pencils, paper, and lunches — or is it iPads and styluses — into backpacks to begin hitting the road to do battle with the other sleep deprived parents, or maybe they’re exploring plans for a new Fall season of experiments, internal apps, traveling, rebranding, redesigning, reducing, and ping pong tournaments? Because that’s how Rails Machine plans to spend the rest of this year, by improving all the things.

And when I say all the things, I mean all the things. We’ve looked at everything from how we can become more efficient and productive in our efforts to complete customer requests to how we can increase our output and support for our open source deployment tool, Moonshine. We’ve looked at how we can improve the capacity and speed of our servers while reducing our energy consumption. We’re going to be including more automation in our daily tasks and we’re even sending members of our team out on the road to talk about some of the great things we have in the works.

The best part, our customers are going to benefit the most of all from our internal tinkering. We can’t wait to roll up our sleeves and roll out the new machine.

ChatOps - Using New Relic in Campfire

Wed, 18 Jul 2012 00:00:00 +0000

Just awhile back, I wrote about how to build a Campfire bot with hubot. I’ve been doing a lot of work on our very own hubot since then (Claptrap, by name), and it’s been a pretty rad experience.

You’d think that hubot is all fun & lulz if you take a look at the Hubot Script Catalog, but there are definitely some ‘productive’ hubot-scripts in the mix. Today, we’re going to checkout one in particular that is totally #opslife.

Meet newrelic.coffee

We are huge fans of New Relic in general, but it’s even better when you can have easy access to it from Campfire. Check it:

There’s just a few steps to get going. First, we head to New Relic to collect some information:

Login at https://rpm.newrelic.com
You’ll be redirected to a URL like https://rpm.newrelic.com/accounts/XXXXXX/applications
- Make note of XXXXXX, we’ll be using it for HUBOT_NEWRELIC_ACCOUNT_ID
Click through to one of your applications, and you’ll be at a URL like https://rpm.newrelic.com/accounts/XXXXXX/applications/YYYYYY
- Make note of YYYYYY, we’ll be using it for HUBOT_NEWRELIC_APP_ID
From the topbar, click on your account name, then Account Settings
- Select the ‘Data sharing’ tab
- ‘Enable API access’ if it’s not enabled
- Make note of the API Key: 12345, we’ll be using it for HUBOT_NEWRELIC_API_KEY
- If you don’t see Account Settings, you’ll need to get an account admin to do this, or make you an admin

With this information, we can apply it the shell environment:

export HUBOT_NEWRELIC_ACCOUNT_ID=XXXXXX
export HUBOT_NEWRELIC_APP_ID=YYYYYY
export HUBOT_NEWRELIC_API_KEY=12345

And then update hubot-scripts.json to include newrelic:

[
  # other scripts here
  'newrelic.coffee',
  # even more scripts here
]

Restart hubot, and you’re good to go!

Our Special Sauce

One thing you may have noticed about the NewRelic hubot-script is that you can only configure a single application. We highly recommend NewRelic for our managed hosting customers, so this was immediately a problem for us.

To work around, I’ve taken the normal script, and lovingly seasoned it with data from our internal servers. By doing this, we can go from any server on our infrastructure to correct application in NewRelic:

Wrapping up

All in all, we’re pretty happy with this action so far. There’s always room for improvement, but fortunately, we’re just consuming the NewRelic REST API , so it’s only a matter of writing some code to have even more awesomeness in Campfire.

When Things Go From Good to Sad... to Seriously Serious

Thu, 05 Jul 2012 00:00:00 +0000

(this was written by our intrepid Director of Operations, Travis Graham, who was too busy to proofread and edit it, so I did it for him. I swear he wrote, like, 95% of it. I added the top tips and the thing about open office hours. I certainly didn’t have to fix any typos or weird comma things. — Kevin)

Some of the most common alerts that come in are http timeout, memory utilization, server down, and ssh timeout alerts. Depending on the customer and how they’ve built their application infrastructure out gives me an initial idea of where to look. Having the hands on experience of dealing with thousands of these types of alerts and fixing them creates a “run book”, of sorts, to know where to start investigating and what things to look for in order to fix the problem before it becomes seriously serious. Here are some tips for dealing with these basic alerts and how to find and fix them.

HTTP Alerts

The most common cause of http alerts is the passenger global queue has backed up causing the http check to fail.

The first thing I like to check is “top” to get an overview of the current running passenger processes: total number running, their uptime, and memory footprint. If there are no passenger processes, I check to see if a recent deploy has gone out and look to see if apache was restarted or is even running at all. Sometimes a typo makes its way into the code base which causes apache to fail to start. You are deploying to staging; aren’t you!?

A couple quick tips on top: You can change the column the list is sorted on by using < and >, and see what command is actually being run by hitting c. Knowing which URL is misbehaving in your app is priceless and being able to sort by CPU and memory usage is nice too.

If there are passenger processes and they look normal across the board, I jump to check passenger-status to see if the global queue has backed up. Most often, this is the case, and a simple apache restart will clear the global queue, and the site will start loading again. If an apache restart clears things up, you’re good to go. If the global queue fills up quickly after restarting apache, check the number of requests each passenger process has served and make sure they are incrementing. If they aren’t incrementing and you see “Sessions: 1”, this means something about the request is either long running or blocking. Debugging passenger processes is a post in itself; so, more on that at a later date.

Sometimes, there may be an abundance of connections to the server and apache gets overloaded. You may just be getting a legitimate increase in traffic; so, I would start by increasing the number of MaxClients in your apache config and see if that gets your app back up and running. If you think the traffic might be malicious in nature, you’ll want to track down the connections and count them to see if there’s an extremely high number of connections from a single or a few IPs. If so, you’ll want to see if it’s a valid user that’s abusing your app or someone up to no good. Using this “netstat -an | grep -e ‘:80 ’ -e ’:443’ | awk ‘{ print $5 }’ | awk -F: ‘{ print $1 }’ | sort | uniq -c | sort -n | tail -n 20” will give a sorted list of the top 20 connections to the server via http/https. You can then investigate the IPs and make a judgement call on blocking connections. Using “sudo iptables -I INPUT -s 178.172.178.172 -j DROP” will drop all traffic coming from that address.

Memory Alerts

Often times, passenger processes have gone off the reservation, have become leaky, and need to be killed. We have a moonshine plugin that monitors passenger processes for memory utilization and ensures they are still being maintained by the passenger application spawner. If a passenger process is no longer being managed by the spawner, it can’t be killed when traffic decreases or, depending on your passenger setting, it maxes out on requests served or times out.

If your application hasn’t been broken out into separate roles per server, you may have overloaded what your current server is capable of. Often times, a database gets big or resque processes start to grow as they fork. Restarting these services can release memory that’s not had a chance to be released and will buy some time to find a longer term solution, such as debugging memory leaks, planned downtime for a server upgrade, or separating your database out onto its own server.

Another common memory hog is a late night cronjob that kicks off a rake task. Just because it’s 2am doesn’t mean you can load the whole database into memory to update a few million rows. Don’t let nasty rake tasks keep you or your ops team up at night. Find and fix the memory hungry code or break the single task into smaller parts.

One thing to be careful of is the very real chance the OOM killer starts killing off important processes. While working to reduce memory usage, sometimes you need to ensure data integrity. I like to set the OOM killer to ignore things, like mysql or redis-server, if the server looks close to OOMing and I need more time. You may also want to protect your SSH session; otherwise, you’ll be disconnected and can’t fix the problem without rebooting the server. You can do this using “echo -17 > /proc/$PID/oom_adj” where $PID is the PID of the process you are wanting to protect. Once you have your processes protected, kill off anything else that isn’t essential that might be using memory to buy enough time for a clean shutdown or restart.

Server Down

Yep, not much you can do to rescue this type of alert unless it’s a false positive. If the server is down, restart it and begin digging into the logs. The most common cause of a server down alert is such a rapid growth of memory that the server OOMs before a memory alert can come in. These alerts are hard to pinpoint with a cause, but good monitoring can be a life saver. We use Scout to monitor our servers, so we can check our Scout graphs for a rapid increase in memory usage or possibly disk usage that took the server down. Some other interesting pieces of information about what might have lead up to the server going down can be gathered from traffic graphs and passenger graphs to see if there was a sudden increase in traffic that caused too many passenger processes to spin up. Checking through the server logs and apache logs to see if there’s a common thread to follow in order to find the cause can sometimes be tedious; because when a server OOMs and goes down, a lot of valuable information might be lost because the logs can’t be written to.

If your server is up, but your app isn’t responding you might be getting DoS’d or flooded. If you can connect to your server, possibly via KVM or IPMI, you can still troubleshoot the cause and block traffic that’s preventing your ping check from being green. You’ll want to use the netstat command I mentioned earlier, but taylor it to fit the need of the moment.

SSH Alerts

Normally, SSH alerts take some time to investigate because you have to wait for the server to become available again to checkout what’s happening. You might get lucky and make it in the first or second attempt. Jump to the log directory and check the auth log or security log for failed SSH attempts. If you have a script kiddie that happens to be doing a dictionary brute force attack on your server, you’ll see a plethora of failed attempts for either the same user with bad passwords, or an alphabetical list of random usernames. We use SSH keys and no passwords to increase the security of our servers; so, a valid user trying to login and failing that many times is very unlikely. You’ll see the IP address of the machine trying to connect, so you’ll be able to block it in the firewall. Using iptables on the server is an easy way to quickly block the offending IP address from making future attempts until iptables is restarted or a reboot happens. Use the iptables command mentioned earlier to block this traffic.

A long term fix would be installing and configuring something like fail2ban and letting it automatically block IPs based on the rules you setup. This can be a premature optimization, so I would wait until SSH alerts become a problem before spending time setting it up. More often than not, the quick block using iptables is sufficient. If you do go this route, be sure to configure your hosts.allow correctly by white listing your IP so you don’t lock yourself out late at night while making a typo run on your password.

To wrap things up with a word of advice, all these solutions to common problems we see are based on having monitoring setup and alerting when a set of conditions have been met. Sometimes, you’re able to set the thresholds for alerts low enough so you have time to react and fix things. Sometimes, it’s just not possible to be proactive. The best way ahead is with a plan and remaining calm. If you don’t do the basic investigative steps to correctly identify the problem, you may make mistakes and “fix” things that are not the real problem and possibly cause more harm. While things can be stressful, knowing the basic steps of investigation and paths to fix common problems goes a long way to keeping your app up and running and your customers happy.

This is also probably a good time to mention that we’re doing open office hours on July 12th from 2-3PM Eastern if you’ve got questions about anything in this post or pretty much anything else.

Open Office Hours

Fri, 29 Jun 2012 00:00:00 +0000

Come hang with us! We are excited to announce our new Open Office Hours in July where you can ask anything you want about our open source products, like Moonshine for example, advice on how to be ready when you end up on the homepage of Reddit, or why we do the things we do.

Open Office Hours will be held Thursday, July 12th at 2pm through a Google Hangout, spots are limited to 9 people, but you will still be able to view the hangout. You can also participate via Twitter using the #oohrm hashtag to send us questions and comments.

We hope that you’ll join us. We look forward to meeting everyone and making the web a better place.

Open Office Hours
Thursday, July 12th
2pm–3pm

You’ll need a Google Plus account to join in the fun, add us to your circles via the Google Badge below.

Upcoming July Events

Thu, 28 Jun 2012 00:00:00 +0000

One of the many things we look forward to at the Rails Machine office are events in Savannah that bring some awareness to the local tech scene and it just happens that two such events are happening this month, Tech Crunch Mini Meetup and the Downtown Tech Crawl.

TechCrunch Mini-Meetup

July 6th–11th, Tech Crunch, is doing a tour of tech companies in the Southeast, blogging about it along the way, and Savannah is their first stop of the tour this year. From there they’ll hit Atlanta and then right up to the coast until they reach Charlotte, NC. The Creative Coast is lending their great space, we’re going to be supplying the drinks, and the food will be supplied by the farm to fork liaison, Revival Foods, so we hope for a spirited evening of spreading the good word about what Savannah has to offer the tech world.

“The goal here is networking and connecting with the exciting projects happening in your cities. These mini-meetups are a great way to get noticed and to chat about what you’re working on and, in addition, get some advice on next steps.”

Tech Crunch Mini Meetup
Friday, July 6th
at The Creative Coast Office
15 W. York St.
6pm – 10pm

Downtown Tech Crawl

July 17th, Made in Savannah, is putting together the first Downtown Tech Crawl to give curious Savannah residents a chance to peer into the local tech scene, via trolley rides no less. We’re the last stop of the tour before the closing party starts at Thinc and we’re excited to be a part of the event. The variety of companies participating in the inaugural crawl presents a great way for someone wanting to get a bird’s eye view of what Savannah’s creative companies are all about. If you’re thinking about planning a trip to this haunted city, this would be a great way to start your trip. You can reserve a free spot on the trolley via EventBright. Oh, I didn’t mention that the whole crawl if free of charge? Free food, drinks, and transportation, how cool is that!? We hope to see you there.

“Hop on an Old Savannah Tours and get an inside look into the growing technology community in downtown Savannah. Established Start-up Savannah Technology companies and start-up ventures will open their workspaces and give a quick show and tell of their product and team. Curious Savannahians will be able to check out their digs, hear about their products and meet the people leading the technology movement in Savannah.”

Downtown Tech Crawl
July 17th
Start at 35 Barnard St.
4pm – 7:30pm