A lot of engineers looking to join ZenPayroll are interested in hearing more about our development workflow. Put more simply, how do ZenPayrollers write software? The answer to that is constantly changing, as we’re always refining our workflows to fit our ever changing needs, but I’ll cover in some detail the way we do things today.
Before going into the nitty-gritty, it’s important to understand our high-level engineering principles:
Push high-quality software. This is payroll after all. The code we write is ultimately used to move hundreds of millions of dollars and pay thousands of people. There’s no room for error.
Don’t optimize for the short-term. This applies to more than just the engineering team. It’s a company-wide value that every person in the company — engineer or otherwise — lives by. We’re building our company to last 50 years or more, and we want our code to last just as long.
Keep it simple and straightforward. Lots of moving parts in a system make it brittle. We try to keep our processes as simple as possible, but no simpler.
With those principles in mind, here’s an overview of how a feature goes from an idea to production here at ZenPayroll.
Everything we build starts with a spec. Sometimes the spec can be as simple as a few lines of text explaining what the feature will do (common for small tweaks). Other times, the spec can be full-blown balsamiq mockup.
The spec for a feature can come from 3 places: The product team, the engineering team, or the support/compliance team. Generally, specs for large user-facing features come from our product team, as they have a great sense of the important new features that our customers need. Specs for back-end systems generally come from the engineers themselves — things like fraud-monitoring systems, payroll calculation systems, and systems to automate things like faxing forms to the IRS. Finally, our support team oftentimes come up with specs on tools they need to serve more customers and serve them better.
Wherever the spec comes from, a developer is intimately involved in its development by providing feedback, suggestions, and a sense of engineering tradeoffs.
2) Branch off development
We use git for source code management. Once a spec is finalized and ready to be implemented, we will “git checkout -b new_feature_branch_name” off of our development branch and start laying down some code.
Coding commences! Since code-quality is very important to us, we strive for 100% test coverage in everything we write. This applies to both our back-end and front-end. We use RSpec for testing our backend payroll API (Ruby on Rails) and Mocha for the zenpayroll.com app, which is a large single-page application (Coffeescript on Backbone.js). In addition to unit and functional tests, we also write integration tests using Capybara.
4) Pairing (optional)
We generally don’t pair at ZenPayroll, but there are times when it’s useful to work on a feature together, especially if some part of the feature is particularly complex or requires intimate knowledge of another person’s code. For this reason, we have a couple dedicated pairing stations set up around the office.
5) Code review
Once development on a branch is complete, we then send the branch for code review to another engineer. We use an awesome tool called Phabricator to do this. Usually, the code reviewer is someone who tangentially has some knowledge of the part of the code that the feature is built upon.
Code reviews are immensely beneficial to us. Most importantly, is a great way for an engineer to get suggestions on how to better improve and refactor the branch. It also roots out any potential bugs that may have been overlooked. Another benefit is that code reviews help distribute knowledge of the codebase to other engineers. Finally, a lot of learning of better programming techniques occurs during reviews, improving the skills of everyone on the team. As iron sharpens iron, so we sharpen each other.
Usually there are a few iterations of reviews until the reviewing engineer is satisfied and accepts the branch. It’s also worth mentioning that we code review almost everything, as we’ve found that even seemingly simple one-line changes can have huge unintended consequences.
6) Rebase development
Once a code review is accepted, the feature branch is rebased on the development branch (“git rebase development”). After any conflicts are resolved, the feature branch is merged into development (“git checkout development; git merge feature_branch_name”). The nice thing about using Phabricator is that this rebase/merge combo can be executed with a simple “arc land” command.
7) Leeroy – Run test suite
Whenever a branch is merged into development, we have a really beefy test server that runs our entire test-suite (back-end, front-end, and integration tests) on the development branch. The name of this test server is Leeroy, who does all this with the help of Jenkins. We also have a sweet team dashboard displayed on a TV in the office that lets us know if tests fail. Another useful thing Leeroy does is locally run any new model validations on our (anonymized) production data to make sure the new code plays nicely with real data. If every single test passes, we will proceed to the next step.
8) Deploy to staging
Jenkins automatically deploys our development branch to a staging environment. Our staging environment is meant to almost exactly mirror our production environment. Under the hood, Jenkins is really just issuing a “cap staging deploy” command. When our development branch is successfully deployed, Jenkins lets us know via email.
9) Sanity testing on staging
Our feature branch is close to being on production! Someone on the product team will usually play with the feature on our staging environment trying their best to break things.
10) Merge to master and deploy to production
If things look good on staging, we’ll merge our development branch into our master branch and then deploy to zenpayroll.com using a simple “cap production deploy” command. Hooray! Our feature is now live!
Generally, we’re deploying to production several times a week. We try to keep our features small and will often break large features into smaller chunks to keep them more manageable. There’s nothing worse than trying to code review huge branches.
We’re always looking for ways to improve our development workflow, so if you have any suggestions or tips of what was effective for you, please let us know!
P.S. ZenPayroll is hiring! If you’re a talented engineer looking to work with smart, passionate people on a product that will change the world, please reach out to us!