Github Universe wrapped up a couple of days ago. It was absolutely wonderful. I’ve learned tons about building, collaborating on and deploying software. I’ve got a chance to hear from some of the best in our industry (and from other industries, too) in the form of incredibly compelling stories. There’s a few major themes that I’ve taken away. I’d like to write about all of them at some point and continue to explore them further, but for now I want to talk about how software development is a team sport, and more than I ever thought it so before.
Growing from 1 to 10
The use case I want to present for this article is the process of growing an engineering team from 1 to 10. Not only is this the phase I am currently in right now with Handshake, but also I think (admittedly with limited knowledge about post-10 employees) where the importance of operating an engineering team as a true, unified team rather than a collection of sole programmers really comes into play.
At 1 or 2 engineers, life is much different. The system is new and fresh — legacy code doesn’t exist yet, both of you know the codebase in and out, and knowledge is kept effectively in your head. Arguably, you don’t need to be thinking about how to operate as a team at this point. Working fast, lightly and iterating to a point of MVP is the most important goal at this point in a company. Yes, you don’t want to write crap code, and yes you want to at least document some of the important things, but those are less important than proving your business has value and on-boarding customers.
Software Change at 10 Engineers
At 10 engineers, however, software changes much differently.
First, every deploy is touched by 10 engineers. If you actually think about that, and what that means, it’s significant. We deploy every week and every week we ship new code (across primarily one application in our case) built by ten separate engineers. Some of those engineers are new — they’ve been on for a month or less — and are unfamiliar with the majority of the codebase written by the early engineers. Some have been around for years and are working with new code written by the new guys.
But regardless of how new or old, each and every change put in place by each of those engineers must work correctly. In addition, every engineer should be able to understand any changes put into place. And, lastly, new engineers should be able to understand and get up to speed on existing knowledge.
The Problems
How do you orchestrate a team of ten engineers who are pushing code all day every day to build a correct piece of software?
Bugs
First — you have potential for bugs. With large teams, it’s very likely you’ll have two engineers working on the same thing at the same time. And I’m not talking about just same line changes which can be resolved in git. I’m also talking about logic changes. One person changes a function that another relies on. A new parameter is added that throws another call off by one argument. A page is completely revamped by engineering A while engineer B fixes a bug on the old page. There is an incredible amount that can go wrong. How does engineer A know that engineer B is changing something you rely on? Who fixes it? How do you find the problem in the first place?
Performant
With a large number of concurrent changes, you’ll eventually make performance regressions. For example, an engineer adds a new attribute to a JSON serializer for work they are doing on a form. However, that adds it to the same serializer for engineer B who is working on making search faster, and now the search results contain way more data than needed, thus slowing it down. How do you avoid this? What do you test for to make sure some theoretical piece of data isn’t added in the future? Do you actually want to test for that?
Usable
Having a designer on your team is incredibly valuable. But, unless you are able to design and mock up every UI element you add, engineers will often have to design things themselves. Good design and UX often relies on high level knowledge about the industry, market, and most importantly the users. Does the engineering team spend time conducting user studies with the designers? How do they gain knowledge of their users otherwise? When that knowledge changes, how are they updated?
Knowledge and Change
In addition to correctness, how is each team member kept up to date? When your search backend is rewritten, how do they know? And how do they learn how to use it? Who teaches them? Whether they are new engineers being on-boarded or existing engineers, knowledge around how the application works needs to be shared.
Moving Towards Team Oriented Development
How do we move towards a more team oriented development process?
Inspire sharing change
Sports teams are always learning, practicing, and bettering themselves. When they learn tricks or better themselves, they share with the team. When the team has a new play, every team member learns it.
I don’t see why software should be different. For a given codebase, the engineers working on that code base should be, somehow, in communications and aware of what is changing and how that affects them.
One common way to do this is code reviews. There are a lot of other advantages, but one of the big ones is shared knowledge about what is changing in the codebase. It also lets engineers see and read other code. You can learn new tricks, and see how each engineer has their own style or approach to problems.
Another way for sharing change is post-sprint presentations where engineers share what they worked on. Not only do they get to be proud of their achievements but each engineer (and other team members) can get brought up to speed on what they’ve built, how it works, challenges they faced along the way and how they solved it, and what impact it has.
Write Down Knowledge
For an engineering team to scale, knowledge simply cannot be kept in each engineer’s head. Not writing down knowledge promotes information hoarding, limits abilities of new team members, causes chokepoints in processes such as deploying new code or touching a certain piece of the codebase, and makes on-boarding a nightmare.
In addition, writing down your process today let’s you give a measurement to compare against when you iterate and improve the process over time. It let’s you build up rather than spin your wheels in place, by giving a foundation for the whole team to work off of, and avoid repeated work by separate team members.
Don’t be afraid to write about the small things. For example — we like to write about the most common gotcha’s in Rspec testing so others can learn from our pain rather than repeat it.
Automated Testing and Analysis
Automated testing has (fortunately) become more and more commonplace, and it’s hard to believe that any serious software company isn’t doing it these days. Not only is it an incredibly effective way for finding and avoiding regressions, but also can act as a sort of specification for your software.
Analysis tools, like rubocop or rubycritic can be an effective way to assure the team operates in a conventional and mutually understandable way. Rather than each team member having their own style, there is one, well defined style that can be agreed upon.
Find a Buddy
Aside from code reviews, there are other ways to work with teammates directly. Pair programming of course, but also an idea we started on recently at Handshake called co-developing.
For the bigger engineering items we face, we let developers choose co-developers. Co-developers don’t write any code, but are rather there to help bring their knowledge to the table and expertise and act as rubber duck for the developer. The developer explains how they plan to approach the problem, what things they are worried about, what their deploy strategy is (if it’s broken up into multiple), and ask any other questions the may have.
This has a lot of the advantages of code review (and often the co-developer will also be the code-reviewer), but we think can be a more proactive approach. They get the opportunity to talk through a strategy before they spend the time to implement.
Be Asynchronous
I’ve presented a few ways you can be team-oriented in software development, but one key is that this is asynchronous as possible. I don’t mean that, for example, while co-developing all communications must be asynchronous. I mean primarily that:
- Knowledge sharing, processes, and actions taken by team members should be visible and transparent to the rest.
- Flow should not be disturbed.
Number 2 is simple. Some might think it’s silly to include, but I strongly believe that the primary of engineering work is completed while in flow.
Number one is focused around making progress as a team and not in isolation. Sharing knowledge is great, but if it’s not shared and available to everyone (now or hired in the future), it has significantly less value. Implementing a process for deployments is great, but if others cannot read about it, they will not learn how to do it themselves. Make sure process improvements and changed are written down for everyone.
This isn’t solved
Although I’ve presented some of my thoughts on how to approach this problem, I think it’s far from solved. It’s a challenge we face and work on every day, with lots of trade off decisions. I’d love to hear about your thoughts on how to build a software development process that is team oriented.