Hack and /

This is the third in a series of articles on system administrator fundamentals where I focus on some lessons I've learned through the years that might be obvious to longtime sysadmins, but news to someone just coming into this position.

In the first article, I discussed how to approach alerting and on-call rotations as a sysadmin. The second article covered how to automate yourself out of a job. In this article, I explore something that on the surface may seem boring or mundane but is absolutely critical to get right if you want to be an effective sysadmin: ticketing.

By ticketing, I'm referring to systems that allow sysadmins to keep track of tasks both internally and those requested by their coworkers or customers. There are many ways to get ticketing wrong so that it becomes a drain on an organization, so many sysadmins avoid or it use it begrudgingly. Also, ticketing approaches that work well for developers may be horrible for sysadmins, and vice versa. If you don't currently use a ticketing system, I hope by the end of this article, I've changed your mind. If you do use tickets, but you wish you didn't, I hope I can share how to structure a ticketing system that makes everything easier, not more difficult.

Like documentation, tickets are one of those important things in a mature organization that some administrators think are unnecessary or even a waste of time. A ticketing system is important no matter the size of your organization. In a large organization, you have a large volume of tasks you need to keep track of distributed among a group of people. In a small organization, you often have one person taking on many roles. This leads me to the first reason why tickets are important.

Tickets Ensure That Tasks Aren't Forgotten

Sysadmins are asked to do new tasks constantly. These days, there are any number of ways a coworker might ask for your help, from an email, to a phone call, to a message in a chat program, to a tap on the shoulder. If you weren't doing anything else, you immediately could start working on that task, and everything would be fine. Of course, usually sysadmins have to balance needs from many different people at the same time. Even requests that come in through email have a tendency to fall through the cracks and be forgotten. By storing every request in a ticket, no matter how you got the request, it is captured, so that even if you do forget about it, you'll remember it the next time you look at your ticketing system.

Tickets Make Sure the Task Is Done Right

Even if you can remember what someone wants you to do, you may not remember on Monday all of the details that someone told you in person on Friday. A ticket lets you capture exactly what people want done in their own words and provides a way for them to confirm that you completed the task the way they wanted before you close the ticket.

Tickets Help You Prioritize Tasks

Every request is important to the person who makes it. Every request may not be as urgent to you or your team compared to your other tasks, however. When all of your tasks are captured in tickets, the team lead or manager can go through and re-prioritize tasks so they are worked on in the right order. This ends up being more fair for everyone; otherwise, new tasks have a way of cutting in line, especially when the person asking for something is standing over your shoulder.

With a ticketing system, team leads or managers have a full list of important tasks they can point to when they need to explain why they aren't dropping everything for a new request. At the very least, it will help direct the conversation about why a particular task should be put at the head of the line.

Tickets Distribute the Work

If you have only one sysadmin, distributing tickets and projects is easy. Once your team grows though, it's important to distribute the work so no member of the team gets burned out. Coworkers have a tendency of finding that senior member of your team who is most productive and going to them directly when they have any issue. Of course, that team member is probably already working on plenty of other tasks or may be trying to focus on an important project.

When a task is captured in a ticket, the team lead or manager can assign and reassign tickets to different members of the team to make sure no one gets burned out, and also to ensure that everyone learns how to do things. Otherwise, you end up cultivating specialists within the team that always take tickets related to certain systems, which leads to problems later when that team member goes on vacation.

Tickets Provide an Audit Trail for Changes

Every time you change a system, you create an opportunity for something to break. If you are lucky, things break immediately after you make the change. More often, you'll find that it takes some time for a change to cause a problem. You'll discover two weeks later that something stopped working, and with a ticketing system, you can pull up all of the tasks that were worked on around that time. This makes it much easier to pinpoint potential causes of a problem.

Tickets also provide an audit trail for tasks that require approval or proof of completion, like creating or revoking accounts, granting new privileges or patching software. When someone asks who said it was okay for Bob to get access to production and when it happened, you can answer the question. If you need to prove that you applied a security patch, you can point to command output that you capture and then store in the corresponding ticket.

Many different ticketing systems exist, and sometimes when you hear people complain about tickets, what they are really complaining about is a bad ticketing system. When choosing between ticketing systems, you should look for a few things.

Some systems that developers use to track code through the development process result in very complicated workflows. For a sysadmin though, the simpler the ticketing system the better. Because you already are asking a sysadmin to take time out of solving a problem to document it in a ticket, it helps if the ticketing process is fast and simple. I prefer very simple ticket workflows for sysadmins where there may be only a few states: open, assigned, in progress, resolved and closed. (I'll talk more about how I treat each of those states in the next section.)

The fewer required fields in a ticket, the better. If you want to add extra fields for tags or other information, that's fine, just don't make those fields mandatory. The goal here is to allow sysadmins to create tickets based on someone walking up and tapping them on the shoulder in less than a minute.

Ideally, the ticketing system would allow you some other way to generate tickets from a script, either from sending an email to a special address or via an exposed API. If it has an API that lets you change ticket state or add comments, all the better, as you potentially can integrate those into your other automation scripts. For instance, I've created a production deployment script that integrates with my ticketing system, so that it reads the manifest of packages it should install from the ticket itself and then outputs all of the results from the deployment as comments in the ticket. It's a great way to enforce a best practice of documenting each of your software releases, but it does it in a way that makes it the path of least resistance.

Favor ticketing systems that allow you to create dependencies or other links between tickets. It's useful to know that task A depends on task B, and so you must complete task B first. These kinds of ticketing systems also make it easier to build a master ticket to track a project and then break that large project down into individual tickets that describe manageable tasks. These kinds of systems often show all of the subordinate tickets in the master ticket, so a quick glance at the master ticket can give you a clue about where you are in a project.

Each ticketing system has its own notion of ticket states, but in my opinion, you should, at a minimum, have the following:

A well run ticketing system should provide the team with the answers to a few important questions. The first question is “What should I work on now?” To answer that question, each member of the team should be able to claim tickets, and team leads or managers should be able to assign tickets to individual members of the team. It's important for people to claim tickets and start work only after they are claimed; otherwise, it's easy (and common) for two members of the team to start working on the same task without realizing it. Then everyone on the team can start working on tickets in their personal queue, starting with the highest-priority tasks.

The next question a good ticketing system should answer is “What should I work on next?” Once sysadmins' personal queues are empty, they should be able to go to the collective queue and see a list of tasks ordered by priority. It should be clear what tasks they should put on their queue, and if there's any question about it, they can go to the team lead or manager for some clarity. Again, ticket priority helps inform everyone on the team about what's next—higher-priority tasks trump lower-priority ones, not necessarily because they are less important (a ticket is always important to the person who filed it), but because they are less urgent.

I approach ticket priority as a way for users to help inform the team about how important the ticket is to them, but not how urgent it is for the team. The fact is, there's no way every employee in the company can know all of the other important tasks the sysadmin has to perform for other people nor can they be expected to weigh the importance of their need against everyone else's needs.

A good manager should reserve the right to weigh the priority assigned to a ticket against the other tickets in the queue and change the priority up or down based on its urgency relative to the other tasks. It also may be the case where a task that was low urgency two weeks ago has become urgent now because of how long it was in the queue, so a good manager would be aware of this and bump the priority. If you are going to start the practice of changing ticket priorities though, be sure to inform everyone of your intentions and how you will determine the urgency of a ticket.

Another key to managing tickets is to make sure all of your requests are captured in the ticketing system. Sometimes a coworker can be guilty of trying to skip ahead in line by messaging you with a request or walking directly to your desk to ask you to do something. Even in those cases where you really are going to drop everything to work on their request, you should insist on capturing the request in a ticket so you can track the work. This isn't just so you can prioritize it based on other tasks or so you don't forget it, it's so in a week when some problem crops up based on this urgent change, you'll see this ticket along with other tasks completed that day and it will help you track down the cause.

Finally, as a manager, be careful to distribute work fairly among your team. Even if one member of the team happens to be an expert on a particular service, don't assign that person every task related to that service; it's important for everyone on the team to cross-train. Pay attention if employees try to get tickets assigned to their favorite member of the team, and don't be afraid to reassign tasks to spread the work around evenly. Finally, every ticket queue has routine, mundane grunt work that must be done. Be sure to distribute those tasks throughout the team so no one gets burnt out.