Queue Inversion Week

February 12, 2009

Hal Pomeranz, Deer Run Associates

Reliving the last story from my days at the mid-90’s Internet skunkworks, reminded me of another bit of tactical IT advice I learned on that job, and which has become a proven strategy that I’ve used on other engagements.  I call it “Queue Inversion Week”.

One aspect of our operations religion at the skunkworks was, “All work must be ticketed” (there’s another blog post behind that mantra, which I’ll get to at some point).  We lived and died by our trouble-ticketing system, and ticket priority values generally drove the order of our work-flow in the group.

The problem that often occurs to organizations in this situation, however, is what I refer to as the “tyranny of the queue”.  Everybody on the team is legitimately working on the highest-priority items.  However, due to limited resources in the Operations group, there are lower priority items that tend to collect at the bottom of the queue and never rise to the level of severity that would get them attention.  The users who have submitted these low-priority tickets tend to be very understanding (at least they were at the skunkworks) and would wait for weeks or months for somebody in my group to get around to resolving their minor issues.  I suspect that during those weeks/months the organization was actually losing a noticable amount of worker productivity due to these “minor” issues, but we never quantified how much.

What did finally penetrate was a growing rumble unhappiness from our internal customers.  “We realize you guys are working on bigger issues,” they’d tell me in staff meetings, “but after a few months even a minor issue becomes really irritating to the person affected.”  The logic was undeniable.

I took the feedback back to my team and we started kicking around ideas.  One solution that had a lot of support was to simply include time as a factor in the priority of the item: after the ticket had sat in the queue for some period of time, the ticket would automatically be bumped up one priority level.  The problem is that when we started modeling the idea, we realized it wouldn’t work.  All of the “noise” from the bottom of the queue would eventually get promoted to the point where it would be interfering with critical work.

Then my guy Josh Smift, who basically “owned” the trouble ticketing system as far as customization and updates was concerned, had the critical insight: let’s just “invert” the queue for a week.  In other words, the entire Operations crew would simply work items from the bottom of the queue for a week rather than the top.  It was simple and it was brilliant.

So we looked at the project schedule and identified what looked like a “slack” week and declared it to be “Queue Inversion Week.”  We notified our user community and encouraged them to submit tickets for any minor annoyances that they’d been reluctant to bring up for whatever reason.

To say that “Queue Inversion Week” was a raging success was to put it mildly indeed.  Frankly, all I wanted out of the week was to clear our ticket backlog and get our customers off our backs, but the whole experience was a revelation.  First, the morale of my Operations team went through the roof.  Analyzing the reasons why, I came to several conclusions:

  • It got my folks out among the user community and back in touch with the rest of the company, rather than being locked up in the data center all day long.  The people who my folks helped were grateful and expressed that to my team, which makes a nice change from the usual, mostly negative feedback IT Ops people tend to get.
  • The tickets from the bottom of the queue generally required only the simplest tactical resolutions.  Each member of my team could resolve dozens of these items during the week (in fact, a friendly competition arose to see who could close the most tickets), and feel terrific afterwards because there was so much concrete good stuff they could see that they’d done.
  • Regardless of what outsiders think, I believe most people in IT Operations really want to help the people who are their customers.  It’s depressing to know that there are items languishing at the bottom of the queue that will never get worked on.  This week gave my team an excuse to work on these issues.

I think I can reasonably claim that Queue Inversion Week also had a noticable impact on the morale of the skunkworks as a whole.  After all, many of the annoying problems that our users had been just doing work-arounds for were now removed as obstacles.  Like a good spring cleaning, everybody could breathe a little easier and enjoy the extra sunshine that appeared through the newly cleaned windows.

We repeated Queue Inversion Week periodically during my tenure at the skunkworks, and every time it was a positive experience that everybody looked forward to and got much benefit from.  You can’t necessarily have it happen on a rigid schedule, because other operational priorities interfere, but any time it looks like you have a little “slack” in the project schedule coming up and the bottom of your queue is full of little annoying tasks, consider declaring your own “Queue Inversion Week” and see if it doesn’t do you and your organization a world of good.


6 Responses to “Queue Inversion Week”

  1. Kevin Behr said


    Brilliant! I am a big proponent of starving unplanned work in favor of planned work. The tired adage “not being able to see the difference between urgent and important” is super-relevant here. When the urgent starves the important everything must become urgent to become important.

    What a great post!!!

  2. tixrus said

    Yeah cleaning my desk is always a low priority item for me because there are always critical projects ahead of it, and really I CAN find anything even pretty quickly and usually don’t lose anything but reduction of visual cruft makes me smile.

  3. srchalup said

    Highly awesome, Hal! I dropped a shout out to Tom, would love to see you crosspost this to EverythingSysAdmin.

  4. qhartman said

    I can attest to the effectiveness of this. Being a one man IT shop, I’ve been doing something similar for some time, though I never gave it a catchy name. Whenever I get burned out on whatever “critical” items I’m working on, I’ll take a day, or even just a few hours, and chew through as many low-priority tasks I can. The feeling of accomplishment I get is often the fuel I need to push through whatever “important” task I’ve been banging my head on.

  5. Joe said

    Fantastic posts… Being a “one man IT shop” like the previous poster, I always appreciate (and am in search of) tips for how to be a better sysadmin. I’ve never had the opportunity (nor desire, but for other reasons) to be in a large IT org. I think I miss out on a lot of learning opportunities by not being a part of a large IT org. I appreciate your posts, and hope to see more in the future.

  6. […] from a comment by “Joe” to “Queue Inversion Week“ […]

Comments are closed.

%d bloggers like this: