Advice to Recruiters

Like many tech workers, I regularly get inquiries from recruiters. Lately, these inquiries seem to be coming to me via LinkedIn for the most part… and let’s just say that the quality of most of these leads is extremely dubious. Judging by feedback I’ve received on Twitter, my colleagues in the tech industry are just as frustrated by this as I am.

When I suggested trying to educate recruiters to help them do a better job, my friends pointed out to me that recruiting tends to be a high-turnover business. We could spend a significant amount of time educating one batch of recruiters, only to have to do it all over again later. So I thought I might jot down some notes to recruiters here on my blog, if only so that I have to say these things just once.

It’s not just about matching keywords. I’m known for my Perl programming and I have the keyword “Selenium” on my LinkedIn profile. But even a casual glance at my profile would tell you that I’m not interested or a good fit for a Senior SQA position on your decades-old Perl-based web framework. Similarly, it’s clear from my profile that I’ve been an independent consultant for 15+ years, so I’m unlikely to be interested in full-time employment with your gigantic software company.

Do your homework. Please respect my time, and take a moment to really understand the position you’re trying to fill and the people you’re trying to put there. The best recruiters I’ve worked with understand their own business and the industry they’re working in, and are looking to build relationships for the long haul.

No job description = no response. If you contact me about an “exciting” job opportunity with your firm, but don’t include the job description (or link to one), I’m just going to assume you’re trawling for resumes. I need to evaluate for myself if I think the opportunity is “exciting”. To expect me to respond sight unseen is again disrespectful of my time.

If you just want to leverage my Rolodex, tell me. I get it. I’ve been in the industry a long time, and do work that tends to bring me into contact with lots of different people. And I’m perfectly happy to refer interesting jobs to friends who I think the position is suitable for. But don’t play games with me. Be up-front and say, “This job isn’t right for you but I was hoping you might know somebody who it is appropriate for.” That’s a reasonable and professional request. And I will honestly consider it, and try to fire it off to my “network” of friends, and let you know that I’ve done so.

I’m not going to do your job for you. But if you keep coming back to me over and over again for referrals (especially for positions unrelated to my fields of expertise), or keep bothering me for follow-up after I’ve put your opening out to my network, I’m going to start blocking your messages. If I wanted to be a recruiter, I’d be doing it right now. Again, respect my time. Say, “Thanks for the referral!”, and start following up on those leads yourself.

I believe recruiting is an honorable profession, and a benefit to our industry if done well. Many of my colleagues would love to build a relationship with a recruiter who could help them through all phases of their careers. So please consider the advice above in a constructive frame of mind. I welcome feedback from both recruiters and candidates (and employers!) in the comments.

Getting Started in InfoSec… Or Any Other Career

Lately I’ve received several requests for advice on breaking into the InfoSec field.  I find myself repeating the same advice over and over, so I thought I’d post my thoughts here on Righteous IT to save time (at the risk of turning this into a career advice blog).

What Others are Writing

“Breaking into InfoSec” has been a hot topic in the community lately, and several authors are writing eloquently on this topic.  Rather than repeating their good advice, let me just throw out some important links to read.

Every Tuesday, Lee Kushner and Mike Murray provide solid InfoSec career guidance in   “Career Advice Tuesday” at the Information Security Leaders blog.  One oft-repeated piece of advice in their blog is to develop a “career plan” for where you want to be with at least a five-year time horizon.  While no plan survives contact with the enemy, having a plan means that you’re moving forward in a purposeful direction rather than just wandering at random.

Bruce Schneier recently posted “So You Want to Be a Security Expert” on his blog.  I’m a firm believer in his “Study… Do… Show” mantra.  Bruce gives a specific shout-out to security certifications, which are indeed useful for demonstrating a certain level of knowledge in a general discipline.  But I wish that more people starting their careers put at least as much effort into doing research in their own areas of interest and writing blog posts, talks, and code to document what they’ve done.  This is how we grow as an industry and incidentally it also shows potential employers something that distinguishes you from all the other “highly certified” professionals you’ll be competing against for jobs.

That Bruce Schneier article is part of a larger series of interviews with various InfoSec professionals on how to break into the InfoSec field, which is being created by Brian Krebs over at Krebs on Security.  Brian’s blog is normally some great coverage of recent happenings in the Cyber Crime world, but these (often first-person) accounts of how to get started in InfoSec have been really interesting.

Similarly, Eric J. Huber has been running a series of enlightening interviews with leading lights in the field of Digital Forensic Investigation on his Fistful of Dongles blog.  Somehow he became momentarily confused and also included me in this series.  But apart from that oversight, these interviews always include interesting information on how to get started in the field.

If you’re paying attention, one thing about all of this advice is that it’s equally applicable to getting into any field.  There are no magic tricks for getting started on an InfoSec career path that are different from any other career path.  The corollary to that realization is that any of the classic career guidance books (from “What Color is Your Parachute” to now) can be helpful when you’re getting started in InfoSec or any other career.

It’s All About Your Network

When people ask me for career guidance, the one point that I emphasize repeatedly is that personal connections– your “network” of friends and colleagues– control your career destiny more than any other single factor.  Every good job I’ve ever had, whether as a full-time employee or as a consultant, has come through personal connections.

When you’re just starting out your career, you’re also starting to create your professional network.  This process begins during your educational history.  The contacts you cultivate during college and grad school– both fellow students as well as faculty and administration– are at least as important as what you learn from your books and professors.

Many of you reading this may not have been fortunate enough to attend college, or your college days are long past.  And even the people who did start to build their network in school need to continue building their networks after they leave their educational womb.  You need to constantly be on the lookout for opportunities and venues to meet other people and create a robust, living network.

An important part of your personal network comes from your on-the-job friends and co-workers.  If your employer sends you for training, part of your job at that training event is to make useful contacts with other people in the room.  If they’re at the same training event with you, they’re almost certainly part of the same field and will be great people to interact with in the future– whether that’s getting help with a problem you’re stuck on or finding a new job.

But also look around your area for regular meetings of different groups  and invest the time to attend the meetings.  This could be anything from a Security BSides event, to a SAGE or LOPSA local group, or an ISACA or ISSA chapter meeting, or even Toastmasters.  InfraGuard may have an active chapter in your area.  SANS often has a “community night” associated with its conferences which you can attend for free and network with other people in your area.

Don’t have a local group in your area?  Go start one!  Try using LinkedIn to search for other IT and InfoSec professionals in your area and reach out to them.  It doesn’t have to be anything formal.  Just meet for dinner/drinks every month and talk about your experiences and research projects.

Social networking has become an extraordinary resource for reaching out and networking with other InfoSec professionals.  While it will never fully replace face-to-face interactions, “knowing” somebody by interacting with them first via Twitter, LinkedIn, or Facebook can get you past the awkward chit-chat phase when you finally do meet them in real life.  And it can help you engineer those meetings when you’re in the same geographic region.

When you come into an established group for the first time, I urge you to sit back and just listen for the first couple of meetings.  Figure out who the “players” in the group are and get a feel for the “social norms” and nuances in the new group.  You’ve probably had the experience of boorish newcomers coming in and making a pain of themselves in groups that you’re already a member of.  Don’t be “that guy”.

Instead you want the group to recognize your positive contributions.  That could be anything from providing helpful summaries of information provided at the meeting, to helping with setup and tear-down at meetings, to providing food and beverage, to providing additional links that are relevant to the meeting’s focus, to contributing your own research and presentations.  Even just making new people (like you) feel welcome and accepted is a valuable contribution!

Small Fish, Big Pond

If there aren’t currently any gatherings for professional InfoSec people in your area, and you’re having trouble tracking people down on LinkedIn to start your own gathering, then this may be a sign that you’re in the wrong geographic location.  Being the biggest fish in your small pond may be comfortable, but you need to put yourself in an uncomfortable situation in order to grow.

You need to be in a situation where you’re constantly being exposed to new information and new ways of doing things.  You might think you’re getting this from reading articles and blogs on the Internet.  But you really need people around you who will push you to improve your game.  If you’re on your own reading about new technology on-line it’s easy to think, “That’s cool, I should look into that.” But meeting up with your InfoSec pals every month will do more to push you into actually doing that research than anything else.

When you’re learning on your own it’s easy to have “blind spots” and miss out on important information.  While social media can help with this somewhat, it’s not a replacement for being in a room with a group of like-minded folks who are bouncing ideas and solutions off one another at a rapid rate.

Being in the right geographic location also provides more job opportunities, which also translates to more “interesting” job opportunities.  Feel like you’ve topped out at your current job and aren’t being challenged?  Things are much easier if your next job doesn’t require you to move your home.

But how do you get moved to the “big pond”?  In my case, I took a pretty lousy job for a year because the job was willing to relocate me to the Silicon Valley.  Remember that advice about having a “career plan”?  It’s a lot easier to take a lousy job for a year if you view it as a step on the road to the career you want.  During that year, I was busily getting plugged into various tech groups in the local scene, and by the end of the year it was almost embarrassingly easy for me to step into my next job, which was a lot of fun. The things I learned during my 12 years in the Silicon Valley were instrumental in shaping my career and massively increasing my knowledge-base.  And the friends and contacts I made during that period are still with me today.

So pay your dues if you have to, but get yourself to one of the big high-tech centers: Silicon Valley, New York, Washington D.C., or Seattle.  You may never be a “big fish” in any of these places, but you’ll be better for having had the experience.

Consulting (Part 6) — Work Finds Experts

In my last post in this series of articles on consulting, I suggested that you’re better off if you don’t have to spend a lot of unbillable overhead time searching for work.  The article covers several ways to position yourself so that you’re more likely to run into unsolicited contract offers– “make the work come to you” rather than having to go find your next assignment yourself.

Another mechanism for putting yourself into the path of potential job offers is to build a reputation as an expert in your chosen field.  This kind of reputation is more than just a way of getting your name in front of potential clients.  It also leads to more interesting and higher-paying jobs as a general rule.  Of course, it can also sometimes discourage people from calling– “Oh, they’d be too expensive to hire!”  But these are likely not people who you would want as customers.

Being recognized as an expert can “just happen” to some people who are at the right place at the right time, but it is also a reputation that can be developed through continuous improvement.  I actually started this process during my time as a full-time employee, 5-6 years before I went out on my own as a consultant.  You can start right now.

Leveling Up

Building your reputation as an expert is not unlike “leveling up” in your favorite computer game universe.  You start out as a novice and gradually work your way up through challenges, each of which unlocks the next level of challenges.

Before you start out on your quest, you must first identify the subject matter area to specialize in:

  • It should be something marketable. Becoming the world’s foremost authority in underwater basket-weaving might be extremely interesting, but it’s unlikely to bring in much cash.
  • Pick something that’s hard.  This ties into being marketable.  I’m sure you’ve heard the expression, “If it was easy everyone would be doing it.”  The consultant’s corollary is, “If everybody’s doing it, you can’t bill much money for it.”  Pick a field where there are significant technical “barriers to entry” for people getting into the field.
  • Don’t get too specific. Particularly in the technology field, becoming an expert in a specific technology niche hurts you when that technology ultimately gets abandoned in favor of the next big thing.  Yes, there are still COBOL programmers making lots of money, but fewer and fewer every year.
  • Don’t get too general.  “Jack of all trades, master of none” doesn’t help you sell your services.  Plus there’s too much to learn everything.  Pick a field and a specialization– my consulting business started out doing Unix administration and security, focusing heavily on Solaris, which was the most popular enterprise Unix platform at the time.
  • Pick something you enjoy doing.  Yes, you need money to live.  But you should also have fun with what you do for a living.

The next step is to build up knowledge in your chosen field.  You’re not enough of an expert to go out and consult, so you’ll probably spend some time learning on the job.  Look for jobs that stretch your capabilities and force you to learn new things about the area you’re interested in.  Yes, you’ll also likely be playing around with things in your own home lab environment.  But you need to face “real world” scenarios where you must balance competing forces: from the purely technical, to the political, to the budgetary.  If you only know something from doing it in a lab, then you don’t really know it.  And you don’t yet know what you don’t know.

The next step is what I think of as “getting on the radar”, or creating your initial body of work.  Most likely, this will take the form of blog postings and other self-published work.  Presenting at local user group meetings is another good mechanism for introducing yourself to the community.  Submit papers for conferences, but be prepared for rejection.  Learn from the feedback you get, and keep submitting better proposals until you get accepted.

Over a period of what can take several years, you will gradually work your way up into more prestigious venues that gets your work more visibility.  For example, you might transition from writing blog posts to getting your work published in journals and trade magazines (as the former Technical Editor for Sys Admin Magazine, I can tell you that technical publications are always desperate for good content).  Your local user group talks will move from regional to national venues and possibly even international events.

There are some signs to look for to show that you’re really starting to “break through” into expert status.  Getting invited to present at a conference is one indicator.  Other experts referencing or citing your work is another.  The clearest indicator is when you start to get paid for your writing and speaking.  Congratulations!  You’re an expert, though you may not feel that different from when you were a novice.  The more you know about a field, the more you realize how much you have to learn.

The other important aspect of this “leveling up” process is that it forces you to participate in the community and creates a professional network that you can fall back on when you do go out on your own.  And if you’re shy or introverted, you will be forced to come up with a strategy for coping with that so that you can succeed.  Also, teaching something to other people really makes you learn the subject to an amazing level of depth.

It’s also worth thinking about the possibility of writing a book.  Everybody’s heard of the person who “wrote the book” on a certain subject.  There’s a reason that phrase is in common usage– being the person who literally “wrote the book” is an invaluable calling card and addition to your stature in the field.  That being said, reputation is pretty much the only reason to write a book.  If you look at it as a short-term financial transaction, your “hourly rate”– amount of money you get as the author divided by the amount of time you spent writing– is trivial compared to what you could be earning on your job or as a consultant.  So you may want to write that book before you start consulting.

Ultimately, work will start finding you. Initially, it will probably take the form of full-time job offers.  This is great.  You can pick the ones that seem the most interesting and which have the most to teach you.  If you end up finding your happiness in a succession of full-time jobs and you never end up consulting, that’s great too.  You didn’t waste your time “leveling up” to expert status.  All that effort helped you land the sweet jobs you get offered.

My experience was that after working in the industry for about 10 years with ever-increasing seniority, I had “topped out” in the technical track.  At this point a weird thing happens: most companies try to make you a manager.  Try it, you might like it.  Personally, I hated it because I like being “hands on” with technology.  The only role left for senior people who wish to stay “hands on” is consulting.  After being a manager for 18 months, I worked through my professional network to find a former co-worker who needed a consultant for a six-month engagement.  I signed the contract, and gave two weeks notice at my last employer.  I’ve never looked back.

Keeping It Up

The tricky part about being perceived as an expert is that it’s an ongoing process.  Our field is constantly evolving and you have to keep learning and publishing to stay ahead of the curve.  I spend a great deal of unbillable time on continuing education.  I will earn some money for writing articles about my research, and much of what I learn ultimately gets turned into training that I get paid for as well.  But in the final analysis, this “overhead” or opportunity cost for the time I spend on research is my sales and marketing budget.  And it’s a lot more interesting than printing glossy brochures or throwing expensive parties.

I give a lot of my work away for free.  Like my friend Celeste says, “Contribution is marketing.”  People often ask me if I think it hurts my business to give information away for free.  Not at all.  Consider:

  • If it’s easy enough to do that somebody could just read one of my articles or presentations and do it themselves, then it’s probably so uninteresting that I wouldn’t want to do it myself.  And I wouldn’t be able to charge much money for it if I did.
  • Even if an organization has the technical capability to do something, they may lack the resources to get it done.  In those instances, they’re going to call me, the person who “wrote the book” (or article or blog post as the case may be).

And ultimately, I do it because it’s just the right thing to do.  I get enormous value out of what others are contributing to the community.  I’m just trying to give back some of that value.

Consulting (Part 5) — Finding Work

One thing I haven’t addressed in this series on consulting is how to go about finding work for yourself.  This is a huge topic in and of itself, and I’ll likely spend several posts covering this subject.

At a high level, there are two basic approaches to getting your next assignment: you can go find the job, or it can come find you.  Going and finding the job means watching mailing lists and job boards for possible openings, and running down the leads.  Possibly you’re looking at even “cold calling” organizations in your area to see if they can use your expertise.  This is process is quite a bit of work, which you must remember ends up on the unbillable “overhead” side of the ledger.  It can also be difficult to conduct this kind of search while you’re working another contract.  And as I mentioned in a previous post, it’s desirable to have your next assignment lined up before your current one ends.

One option is to out-source your job search to somebody else– whether that’s a recruiter or a professional sales organization.  This, of course, has a cost associated with it.  I’ve never entered into such a deal myself, so I can’t speak to the exact costs, but you’ll have to decide whether the amount of work you get is worth the cost of acquiring the business through one of these means.  If you’re a solo consultant like me, I imagine a really motivated external sales person could bring in way more work than I could handle, which would make the whole arrangement less valuable on both sides.

So as you can no doubt guess by now, I’m going to advocate for the “let the work come to you” strategy.  First there’s the benefit of less overhead costs in finding your next assignment.  Second, you can generally command a higher billing rate.  Consider that the organization contacting you has identified a problem they’re having and recognized that you may have the expertise to help them solve it.  They wouldn’t be calling you if it weren’t urgent.  And the combination of those factors makes it easier to get the billing rate you want, and with less negotiation.

While it’s all very nice to say, “I want my work to come to me”, you can’t just wish things were that way.  You have to put yourself into a situation where that’s likely to happen.  So think about some of the directions that unsolicited work can come from and then position yourself in the path of those forces so that the work hits you.

Repeat Business

This one might seem obvious, but I often feel that a lot of consultants don’t think enough about this.  The best customer to acquire is one you already have. You already have a trusted working relationship in place, and you’ve probably already dealt with the annoying contract and accounts payable issues that waste time at the beginning of every new engagement.  So from a “cost of acquisition” perspective, getting additional work from a current or former client is a no-brainer.

Also, the more work you do for an organization, the more valuable you become to them.  You have knowledge of their processes, procedures, and systems– perhaps because you’ve implemented many of them!  You know the people at the company and have probably identified the “gate-keepers” who can either facilitate or thwart new projects.  That means you can (and should) demand higher billing rates on subsequent contracts.  And it will be worth it to the client because you’ll spend less time “ramping up” on their environment.  So while your hourly rate will be higher, you’ll still cost the customer less than bringing in a brand new firm to do the same job.

And even if you don’t end up doing multiple contracts for a given firm, there’s still the chance that they may recommend you to their friends in other organizations.  Referral business is great, because a “trusted third-party” is vouching for you with the new firm.  And this is one of many reasons why you need to work hard and focus on doing an outstanding job on each engagement.  Because nothing sells your service in the future better than your past performance with your clients.

Referral Arrangments

While we’re on the subject of referral business, it is possible to formalize such arrangements.  One approach is to create an arrangement to provide specialized services to an organization that can’t or doesn’t wish to maintain an in-house capability.  For example, this would be me making a deal to provide forensic services for a law firm that perhaps doesn’t have enough need to employ somebody full-time.  If I could make arrangements of this type with several smaller firms, then I’d likely have as much work as I could handle.

Another example would be a sub-contracting arrangement, similar to the one I currently have going with Mandiant.  When they get busy, they have a small group of consultants that they can call on to help deal with the overload.  Obviously, if I’m on another assignment when they call then they’ll have to get somebody else to fill in.  And when they’re less busy, I still need to find my own work.  But so far the arrangement has been quite agreeable.

Finally, as a individual, there are often times when job offers come in while I’m busy on another contract.  It’s better to be able to at least give the prospective client a referral to somebody else than it is to just say, “I’m too busy”, and leave them to find somebody for themselves.  People will remember you helped them, even if that help is getting them to the person who did the work for them.

So it’s good to have your own network of trusted friends in the consulting business who you would feel good about referring the business to.  You can try formalizing this arrangement if you want.  At various times I’ve made agreements with other consultants to receive a “finder’s fee” for work we refer to each other.  But because this is such a small industry, keeping track of how a given firm actually acquired a particular customer can be a difficult headache.  And there can be hard feelings if one side of the arrangement thinks they’re not getting their fair fees.  I find it’s better in the long run to just refer business without expecting direct compensation in return.  Karma is a powerful force– believe that you’ll eventually get what you deserve.  Because you will.

Professional Networking

But in order to have a trusted group of people to refer business to, you have to get out and network with your peers in the industry and figure out who’s smart and trustworthy.  So this means a level of interaction greater than just shaking somebody’s hand and exchanging business cards at some social event.  This is one of the reasons why technical gatherings like conferences and local user group meetings are so important.  You have the chance to meet people– sometimes at multiple events– and see how they interact with their peers when discussing technical challenges.  And of course you have the opportunity to model your own behaviors under the  same conditions, which makes it something of a double-edged sword.

To leverage your professional network for business, you need to “stand out” in a positive way and not just be somebody who’s there but fades into the background.  That means providing value to the community you’re interacting with.  Value can come from doing your own research and publishing the findings, giving presentations, answering questions in a helpful, timely manner on community mailing lists and forums, organizing events and gatherings, and even just making people in the community or who are new to the community feel more comfortable and accepted.

How did I end up in this subcontracting arrangement with Mandiant?  Because of my professional network.  Rob Lee and I are both active in the SANS Instructor Community and had talked a lot about issues in Forensics.  And I’d helped him with Linux questions and issues with the SIFT Workstation.  So when he was looking for people to help Mandiant, I was a “trusted entity” he felt good about calling on.  And I got involved with SANS in the first place (almost 20 years ago now) through my professional network as well: one of my former co-workers, Michele Guel.

So your professional network is one of your most important tools.  Try to give more than you take, and you’ll do great.  Besides the unsolicited referrals you may get from other members of your community, people will be more likely to help you when you ask them directly.  The trick is to build up enough good will so that when you do have to make an “ask” request, people will be motivated to help you.

Consulting (Part 1) — The Case for Consulting


January 2012 will mark the 15th anniversary of the founding of the consulting business I run with my wife.  Lately I’ve had a number of people asking me questions about consulting– how to get started, how it works, pitfalls, etc.  I’m more than happy to answer these sorts of questions because I’m still “paying it forward” for all of the great advice I received when I was just starting out.

However, in an effort to reach a larger audience and to not have to repeat myself as much, I’ve decided to devote some blog space to the basic advice that I cover in my usual consulting talks.  This is a huge topic area, and I’m expecting to write several posts to cover just the foundational stuff.  I’ll crank them out as time allows.  If there’s anything you’re particularly curious about, be sure to leave a comment and I’ll try to address questions as the series rolls along.

In this first installment, I wanted to talk about some of the basic pro/con arguments you hear about being a consultant, and give you the view from where I sit.  Let’s call this installment…

The Case for Consulting

Pro: Consultants Make a Lot of Money

This is definitely one of the first items that piques people’s interest in becoming a consultant.  You hear about consultants making hundreds of dollars per hour, divide your annual salary by 2000 hrs/yr, and start thinking the grass is greener on the consulting side of the fence.

Yes, top consultants bill at hundreds of dollars per hour.  But guess what?  We don’t get to bill 2000 hours per year.  There are all sorts of unbillable “overhead” tasks that take away from our billable time:

  • Marketing, finding new clients
  • Invoicing, collections, time and expense reporting
  • Taxes and other official paperwork
  • Arranging insurance and other benefits
  • Continuing education, training

The list goes on, but the point is that when you become a consultant you’re really working two jobs: the work you’re doing for your client that you get paid for, and the work you do to keep your own business running which you do as “overhead”.

Also, there are costs that you pay when you’re on your own that you never see as a full-time employee (FTE, for short).  Normally your employer covers a portion of your healthcare and other benefits and sometimes contributes to a retirement account on your behalf, as well as paying the employer’s share of taxes.  If you talk to your employer, you’ll find that they typically figure these costs as being 50-100% of the employees’ base salary (you’ll hear this referred to as an employee’s “loaded salary”).  So you have to factor in these costs when trying to figure the net take home pay as a consultant.

The compensation discussion is a huge topic in itself, and will be covered in detail in a later post in this series.  Yes, if you have financial discipline and a clear understanding of your costs, you can make a lot of money as a consultant.  But be wary of straight “apples to apples” comparisons between full-time employees and consultants, because things are never that simple.

Con: Consulting is “Risky”

People ask me all the time if I’m worried about where my next job is coming from.  In fifteen years, I’ve lived through two major downturns.  Yes, there have been times when consulting work has been scarce.  This is another reason that consultants bill at such high hourly rates– we’re factoring the inevitable cost of being out of work.  Sometimes this is just a brief period while were transitioning from one contract to the next, and sometimes there’s a protracted drought.

The difference between a successful consultant and somebody who’s going hungry is an understanding that downturns happen and preparing for them.  The best advice I ever got when I was first starting out what to make sure I had six months of expenses (rent/mortgage, utilities, car/insurance payments, food, medical, etc) in the bank before I started my consulting business.  I’m going to come back to this point over and over because it’s important in lots of ways, but at its most basic your “six months in the bank” is shelter against bad times.

What fascinates me, however, is the belief that a lot of people seem to have that as a FTE they somehow have more job security than your average consultant.  In practice, I believe these people couldn’t be more wrong.  At least here in the United States, most people are “at will” employees and they can be let go at any time at the complete discretion of their employers and with little or no notice.  So really we’re all what the HR types like to refer to as “contingent employees”.  Why shouldn’t you be compensated like one?

I know that many people can understand this argument intellectually and still have a hard time with the notion of going out their own.  Sometimes our gut overrules our brain and makes the consulting lifestyle untenable.  But even if you don’t end up as a consultant, I recommend you think about putting some money away for the rainy day when you might be out of work.

Pro: Consultants Have “Freedom”

I usually hear this one from folks who are unhappy with their current job duties and are envious of my ability to “pick and choose” the work that I take on.  During good economic times, I do have a certain amount of leeway on the jobs I decide to take on and can optimize for more interesting assignments.  But during the bad times, you take whatever you can get.

Also, having taken on an assignment, you have to see it through to the end.  As an independent consultant, I have a limited of “bandwidth” and can typically only support one or two major clients at a time.  If a really interesting project comes along when I’m busy with other work, I have to let it go by or risk alienating my current clients.  In this business your reputation is the key to your success.  Doing a bad or incomplete job because you let yourself be distracted by the new, shiny contract is a sure path to the end of your consulting career.

Another consulting freedom that I hear FTEs envy is the ability for consultants to take time off “whenever they feel like it”.  Sure, if a client is not expecting me on-site and I don’t have any pressing deadlines, I can take time off whenever I feel like it.  It’s definitely a benefit of my lifestyle.

But you have to understand that I don’t get paid during this time.  Vacation, medical leave, and all other periods of “downtime” that are necessary to ensure your health/sanity and prevent burn-out are all part of that unbillable “overhead” I talked about earlier.  So a better way to talk about this freedom is to say consultants can take time off whenever they can afford to.

One more consulting freedom I wanted to mention is the freedom from a certain amount of organizational politics.  Normally, by the time an organization has made the choice to hire a high-priced expert, they’ve already realized that they have a significant problem and have “cleared the decks” of the typical political impediments to making the problem go away.  This is a wonderful thing.

In Summation

I love the consulting lifestyle, but recognize that it’s not for everybody.  There is substantial risk and you spend a lot of time working on mundane aspects of running your business.  But you can earn good money and enjoy substantial freedoms unavailable to FTEs.  I hope you’ll join me for future articles in this series when I drill down on specific details like figuring your billing rate and managing your cash flow, finding and managing clients, and classic blunders that all new consultants commit.

Follow the Money

I’m eternally amazed at how much cheaper computers, disks, networking gear, and pretty much everything IT-related has become since I started working in this industry.  In general, it’s a great thing.  But my friend Bill Schell recently pointed out one of the darker aspects of this trend during a recent email exchange.  Back in the mid-90’s Bill was running the Asia-Pacific network links for a large multi-national.  The “hub” of the network was a large Cisco router that cost upwards of a quarter of a million dollars.  As Bill pointed out, the company thought nothing of paying Bill a loaded salary of roughly half the purchase price of that router in order to keep it and the corporate WAN running smoothly.

Fifteen years later, you can get the same functionality in a device that costs an order of magnitude or two less.  And guess what?  Companies are expecting the costs associated with supporting these devices and the services they provide to be dropping at roughly the same rate as the cost of the equipment.  This translates to loss of IT jobs, or at least their migration to other IT initiatives.  It doesn’t matter that the functionality of the newer, cheaper devices is the same or perhaps even more complicated than the more expensive equipment they’re replacing.  Nor does it matter that the organization is expecting the same service levels or indeed even increased support for new applications and protocols.  “Do more with less” is the mantra.

This trend has all sorts of implications: hidden inefficiencies because reduced support levels impact critical business processes, significant security holes allowed to remain open due to insufficient levels of staffing and expertise, etc.  But what I want to talk about today is the implications for the career path of my fellow IT workers who are reading this blog.   And let me cut right to the bottom-line.  If you want your IT career to be long and profitable, make sure you’re supporting technology that costs a lot of money.  When you see the price of the equipment you’re managing dropping precipitously, start retraining on something new.

Let me give you an example from the early part of my career.  My first job out of college was doing IT support in an environment where they were dumping their Vax systems that cost hundreds of thousands of dollars for Unix workstations that cost tens of thousands of dollars.  Bye-bye Vax administrators, welcome the new, smaller coterie of workstation admins.  And it’s worth noting also that the Vax admins had replaced a small army of mainframe support folks from the previous generation.

And now 20 years later, commodity hardware and virtualization are forcing my generation of system administrators to move up the food chain in search of employment.  Some folks were lucky enough to keep their jobs in pursuit of server consolidation efforts, but notice that they’re now supporting orders of magnitude more systems in order to justify their salaries in the face of reduced equipment costs.  Storage technology was a nice pot of money to chase for a while there, and many of my people made the transition into SAN administration and similar jobs.  But again downward price pressure is being felt in this arena and the writing is on the wall– “do more with less.”

Some IT career choices seem to have historically provided safe havens.  The cost of database installations seems to have held steady or even increased as organizations have wanted to harness the power of larger and larger data sets and as the number of databases in organizations has exploded.  So DBA has always been a good career choice.  Information Security has also been a steady career choice because its budget is typically a constant fraction of total IT spending, rather than being tied to any particular technology.  Plus all of the recent regulatory requirements have ensured that Information Security’s percentage of the total IT budget has been going up, even as total IT budgets are shrinking.

So please keep these thoughts in the back of your mind as you’re plotting your next career moves in this difficult economy.  I’ve seen too many good friends pushed out the door in the name of “efficiency”.

Barbara Lee (In Honor of Ada Lovelace Day)

March 24 is Ada Lovelace Day.  To honor one of the first female computer scientists, the blogosphere has committed to posting articles about women role models in the computer industry.  This is certainly a scheme that I can get behind, and it also gives me the opportunity to talk about one of my earliest mentors.

When I graduated from college in the late 1980’s, my first job was doing Unix support at AT&T Bell Labs Holmdel.  I learned a huge amount at that job, and a lot of it was due to my manager, Barbara Lee.  “Tough broad” are the only words I can think of to describe Barbara, and I think she’d actually take those words as a compliment.  Completely self-taught, Barbara had worked her way up from the bottom and had finally smacked into a glass ceiling after becoming manager of the Unix administrators for the Holmdel Computing Center.  Barbara was also extremely active in the internal Bell Labs Computer Security Forum, and had earned her stripes tracking down and catching an attacker who had been running rampant on the Bell Labs networks many years earlier.

My vivid mental picture of Barbara is her banging away on her AT&T vt100 clone, composing some crazy complex ed or sed expression to pull off some amazing Unix kung fu, while occasionally taking drags on her cigarette (yes kids, you could still smoke in offices in those days).  Unfortunately, it was those cigarettes that ultimately led to Barbara’s death.

As tough and combatative as Barbara was when dealing with most people, she also had a strong caring streak that she mostly kept hidden.  Part Cherokee, Barbara arranged for much of our surplus equipment to make it to reservation schools whenever possible.  As I recall, we even shipped an entire DEC Vax to a reservation while I was there.  I always wondered what they did with that machine, but I’m sure it got put to good use.

And though she didn’t suffer fools gladly, Barbara occasionally took ignorant young savages like me under her wing.  Seeing that I had an interest in computer security, Barbara actually took me along to some of the Bell Labs Computer Security Forum meetings and to the USENIX Security Conference.  Less than I year out of college and I was getting to hang with folks like Bill Cheswick and Steve Bellovin.  How cool was that?  Without this early prodding from Barbara, I doubt my career would have turned out the way it did.

My favorite Barbara Lee story, however, involves an altercation I got into with the manager of another group.  At Bell Labs, the Electricians’ Union handled all wiring jobs, including network wiring.  I was doing a network upgrade one weekend and had arranged for the Electricians to run the cabling for me in advance of the actual cutover.  Unfortunately, Friday afternoon rolled around and the wiring work hadn’t even been started.

So I called the manager for that group and asked what the status was.  He told me that he was understaffed due to a couple of his people being unexpectedly out of the office and wouldn’t be able to get the work done.  The conversation went down hill from there, and ended up with me getting a verbal reaming and the promise of the Union taking the matter up with Barbara first thing Monday morning.

Needless to say, I was sweating bullets all weekend.  And I can remember the sinking feeling in the pit of my stomach when Barbara walked into my office Monday morning.  “Hal,” she said to me, “you just can’t talk to other managers like you talk to me.”  Then she turned around and walked out and never said another word to me about the incident again.

I’d have walked through fire for that woman.

Never Argue With Your Boss

Early in my career, I had the opportunity to listen to a talk by Bill Howell on “managing your manager”.  I don’t recall much about the talk, but one item that stuck with me was his advice, “Never argue with your boss, because even if you ‘win’, you lose.”

At the time, I was young and cocksure and tended towards confrontation in my interactions with co-workers.  If I disagreed with somebody, we each threw down our best technical arguments, wrangled over the problem, and may the biggest geek win.  Being “right” was the most important thing.  So Bill’s advice seemed outright wrong to me at the time.  Of course one should argue with their boss!  If they were “wrong”, then let’s mix it up and get to the “correct” solution.

Flash forward a few years later and I was working as a Senior Sys Admin at a company in the San Francisco Bay Area.  We were trying to roll out a new architecture for supporting our developer workstations, and I was clashing with my boss over the direction we should go in.  Worse still, the rest of the technical team was in favor of the architecture that I was championing.  True to form, I insisted on going for the no-holds-barred public discussion.  This, of course, transformed the situation from a simple technical disagreement into my completely undercutting my boss’ authority and basically engineering a mutiny in his group.

Matters came to a head at our weekly IT all-hands meeting.  Because of the problems our group was having, both my boss and his boss were in attendance.  Discussion of our new architecture got pretty heated, but I had an answer for every single one of my boss’ objections to my plan.  In short, on a technical level at least, I utterly crushed him.  In fact, in the middle of the meeting he announced, “I don’t need this s—“, and walked out of the meeting.  I had “won”, and boy was I feeling good about it.

Then I looked around the table at the rest of my co-workers, all of whom were staring at me with looks of open-mouthed horror.  I don’t think they could have been more shocked if I had bludgeoned my boss to death with a baseball bat.  And frankly I couldn’t blame them.  If I was willing to engineer a scene like had just transpired in our all-hands meeting, how could they trust me as a member of their team?  I might turn on them next.  Suddenly I didn’t feel so great.

I went home that night and did a great deal of soul-searching.  Bill Howell’s words came back to me, and I realized that he’d been right.  Admittedly, my case was an extreme situation, but if I had followed Bill’s advice from the beginning, things need never have escalated to the pitch that they finally reached.  The next morning, I went in and apologized to my boss and agreed to toe the line in the future, though it certainly felt like a case of too little too late.  I also started looking for a new job, because I realized nobody there really wanted to work with me after that.  I was gone a month later, and my boss lasted several more years.

My situation in this case was preventable.  As I look back on it now, I realize that my boss and I could have probably worked out some face-saving compromise behind closed doors before having any sort of public discussions.  Of course, sometimes you find yourself in an impossible situation: whether because of incompetence, malice, or venality on the part of your management.  In these cases you can sit there and take it (hoping that things will get better), fight the good fight, or “vote with your feet” and seek alternate employment.  The problem is that fighting the good fight often ends with you seeking alternate employment anyway, so be sure to start putting out feelers for a new job before entering the ring.  Sitting there and taking it should be avoided if at all possible– I’ve seen too many of my friends’ self-esteem totally crippled by psycho managers.

Bottom line is that one of the most important aspects of any job is making your boss look good whenever possible.  This doesn’t mean you can’t disagree with your boss.  Just make sure that you don’t have those disagreements publicly and make it clear at all times that you’re not attempting to pre-empt your manager’s authority.  “Managing up” is a delicate skill that needs to be honed with experience, but as a first step at  least try to avoid direct, public disagreements with those above you in the management chain.

And thanks for the advice, Bill.  Even if I didn’t listen to you the first time.

Future Cloudy, Ask Again Later

Recently my pal Bill Schell and I were gassing on about the current and future state of IT employment, and he brought up the topic of IT jobs being “lost to the Cloud”.  In other words, if we’re to believe in the marketing hype of the Cloud Computing revolution, a great deal of processing is going to move out of the direct control of the individual organizations where it is currently being done.  One would expect IT jobs within those organizations that had previously been supporting that processing to disappear, or at least migrate over to the providers of the Cloud Computing resources.

I commented that the whole Cloud Computing story felt just like another turn in the epic cycle between centralized and decentralized computing.  He and I had both lived through the end of the mainframe era, into “Open Systems” on user desktops, back into centralized computing with X terminals and other “thin clients”, back out onto the desktops again with the rise of extremely powerful, extremely low cost commodity hardware, and now we’re harnessing that commodity hardware into giant centralized clusters that we’re calling “Clouds”.  It’s amazingly painful for the people whose jobs and lives are dislocated by these geologic shifts in computing practice, but the wheel keeps turning.

Bill brought up an economic argument for centralized computing that seems to crop up every time we’re heading back into the shift towards centralized computing.  Essentially the argument is summarized as follows:

  • As the capital cost of computing power declines, support costs tend to predominate.
  • Centralized support costs less then decentralized support.
  • Therefore centralized computing models will ultimately win out.

If you believe this argument, by now we should have all embraced a centralized computing model.  Yet instead we’ve seen this cycle between centralized and decentralized computing.  What’s driving the cycle?  It seems to me that there are other factors that work in opposition and keep the wheel turning.

First, it’s generally been a truism that centralized computing power costs more than decentralized computing.  In other words, it’s more expensive to hook 64 processors and 128GB of RAM onto the same backplane than it is to purchase 64 uniprocessor machines each with 2GB of RAM.  The Cloud Computing enthusiasts are promising to crack that problem by “loosely coupling” racks of inexpensive machines into a massive computing array. Though when “loose” is defined as Infiniband switch fabrics and the like, you’ll forgive me if I suspect they may be playing a little Three Card Monte with the numbers on the cost spreadsheets.  The other issue to point out here is that if your “centralized” computing model is really just a rack of “decentralized” servers, you’re giving up some of the savings in support costs that the centralized computing model was supposed to provide.

Another issue that rises to the fore when you move to a centralized computing model is the cost to the organization to maintain their access to the centralized computing resource.  One obvious cost area is basic “plumbing” like network access– how much is it going to cost you to get all the bandwidth you need (in both directions) at appropriately low latency?  Similarly, when your compute power is decentralized it’s easier to hide environmental costs like power and cooling, as opposed to when all of those machines are racked up together in the same room.  However, a less obvious cost is the cost of keeping the centralized computing resource up and available all the time, because now with all of your “eggs in one basket” as it were your entire business can be impacted by the same outage.  “Five-nines” uptime is really, really expensive.  Back when your eggs were spread out across multiple baskets, you didn’t necessarily care as much about the uptime of any single basket and the aggregate cost of keeping all the baskets available when needed was lower.

The centralized vs. decentralized cycle keeps turning because in any given computing epoch the costs of all of the above factors rise and fall.  This leads IT folks to optimize one factor over another, which promotes shifts in computing strategy, and the wheel turns again.

Despite what the marketeers would have you believe, I don’t think the Cloud Computing model has proven itself to the point where there is a massive impact on the way mainstream business is doing IT.  This may happen, but then again it may not.  The IT job loss we’re seeing now has a lot more to do with the general problems in the world-wide economy than jobs being “lost to the Cloud”.  But it’s worth remembering that massive changes in computing practice do happen on a regular basis, and IT workers need to be able to read the cycles and position themselves appropriately in the job market.

Making Mentoring a Priority


I always appreciate (and am in search of) tips for how to be a better sysadmin. I’ve never had the opportunity … to be in a large IT org. I think I miss out on a lot of learning opportunities by not being a part of a large IT org.

from a comment by “Joe” to “Queue Inversion Week

This comment reflects an industry trend that I’ve been worrying about for a while now.  Back in the 80’s when I was first learning to do IT Operations, it seemed like there were more opportunities to come up as a junior member of a larger IT organization and be mentored by the more senior members of the team.  It’s not overstating the case to say that I wouldn’t appear to be the “expert” that I seem to be today without liberal application of the “clue bat” by those former co-workers (and thanks to all of you– some of you don’t even know how much you helped me).

These days, however, it seems like there are a lot more “one person shops” in the IT world.  And a lot of IT workers are learning in a less structured way on their own– either on the job, or by fooling around with systems at home.  When they get stuck, their only fallback may be Google.  This has to lead to some less-than-optimal solutions and a lot of frustration and burn-out.

So if you’re a one person shop and you’re feeling the lack of mentoring, let me give you some suggestions for finding a support network.

Local User Groups

See if you can find a user group in your area.  Aside from the fact that most local groups sponsor informative talks, they’re also a good way to “network” with other IT folks in your area.  These are people you can call on when you get stuck on a problem.  There’s also the pure “group therapy” aspect of being able to be in a room with people who are living with the same day-to-day problems that you are and understand your language without need of Star Trek technology translation devices.

Google can help you find groups in your area.  Both SAGE and LOPSA also track local IT groups that are affiliated with those organizations.

If you can’t find an existing local group in your area, you might consider starting one.  I’ve found LinkedIn to be helpful for finding other IT people in my geographic area and contacting them.

Mailing Lists and Internet Forums

I subscribe to several IT-related mailing lists with world-wide memberships.  Some of the most active and useful mailing lists for getting questions answered seem to be the SAGE, LOPSA, and GIAC mailing lists, though there are membership costs and/or conference fees associated with getting access to these lists.  Also, there’s nothing that says you can’t subscribe to the mailing lists for various local user groups, even if you’re not actually close enough to attend their meetings.

There are of course different Internet forums where you can post questions and where you might actually get questions answered occasionally.  I haven’t done an exhaustive survey here, but I have found good Linux advice at the Ubuntu Forums and  If you have favorites, you might mention them in the comments section.

Live Mentoring

This one is scary for most people, but you might consider contacting somebody who you think is an “expert” and asking them out to coffee/beer/lunch/dinner.  If they’re too busy, they’ll tell you.  But if you don’t ask you’ll never know, and you might be missing out on a great opportunity.

You must understand that my expectation is that if somebody helps you in this way, you are morally obligated to help someone else in a similar fashion in the future.  This is why I think you’ll find that most “experts” worth their salt are more than willing to extend this courtesy to you– somebody in their past provided them with guidance, and they’re just “paying back” by helping you.

Teaching Others

If my last idea was scary, this one will probably make you want to hide under a rock.  But teaching others is a great way to motivate yourself to learn.  I find that I don’t really master a subject until I have to organize my thoughts well enough to convey it to others.

Can’t locate anybody nearby to teach at?  Start a blog and write down your expertise for others to read.  Answer questions for some of the users on the Internet forums mentioned above.  Submit articles to technical journals (as the former Technical Editor for Sys Admin Magazine, I can attest that most of these publications are absolutely desperate for content)– some of them even pay money for articles.

If you’ve taken a SANS course and obtained your GIAC certification, you may be eligible to become a SANS mentor.  This can be an entre into becoming a SANS Instructor, and is therefore well worth pursuing.

In Conclusion

It’s unfortunate that there are so many folks out there without the built-in support network of working in a large IT organization.  But if you search diligently, I think you may be able to find some other people in your area to network with and get guidance from.  Remember that we all have different levels of expertise in different areas, so sometimes you’re the apprentice and sometimes you’re the “expert” (I’m constantly learning things from my students– yet another reason to teach others).

For the Senior IT folks who are reading this blog, I ask you to please make it a priority to reach out to the more junior members of our profession and help bring them along.  Somebody did it for you, and now it’s your turn.