Skunkworks in the Clouds

Skunks... clouds... get it?I was recently asked to make a guest appearance on a podcast related to information security in “the cloud”.  One of the participants brought up an interesting anecdote from one of his clients.  Apparently the IT group at this company had been approached by a member of their marketing team who was looking for some compute resources to tackle a big data crunching exercise.  The IT group responded that they were already overloaded and it would be months before they could get around to providing the necessary infrastructure.  Rebuffed but undeterred, the marketing person used their credit card to purchase sufficient resources from Amazon’s EC2 to process the data set and got the work done literally overnight for a capital cost of approximately $1800.

There ensued the predictable horrified gasping from us InfoSec types on the podcast.  Nothing is more terrifying than skunkworks IT, especially on infrastructure not under our direct control.  “Didn’t they realize how insecure it was to do that?” “What will happen when all of our users realize how easily and conveniently they can do this?” “How can an organization control this type of risky behavior?” We went to bed immersed in our own paranoid but comfortable world-view.

Since then, however, I’ve had the chance to talk with other people about this situation.  In particular, my friend John Sechrest delivered an intellectual “boot to the head” that’s caused me to consider the situation in a new light.  Apparently getting the data processed in a timely fashion was so critical to the marketing department that they figured out their own self-service plan for obtaining the IT resources they needed. If the project was that critical, John asked, was it reasonable from a business perspective for the IT group to effectively refuse to help their marketing department crunch this data?

Maybe the IT group really was overloaded– most of them are these days.  However, the business of the company still needs to move forward, and the clever problem-solving monkeys in various parts of the organization will figure out ways to get their jobs done even without IT support. “Didn’t they realize how insecure it was to do that?”  No, and they didn’t care.  They needed to accomplish a goal, and they did.

“What will happen when all of our users realize how easily and conveniently they can do this?” My guess is they’re going to start doing it a lot more.  Maybe that’s a good thing.  If the IT group is really overloaded, then perhaps it should think about actually empowering their users to do these kind of “one off” or prototype projects on their own without draining the resources of the core IT group.  Remember that if you let a thousand IT projects bloom, 999 of them are going to wither and die shortly thereafter.  Perhaps IT doesn’t need to waste time managing the death of the 999.

“How can an organization control this type of risky behavior?” You probably can’t.  So perhaps your IT group should provide a secure offering that’s so compelling that your users will want to use your version rather than the commodity offerings that are so readily available.  This solution will have to be tailored to each company, but I think it starts with things like:

  • Pre-configured images with known baseline configurations and relevant tools so that groups can get up an running quickly without having to build and upload their own images.
  • Easy toolkits for migrating data and out of these images in a secure fashion, with some sort of DLP solution baked in.
  • Secure back-end storage to protect the data at rest in these images with no extra work on the part of the users.
  • Integration with the organization’s existing identity management and/or AAA framework so that users don’t have to re-implement their own solutions.
  • Integration with the organization’s auditing and logging infrastructures so you know what’s going on.

Putting together the kind of framework described above is a major IT project, and will require input and participation from your user community.  But once accomplished, it could provide massive leverage to overtaxed IT organizations.  Rather than IT having to engineer everything themselves, they provide secure self-service building blocks to their customers and let them have at it.

Providing architecture support and guidance in the early stages of each project is probably prudent.  After all, the one hardy little flower that blooms and refuses to die may become a critical resource to the organization that may eventually need to be moved back “in house”.  While the fact that the building blocks that were used to create the service are already well-integrated with the organization’s centralized IT infrastructure will help, having a reasonable architectural design from the start will also be a huge help when it comes time to migrate and continue scaling the service.

Am I advocating skunkworks IT?  No, I like to think I’m advocating self-service IT on a grand scale.  You’ll see what skunkworks IT looks like if you ignore this issue and just let your users develop their own solutions because you’re too busy to help them.


2 thoughts on “Skunkworks in the Clouds”

  1. Hal, as always I enjoyed the post.

    IT projects like the one you defined above will happen as long as IT people fail to redefine their roles to match current corporate strategies. Keep in mind, I run an engineering and engineering services team so I’m familiar with the headaches and the trade-offs between long term strategic goals, security needs, and best practices.

    However, IT often is at the intersection of the tactics teams must use to either reduce operational costs or achieve revenue targets. IT leaders must constantly reach out to their peers to:

    – Maintain alignment with the greater corporate strategy.
    – Communicate key IT priorities to ensure time is budgeted for them, but understand that IT-internal priorities are but one piece of their deliverabls.
    – Work across teams to understand which team’s goals are the highest priorities and constantly communicate on why IT resources are being invested in certain ways.
    – Support non-internal IT driven projects in a manner that does not put critical corporate data at risk but enables business units to succeed when internal IT resources cannot assist.

    It’s a balancing act but I would argue that IT leaders that focus exclusively on IT-internal projects will not be the IT leaders of tomorrow.


  2. It is also a matter of management and setting priorities. It isn’t always up to IT to determine what is important.

    How did the marketing department present their need? Did they give explicit detail showing how this was so critical it had to be done overnight? Was there ome higher-level authority involved who would be setting the priorities for IT projects or the company goals/needs which IT was already invovled in? Did IT explain to Marketing that there were other, higher priority projects in the works and what Marketing’s steps should be to get approval to jump tot he front of that line?

    I hear what you’re saying about overloaded IT staff and consideration of what is important to the business. Hwoever, when items like this come up, it isn’t always just up to IT to make such a decision as to what is important to the business, depending on how the overall organization is run and how IT’s priorities are set and reset in line with the business priorities.

Comments are closed.