Is the presence of "Shadow IT" an indication of IT department failure?

Information Technology Nov 1, 2021

Yes, this title is a bit of a hot take, and I could be reasonably accused of manufacturing a clickbaity headline.

I didn’t pick it because I am trying to stir controversy though, I picked it because I genuinely feel we should re-examine how we look at “shadow IT” operations inside a company. As a part of the technology community, I feel like we hold a responsibility to shift our attitudes on the topic, and even *gasp!* learn from any shadow IT we might come across.

The failure may not be the fact that there is shadow IT happening, instead, the failure may be with the attitudes towards it. If we called it “getting things done” instead of “shadow IT”, would we approach it differently?

If you aren’t familiar with the term, shadow IT refers to departments or individuals implementing their own information technology solutions in a way that doesn’t align with policies, procedures, or existing solutions from the IT department.

Wikipedia describes the concept by saying it “…refers to information technology (IT) systems deployed by departments other than the central IT department, to work around the shortcomings of the central information systems”. Immediately after this definition though, the statement is made: “Shadow IT systems are an important source of innovation, and shadow systems may become prototypes for future central IT solutions.”

Some examples of shadow IT might include:

Using an unsanctioned SaaS, especially with company data being supplied to it
Spreadsheets, access databases and applications, crud apps, and no/low-code applications being deployed and used throughout the company
Unauthorized software being installed and used on company computers
Collecting data from unauthorized sources, and using it in unintended ways. For example – using information from an operational application to calculate or infer financial data

Before taking the discussion any further, I need to make something clear: companies must protect their data and systems; not doing so risks the entire company and is irresponsible on multiple levels. Security, compliance, and systems performance all have to be considered.

I am not advocating for some kind of a free-for-all across that data or an anarchist approach to maintaining those systems, but I am saying that IT policies and attitudes often don’t keep up with many available innovations that could make a company more competitive.

The different technologies that get labeled as “shadow IT” are like any other technology – they aren’t good by themselves and they aren’t necessarily bad. If there are solutions in place that are labeled as shadow IT, it may be an indication that there is room for improvement with how IT is administered and managed at a company.

Examining shadow IT from a traditional perspective

With that out of the way, let’s all put our traditional, big-company CIO hats on for a bit and take a look at some scenarios:

“Why in the world did they sign up for HubSpot? We already have salesforce. We are paying through the nose for it, and I have 5 people on staff just to keep all our integrations working. I can’t have another CRM to worry about”.
“You need to take down that Tableau server right now. It’s hammering our application database, and the whole thing is going to crash. I need you to tell me right now who gave you access to connect it, and I am going to fire them immediately!”
“Where did all these reports come from? Every department has their own way of measuring things, and they are all wrong! We need to find each of these reports and get them turned off; we can’t have everybody generating these all ad-hoc like this, and actually thinking they can run the business off of them!”
“Why is that software installed across your entire department? We haven’t done a security audit, a risk assessment, or even validated that it is compatible with our systems!”
“We found yet another Dropbox installation. Looks like the whole finance department was all sharing all of our invoices on Dropbox, making it available to who knows who rather than using Onedrive like they are supposed to.”

Reasonable concerns, right? Sure, however, the difference between a successful response here and a failed one will have a dramatic effect on the potential success of the company.

Understanding why shadow IT happens

If we swallow our pride a little bit while we have these traditional, big-company CIO hats on, we might be able to see that there are legitimate reasons why users are doing what they are doing. These guys don’t understand security, compliance, application reliability, or any of the things that that we are responsible for in the IT department – they just want to get their jobs done. Rather than insisting that their lack of knowledge is proof that they shouldn’t have anything to do with technology systems, is there something I could have done differently to achieve a different outcome?

Do people actually have access to the data that they need to perform their jobs?

Data has to be controlled, right? Well, yes, but probably not as tightly as you think.

Does everybody need to see the credit card numbers involved in all recent transactions or the social security numbers of all your users? No, probably not.

Should all managers have access to item-level detail about every single sale the company has ever made? Maybe! Company leadership needs to make that decision, and if it will help those managers do their jobs better then by all means they should have access.

If you think that business users need to come groveling to you to get a new report (which means going through several design sessions, just to have it put in a development queue that is currently 3 months long), your company is going to be out-competed by a business where the users can generate their own report in a few clicks.

If you think that you own and define the data and have to be the gatekeeper and final arbiter for what tools people use and how they access the data, your business will suffer for it. Your business users almost certainly understand the data better than you do, so why limit what they see and what they can do with it? Sure, learn from what they are producing and standardize what is available to everyone, but don’t prevent them from having what they need to do their jobs.

Is there a reasonable path for new software to be used at the company?

If the process to request and approve new software systems or services looks like something that would appear in a Dilbert cartoon, then of course people are going to bypass it.

If people are signing up for Hubspot in the example above, they either don’t know about your Salesforce installation or it doesn’t work for them. If you don’t have processes and communication patterns for people to understand what solutions are available to them and discuss with you how they can be improved, then you naturally won’t be part of the conversation when the topic comes up.

You can’t know everything about every solution available to help your company operate better. Let those with domain knowledge decide what tools will allow them to do their jobs the best, and then help them make those tools work.

Do people have the autonomy to create tools that they need?

We’ve all seen those spreadsheets done up by some Excel wizard joining 14 different data sources together. That spreadsheet, or something like it, probably runs half the departments in your company. That Excel wizard might not even work for the company anymore, but the department (and by extension the company) is now completely dependent on that spreadsheet.

There’s a lot of things wrong with this scenario, right? Yes. However, you likely have no idea what the tool does, why it is so important, what those 14 data sources are, or anything else about it. Until you fully understand the problem domain, you shouldn’t be sending your minions in so they can “correct” the situation.

Why not just give the department the tools to better take care of things themselves? Understand what data their spreadsheet is consuming, and give them (controlled) access to all the data they need. Help them unify the data in ways that make sense – and in ways that others in the company can leverage too. Every department has a few people who have mastered Excel, or who can hold their own with advanced SQL queries, or enjoy using Jupyter notebooks to perform predictive analysis or have a collection of little apps that streamline their jobs. Rather than bringing down the ban hammer, provide those people with an environment where they can do all those things in secure, compliant ways. Rather than treating those individuals as threats, recognize them as the motivated, resourceful assets they are and help them go do great things.

Learning from shadow IT

So, when you learn of shadow IT happening, what can you do? If you read this far maybe I can admit that it is slight hyperbole to label every single instance of shadow IT as a failure of the IT organization, but there is still an opportunity to improve what you offer the company.

Ideally, you can proactively create an environment where shadow IT isn’t needed at all, but it might be too much to ask that you can always anticipate every need of every business user in the company. With that in mind, here are a few pointers to consider overall:

Recognize that users aren’t trying to bypass security or cause problems, they just want to get a job done.
Make as much data as possible available to as many people as possible. Data wants to be free, so help it get there! Restrictions should be decided by company leadership, not by somebody filling a bureaucratic, administrative role in the IT department.
If you see somebody implementing a shadow IT solution, embrace it! Assume the best, and barring willfull intent to cause problems, recognize that they found a way to do their jobs better.
Perhaps your definition of shadow IT is worth revisiting. Is it really a concern if there are two distinct CRM’s being run at the company? Is this a decision to be made by the entire leadership team, or something for the IT department to decide?

In short, why not work with everyone in the organization and promote shadow IT to be treated like regular IT? Why not take the innovative ideas and attitudes that brought shadow IT about, and encourage them? Take those innovative solutions, and help them become first-class citizens.

Everybody at a company should be working together to help the company be successful, no matter what role each contributor plays. By supporting everyone trying to achieve that goal, you can improve the success of the company overall as well and make the lives of the different team members inside the company more pleasant.

When team members don’t feel it necessary to go outside the system to get things done, they are happier. When you don’t have people trying to go outside the system, your life is made easier.

Sounds like a win/win to me!