June 5, 2013 in data warehouse consulting
Why data warehouse projects fail
In this article and future articles we’ll take a look at why data warehouse projects fail. This article groups data warehouse, big data and business intelligence under the single term “data warehouse”. The reason being that they are interconnected and have identical problems, people.
At its most basic, a data warehouse system takes in information, passes the information through programmes to create a single set of variables. These variables are then accessed by observers usually for investigation, decision making, “what if” analysis, what can I sell to client profile …, etc.
The diagram below shows this flow. The diagram is an adaptation of Gerald Weinberg’s diagram “the general problem of maintaining identity”. I’ve added 3 additional pieces to the original.
- External environment
- Corporate culture constraints
- Individual constraints
The diagram also shows a large cloud with a leader to corporate culture constraints and a small cloud with a leader to individual constraints. Everything inside the large cloud is subject to corporate culture. This article will only discuss corporate culture constraints.
My choice of a cloud has nothing to do with representing cloud technology and everything to do with representing uncertainty. Anything that can happen, will happen.
The diagram encapsulates the essence of a data warehouse but it cannot represent its purpose. Members of the large cloud will need to define the purpose of the data warehouse which gives rise to the fundamental problem. What is the purpose? Natural systems have no purpose. If they did, we’d know why we existed. Do man made systems have a purpose? They do but it isn’t an absolute. A system’s purpose is relative to the user. For example: Mercedes Benz has a system for making cars. Is the primary purpose of this system:
- to make cars?
- to keep shareholders happy?
- to keep scrap dealers, advertising agencies, etc. in business?
- to create jobs so people can live in relative comfort?
Depending on where you stand, your view of the purpose of Mercedes Benz’s system for making cars is different. Purpose is relative.
The staff members selected to define the purpose of a data warehouse system will view it relative to their own requirements. This will and does give rise to all the negative emotions regarding opinions. Everyone’s purpose will be the best or the most pressing. Thomas Peters states in “In search of excellence” that excellent “companies worked hard to keep things simple in a complex world.” While the book itself is open to criticism, the statement is painfully true.
To modernise a statement by James Clerk Maxwell:
“The success of any investigation depends on the judicious selection of what is to be observed as of primary importance.”
Maxwell goes on to say that we must ignore features which, however attractive they may appear, are not of primary importance or cannot be properly investigated.
So according to Peters and Maxwell, KISS and prioritise. Based on experience, I could not agree more. This may sound like the most obvious statement ever made, but it is still the primary reason that data warehouse projects fail.
On the assumption that you buy in to this statement, the statement begs the question, “who decides what’s important?” Design by committee does not work. Totalitarian or military style organisation structures don’t work either. Consensus is the only way to make the statement work. Consensus is possibly the most difficult thing to achieve in a large organisation.
Consensus decision-making is defined as a group decision making process that seeks the consent of all participants. The aim of consensus decision-making is to improve solidarity. Here’s a good high-level article on wikipedia about it.
The problem of defining purpose can be solved by consensus decision-making but the group of decision makers should be no larger than six. According to Bob Frisch and Josh Peck of Strategic Offsites Group, “when the stakes are high, it is best to limit your session to the key decision-makers, typically from three to six participants.” As relationships are vital in this situation 3 people equates to a total of 3 bilateral relationships and 6 people to 15 bilateral relationships. More people = more relationships = more difficulty.
This group must agree and support with the fervour of an evangelist, the purpose of the data warehouse. Once the purpose (i.e. relationship with the business) of the data warehouse has been agreed on it must become published strategy. By doing this, it reinforces executive support and provides a firewall against unwarranted change.
It is guaranteed that not everyone involved with the project to implement the data warehouse will agree. People will ask for changes. As long as an efficient and credible “grievance” and change control mechanism is implemented, these issues can be resolved. Covert negativity, back-door changes and other such “black ops” activity can only be controlled by an organisation’s management structure, reporting structure and cultural values. If an organisation embraces “black ops” activity as an acceptable form of creativity, the project will fail because it will be spiral out of control.
The first answers to why data warehouse projects fail are:
- The lack of an agreed and supported purpose.
- The inability of an organisation’s management structure, reporting structure and cultural values to control “black ops” behaviour.
We’ll venture more into this area in future articles.
Published by www.datawarehousemanager.com