June 12, 2013 in data warehouse consulting
Why data warehouse projects fail (3)
In this article we’ll take a final look at why data warehouse projects fail. This article groups data warehouse, big data and business intelligence under the single term “data warehouse”. The reason being that they are interconnected and have identical problems, people.
This article may not be a friend-winner for some readers, but I’m not here to make friends, I’m here to make sure that projects get delivered and don’t fail.
Blind faith in technology
My opening statement is that the box of hardware and software chosen to run your system is not the problem. Most people are sensible enough not to use something that is not trusted. The “blind faith” part is that the chosen box is the miracle medicine for the perceived business problems it is meant to cure.
The box itself has 3 potential sources: in-house, 3rd party vendor, a combination of in-house and 3rd party vendor.
When it comes to business decisions, many people at all levels, have less trust in their own system development staff than they do in a 3rd party vendor’s staff. This is an understandable situation as the attitude is that if vendor “ABCD” has been selling the “WXYZ” package for a few years then it must be good. Frequently the vendor becomes the expert not only in the technology but also in how it is applied to and managed by the business. The latter implies a management consultancy role which will have an impact on business strategy, a potentially very dangerous thing indeed.
The main reasons behind this internal problem are:
- a lack of clear strategic vision
- failing to fully understand the underlying business problem
- blind faith in a 3rd party vendor solution or service
Would you buy a sports car for use on a farm? Probably not. You’d probably be better off buying a tractor. The sports car will work but it is inadequate within the environment in which it will be asked to work. This is a simple example but the challenge lies in understanding the problem before selecting the product.
This is in no way intended to criticise vendor solutions or services, but any organisation must accurately assess its needs before looking for a vendor solution. If this is not done and the vendor is asked to do it, there will be problems. The main problem (I’ve lost count of the number of times I’ve seen and heard people alluding this) is trying to shoe-horn your current business model into a vendor’s product. This may even be impossible to shoe-horn it in with the result that your business model is now dictated by the vendor’s product.
Relying heavily on a vendor product is dangerous as anyone who has used MS Windows for a few years will know. Unlike some other IT big names, MS hasn’t gone bankrupt but it constantly changes the way we work. It forces us to do things rather than giving us a choice. When it does give us a choice, it’s akin to “Hobson’s choice“. The same applies to any 3rd party product which is why you must fully understand what you want before you buy one. Blind faith in technology, regardless of who the vendor is, is another reason why data warehouse projects fail. If the shoes don’t fit, don’t buy them.
Data – the more you have, the hungrier you get
Big Data is promising the world. Estimates for future data volumes are booming. All that data must mean that we’re all going to get rich. We’re all going to radically improve our sales by monitoring and targeting replies to Face Book, Twitter, Tumblr, blogs, mobile devices, smart phones, tablets, etc. All this being true, “I’m excited so how do we deal with it?”
OK, dogma time is over so let’s get real! The concept of Big Data appears to be based on a NASA style view of the world. Hundreds of PhD’s around the world interpreting results and making decisions. Listening devices plugged in everywhere. Really futuristic stuff. There’s only one minor flaw, exactly who is going to be monitoring and reacting to all of this data in your organisation? Don’t worry, heuristics will do most of the work. Well that’s great as long as your heuristic logic is flawless. Heuristic analysis is excellent, it always gives an answer we can reliably act on. Or does it? Take my sceptical view as being the second state of blind faith in technology. I heard from someone last week that, “If the system says it’s true, it’s true.” My reply was, “Have you ever been overcharged on a computer generated bill? The billing mistake was almost certainly made due to human error. Humans write heuristic programmes. Humans provide parameters for heuristic enquiries. Computers are as brilliant or dumb as the people who programme them. Point taken?
A reality check is required here. It’s one thing having access to all this previously untapped data, and it’s another thing making sense out of it and putting it to good use.
Data is another reason why data warehouse projects fail. The primordial reason is that no-one can decide what data is and is not important. “It’s all important!” is the usual exclamation, but it isn’t. Most businesses operate using a finite and classifiable set of data. Unstructured data is infinite and in its raw state it cannot be classified. A Big Data adaptor’s first job must be to separate the wheat from the chaff and classify the important data. This is basic common sense because our brains have a load limit and they are extremely limited when it comes to trying to make sense out of chaos. Hopefully what I write here is understandable. It follows a set of rules (language) that English speakers understand. It is constructed in a commonly accepted form (grammar/syntax). It only says what it needs to say (meaning). It uses a fraction of the English vocabulary (scope). The words have a common construction (spelling), i.e. the format is identical therefore we understand the representation of each word. Certain words can change their meaning depending on the sentence content (context).
Failure to define data: scope, meaning, syntax (grammar), format (spelling), context and language spell the death of a data warehouse project. It all boils down to understanding the data within an organisation and the ability to scope all available data into a set that is useful and can be managed. This is far from being an easy task, but it needs to be done in order to create a successful system.
The essence of a data warehouse is to create a standard language to represent the subset of the organisation’s important data it contains. Achieving this demands translation (there wouldn’t be a “T” in ETL if it didn’t) but translation (or as it is less accurately known, “transform”) is a secondary and technical exercise. The important part is to understand the data in terms of its own language. Only then can the exercise in translation begin. Once again, this is no easy job but it is impossible to translate without cross “language” knowledge.
Resources – people not machinery
Data warehouse projects generally don’t fail because of a lack resources. If your organisation doesn’t have the resources, vendors are understandably more than happy to provide extra resources.
Internal resources that are faithful to the organisation are crucial to a data warehouse project’s success. It doesn’t matter whether the internal resources are permanent employees or temporary/contract employees. The key factor is that their loyalty lies with you. This may sound medieval but on a project that crosses multiple business lines and possibly multiple vendors, the last thing you need is a knife vibrating between your shoulder blades.
This is a prickly subject because vendors will claim that they always have the clients best interests at heart. This is true when the vendor wants to add another 5 star name to their client list but it is less true on the financial side as a vendor makes money out of selling a product or service and also providing expertise. The more expertise it can provide the better. This isn’t a cynical observation, it’s a fact.
The problem here isn’t “greedy” vendors. Vendors are professional companies with a great deal of subject focused knowledge. The failure is generated internally through a lack of resource management and quality control. If you’re paying for a resource, make sure that the resource provided is managed and can do the job.
Over dependence on and under-management of 3rd party vendor resources was one of the reasons a project I worked on was stopped. It simply ran out of money.
Internal resources must cover all phases of the project from its birth as an idea through to its implementation and support. Internal resources must be in charge of all core tasks e.g. management, analysis, design, etc. All 3rd party vendor staff must be managed and controlled by internal resources during the project. This is not to say that they should be treated as second class citizens, but simply that they must be properly allocated and managed. Internal/vendor relationships must be kept, at the very minimum, to a professional and courteous level.
Spiraling external HR costs are yet another reason why data warehouse projects fail.
The final set of answers to why data warehouse projects fail are:
- A blind faith in technology as a cure all to business challenges.
- Data issues regarding identifying what’s important and how to manage it.
- Resource management and control
Published by www.datawarehousemanager.com