I've heard this phrase, in one form or another, at virtually every company I've consulted for that has a centralized data warehouse. After recently hearing this—yet again—I thought to myself, "why is the data always f'd up?"
As I pondered this thought, I looked at the issue from both the IT and business sides of a company. I've consulted in both areas so I have an unique view across both worlds.
IT Contributions to Messed Up Data
The following are observations I've made, while consulting on the IT side, which are likely key contributors to messed up data:
- Lack of Data Warehouse Modeling Experience: More times than not, IT departments will put developers (i.e. C++, Java, PHP...etc.) in charge of developing the data model for the data warehouse. The problem here is while one might be a good "coder," this doesn't mean that the same person has the proper skills and training to develop a robust data model for the data warehouse that's easy for a business/data oriented person to work with. Some of the worst data models I've seen were developed by great programmers!
- Overwhelmed with New Data Requests: Often times, if you manage a corporate data warehouse, you're constantly getting requests from marketers to add new data to the data warehouse. If you don't have good processes and procedures in place to handle these requests or you're significantly under staffed, you will likely end up with an ineffective (or worse yet) inaccurate data warehouse resulting in a lot of unhappy business customers.
- Lack of Processes to Manage Source System Data Changes: Unfortunately, mapping new source system data into a data warehouse is not a "set it and forget it" exercise. This is because source system data can change overtime. This can be a disaster on the marketing side, which can lead to inaccurate customer targeting or personalizing communications with the wrong content. If there are no processes in place for the source system data administrators to alert the data warehouse administrators of data changes it's only a matter of time before a marketing crisis occurs.
Business Contributions to Messed up Data
The following are observations I've made, while consulting on the Business side, which are also likely key contributors to messed up data:
- Lack of Basic Understanding About IT and Data: It's vital that at least one business stakeholder have a solid understanding of basic database principles and working with IT departments. Without this knowledge, business requirements and decisions tend to be driven by the IT side. This can lead to systems being delivered that are substandard and have low user adoption.
- Haphazard, Last Minute Data Requests: I see this all of the time. Knee jerk requests from the business to add new data to the data warehouse. Everything is a last minute fire drill. This type of behavior leads to mistakes and it overwhelms IT business partners, thus causing friction between IT and business. In some cases, it's unavoidable to make last minute requests. In most cases though, a little advanced planning could help prevent such fire drills from happening.
- Not Educating IT on How Business Actually Uses the Data: I've found that educating the IT department on how the business actually uses an application or database goes a long way. Without providing this education, the IT department tends to operate in a vacuum and thus delivers substandard solutions to the business.
How to Fix The Problem
I'll leave this to my next blog post so stay tuned...
Comments