Ex3 Café

Our Data Is All F'd Up (Part II)

Last month I wrote a blog post which proposed some reasons why the data always seems to be "f'd up" at companies with centralized data warehouses.  The reasons I cited were equally divided between the business and IT sides of a company.  In this blog post, I will propose some solutions to these issues.

IT Contributions to Messed Up Data

• Lack of Data Warehouse Modeling Experience:  Easy solution.... get developers that have data warehouse modeling experience (theoretical and practical).  If you have existing developers who don't have data warehouse modeling experience, either move them to a different (non data modeling) role or provide them with proper training.  If they don't want to do either, as Alec Baldwin said in Glengarry Glen Ross, tell them to "hit the bricks", as they will be a detriment to you, and your organization, in the long run.
  
  
• Overwhelmed with New Data Requests:  Implement processes and procedures to handle new data requests.  This could also include implementing new technologies to aid in this process.  And if necessary, lobby to get more money and hire more resources.

• Lack of Processes to Manage Source System Data Changes:  Implement processes and procedures, with source system database administrators (DBA's), to alert the data warehouse DBA's that changes have been made on the source system(s) that will effect the data warehouse.  Like above, there may be technologies that could be acquired or built that could aid in this process.

Business Contributions to Messed Up Data

• Lack of Basic Understanding About IT and Data:  Similar point #1 in the IT section above, get business people that have a basic understanding of database principles and have worked with IT people on data previous warehouse projects.  If you have existing business people who don't have this experience, either move them to a different role or provide them with proper training.  If they don't want to do either, get rid of them, as they will be a detriment to you, and your organization, in the long run.

• Haphazard, Last Minute Data Requests:  l know in many cases that this is unavoidable (due to current business conditions) but in some instances this behavior is unnecessary and can be avoided by a little proper prior planning.
  

• Not Educating IT on How Business Actually Uses the Data:  In this instance a little bit of time spent, on the part of the business, can go a long way.  This could include scheduling educational meetings with the IT department to demonstrate how their efforts translate into business value.  Or it could be little efforts on the part of the business.  I remember one time when I was talking on the phone to a DBA in India about a problem I was having with the database.  To better demonstrate what I was doing with the data, I set up a web conference with the DBA to show him exactly how I was using the data (on on front end).  Suffice-it-to-say,  I made a friend for life!

I know most of these solutions are lacking in details, but this (at least) gets the juices flowing.

Our Data Is All F'd Up...

I've heard this phrase, in one form or another, at virtually every company I've consulted for that has a centralized data warehouse.  After recently hearing this—yet again—I thought to myself, "why is the data always f'd up?" 

As I pondered this thought, I looked at the issue from both the IT and business sides of a company.  I've consulted in both areas so I have an unique view across both worlds.

IT Contributions to Messed Up Data

The following are observations I've made, while consulting on the IT side, which are likely key contributors to messed up data:

• Lack of Data Warehouse Modeling Experience:  More times than not, IT departments will put developers (i.e. C++, Java, PHP...etc.) in charge of developing the data model for the data warehouse.  The problem here is while one might be a good "coder," this doesn't mean that the same person has the proper skills and training to develop a robust data model for the data warehouse that's easy for a business/data oriented person to work with.  Some of the worst data models I've seen were developed by great programmers!
  
  
• Overwhelmed with New Data Requests:  Often times, if you manage a corporate data warehouse, you're constantly getting requests from marketers to add new data to the data warehouse.  If you don't have good processes and procedures in place to handle these requests or you're significantly under staffed, you will likely end up with an ineffective (or worse yet) inaccurate data warehouse resulting in a lot of unhappy business customers.
  
  
• Lack of Processes to Manage Source System Data Changes:  Unfortunately, mapping new source system data into a data warehouse is not a "set it and forget it" exercise.  This is because source system data can change overtime.  This can be a disaster on the marketing side, which can lead to inaccurate customer targeting or personalizing communications with the wrong content.  If there are no processes in place for the source system data administrators to alert the data warehouse administrators of data changes it's only a matter of time before a marketing crisis occurs.

Business Contributions to Messed up Data 

The following are observations I've made, while consulting on the Business side, which are also likely key contributors to messed up data:

• Lack of Basic Understanding About IT and Data:  It's vital that at least one business stakeholder have a solid understanding of basic database principles and working with IT departments.  Without this knowledge, business requirements and decisions tend to be driven by the IT side.  This can lead to systems being delivered that are substandard and have low user adoption.
  
  
• Haphazard, Last Minute Data Requests:  I see this all of the time.  Knee jerk requests from the business to add new data to the data warehouse.  Everything is a last minute fire drill.  This type of behavior leads to mistakes and it overwhelms IT business partners, thus causing friction between IT and business.  In some cases, it's unavoidable to make last minute requests.  In most cases though, a little advanced planning could help prevent such fire drills from happening.
  
  
• Not Educating IT on How Business Actually Uses the Data:  I've found that educating the IT department on how the business actually uses an application or database goes a long way.  Without providing this education, the IT department tends to operate in a vacuum and thus delivers substandard solutions to the business.

How to Fix The Problem

I'll leave this to my next blog post so stay tuned...

Note to IBM: Please Make Data Mining Work for Mainstream Marketing This Time

With IBM's recent press release announcing a new consulting organization dedicated to advanced business analytics, I thought to myself, "I sure hope this new organization helps to incorporate advanced analytics/data mining into mainstream marketing this time around."  Because up to now, other than a few BIG exceptions (i.e. Amazon.com), the promise of integrating data analytics and marketing (in a scalable and repeatable manner) has yet to be realized.  It's quite frustrating, as data mining technologies have been around for a number of years and incorporating data mining into marketing seems like such a natural fit.  For example, customer exhibits pattern of behavior for product cross-sell opportunity so complementary product offering is dynamically presented to customer and add on sale is made.  Done.  End of story.  Not so fast.  Here's what's throwing a wrench into this seemingly straight forward scenario:

• The PhD's in machine learning and mathematics (in the Reporting and Analytics Division) and the MBA's (in Marketing) are wired completely different.  Here, the PhD's are concerned about accuracy (to the n'th degree), methodology, sample size...etc. and the marketers are concerned about "just getting the stuff out the door".
• For direct marketing, most marketers are still obsessed with list quantity versus list quality.  "You mean my list count is only 150,000, I thought it was going to be more like 2,000,000?"  In this case, if a list (generated via data mining) has a smaller count than expected, the marketer may request that the list be pulled using more traditional data selection/segmentation methods to increase the list count (completely missing the point).
• The raw data (to be mined) is never right.  So the data mining folks spend most of their time "getting the data ready to mine" versus actually mining the data.
• Very few companies have figured out how to integrate data mining and marketing execution efficiently (or at all).  As such, a lot of interesting discoveries are made, via data mining/analytics, but no direct action is taken to help generate additional sales from these discoveries.  For example, through mining its sales transaction data, an on-line retailer may have discovered that customers who purchase products in category A, are likely to purchase products from category B as well.  Unless there's a process/technology in place to alert the customer about products in category B (i.e. automated e-mail, website message...etc.) this discovery, while perhaps interesting, has limited or no monetary value.

So, with the above realities in mind, my advice to IBM (if the company wants to be successful at selling data analytics products and services to marketers) would be the following:

• Figure out a way to bridge the communication barrier between the data modelers/analysts and the marketers.
• Set expectations for senior management on the process of data mining.  Many senior executives believe the process involved with advanced data analysis and modeling is likened to the Ronco "Set It and Forget It" Rotisserie Oven.  Here the assumption is that a magical software tool is turned loose against a sea of data and (miraculously) revenue generating discoveries pop out the back end with limited or no human intervention.  Obviously there is a lot more to data analytics/modeling than this, so senior management's expectations need to be managed accordingly.
• Make sure the raw data that's being analyzed/mined is complete, accurate and structured correctly on an on-going basis.  Otherwise you will have highly skilled data modelers spending most of their time doing manual, mundane data "grunt work" instead of the work they are hired (and paid handsomely) to do.
• Educate marketers on the value of data quality versus data quantity.  Here various tests could be set up to compare the marketing ROI between segments derived via data analytics and segments derived using more traditional data selection methods. 
  NOTE:  I find that numbers speak much louder than words when it comes to proposing new marketing techniques and technologies to marketers.
• Don't focus solely on analytics and discovery.  Make sure you also provide the customer with a scalable and repeatable method to "take action" when key, revenue generating discoveries are made.  In other words, help the customer fill the gap between marketing discovery and marketing execution.

If IBM is mindful of the above recommendations, I believe this newly formed consulting organization will be highly successful at selling data analytics products and services to marketers.

P.S.  Here's an interesting video I found that helps answer the question "what is data mining?":