Black

Swans &
White Elephants

Return On Investment
In Business Intelligence

David M Walker

Data Management
& Warehousing

ETIS ‒ Istanbul ‒ Oct 09

 Return  on  Investment  
  A  performance  measure  used  to  evaluate  the  efficiency  of  an  
investment  or  to  compare  the  efficiency  of  a  number  of  
different  investments.  To  calculate  ROI,  the  benefit  (return)  of  
an  investment  is  divided  by  the  cost  of  the  investment;  the  
result  is  expressed  as  a  percentage  or  a  ratio.    
  The  return  on  investment  formula:  
ROI  =  (Gain  from  Investment  –  Cost  of  Investment)  
Cost  of  Investment  
  Return  on  investment  is  a  very  popular  metric  because  of  its  
versatility  and  simplicity.  That  is,  if  an  investment  does  not  have  
a  positive  ROI,  or  if  there  are  other  opportunities  with  a  higher  
ROI,  then  the  investment  should  not  be  undertaken.  
 So  where  are  the  ROI  Metrics?  
  Clearly  valuing  BI  is  not  an  exact  science  and  often  comes  down  
to  mindset.  Comparing  BI  to  a  college  education:  It  may  be  
expensive  and  time-­‐consuming,  but  there  are  many  less  
tangible  benefits,  like  increased  earning  power  and  overall  
improved  quality  of  life,  which  come  years  later.  

  It's  not  easy  to  persuade  someone  to  go  to  college  based  on  a  
purely  financial  or  numbers  game,  and  the  same  thing  goes  for  
BI.    You  just  have  to  believe  that  BI  is  absolutely  essential  for  you  
as  an  organization  to  invest,  that  this  is  a  fundamental  core  
competency  that  you  have  to  have.      
Bill  Hostmann,  Vice  President  &  Analyst,  Gartner,  March  2008  
 What  is  Good  BI?  
  Good  BI  is  the  fusion  of  the  right  information,  the  right  time,  
the  right  format,  and  the  right  human  and/or  system  resources.  
If  we  wish  to  improve  BI,  we  ask  these  questions:  
  Do  business  users  have  the  information  needed,  when  they  need  it,  
to  make  decisions?  
  Do  those  people  have  the  expertise,  training  and  mindset  to  use  
that  information  in  the  best  way  for  the  good  of  the  organization?  
  Are  they  doing  their  job  better  because  of  the  information  being  
delivered?  
  How  much  difference  does  that  information  make  to  them?    
Dorothy  Miller,  BI  Metrics,    Feb  2009  

  The  best  approach  to  evaluating  a  BI  solution  is  a  technical  

review  combined  with  a  business  user  perception  survey  
 Proposition  
  No  organisation  has  ever  delivered  the  ROI  originally  
promised:  
  The  business  has  moved  on  in  the  time  it  takes  to  develop  
the  (original)  solution  meaning  that  the  value  has  diminished  
   The  biggest  gains  and  losses  have  come  from  Black  Swans  
and  not  planned  events  
  The  biggest  costs  have  come  from  White  Elephants  and  the  
failure  to  recognise  where  the  costs  are  hidden  

  Some  (mainly  those  highly  committed  to  BI)  have  far  

exceeded  the  promised  ROI  but  most  fail  to  achieve  it    
  Even  if  the  development  costs  are  on  budget  the  on-­‐going  
OPEX  costs  are  hidden  and  far  out-­‐weigh  the  the  benefit  
 Black  Swans  
  High-­‐impact,  hard-­‐to-­‐predict,  and  rare  events  beyond  the  realm  
of  normal  expectations  

  The  term  Black  Swan  comes  from  the  17th  century  European  
assumption  that  'All  swans  are  white'.  In  that  context,  a  black  
swan  was  a  symbol  of  something  that  was  impossible  or  could  
not  exist.  In  the  18th  Century,  the  discovery  of  black  swans  in  
Western  Australia  metamorphosed  the  term  to  connote  that  a  
perceived  impossibility  may  actually  come  to  pass.  
 Black  Swans:  
Product  Master  Data  
  Data  Warehouse  requirements  delivered  
  Data  Warehouse  analysis  identifies  no  management  of  product  
master  data  and  no  product  master  list  
  Project  risk  raised  and  escalated  to  executive  level  
  BI  Project  postponed  and  staff  re-­‐tasked  to  define  and  deliver  
Corporate  Product  MDM  
  Implements  ERP  based  MDM  solution  and  reduces  product  
catalogue  by  90%  
  Directly  affects  corporate  bottom  line  
  Restarts  BI  Project  six  months  later  with  massively  improved  
data  quality  and  data  integration  
 Black  Swans:  
Financial  Reporting  
  Telco  in  DotCom  boom  consistently  overstates  earnings  and  
subscriber  growth  to  boost  share  price  

  Data  Warehouse  project  identifies  issue  and  produces  a  set  of  

correct  numbers  

  Business  refuses  to  adjust  statutory  reporting  as  they  have  been  
using  the  ‘Excel’  numbers,  it  would  be  embarrassing  to  restate  
the  figures  and  the  new  ones  are  just  too  different  
  Business  collapses  and  is  taken  over,  national  newspapers  
report  that  ‘internal  systems’  had  the  correct  figures  

  The  failure  to  use  real  data  was  highlighted  by  auditors  as  a  
significant  factor  in  the  poor  decisions  made  by  management  
 Black  Swans:  
The  Salesman  Fraud  
  Supplier  of  confectionary  to  a  major  supermarket  chain  
  First  Release  of  Data  Warehouse  
  CEO  was  un-­‐happy  with  the  numbers  from  the  new  system  
  He  knew  that  the  supermarket  was  their  biggest  customer  

  IT  defended  the  Data  Warehouse  

  Every  last  loading  step  and  transformation  had  been  tested  &  checked  

  A  Salesman’s  commission  doubled  for  the  first  quarter  on  new  

accounts  before  dropping  to  standard  rate  in  subsequent  quarters  

  The  Salesman  responsible  for  the  supermarket  chain  created  a  new  

customer  each  quarter    
  Corporate  revenues  affected  enough  for  an  exchange  filing  
 Black  Swans:  
External  Collapse  
  Sudden  market  collapse  

  Many  customers  suddenly  change  profile  

  Looking  for  value  –  lower  spend  

  Be  Pro-­‐active  
  An  opportunity  to  demonstrate  the  true  value  of  BI  
  (It  might  save  your  job  too!)  
 White  Elephants  
  A  white  elephant  is  a  valuable  possession  of  which  its  owner  
cannot  dispose  and  whose  cost  (particularly  cost  of  upkeep)  is  
out  of  proportion  to  its  usefulness.  
  The  term  derives  from  the  sacred  white  elephants  kept  by  
Southeast  Asian  rulers.  To  possess  a  white  elephant  was  (and  
still  is)    regarded  as  a  sign  that  the  monarch  was  ruling  with  
justice  and  power,  and  that  the  kingdom  was  blessed  with  
peace  and  prosperity.  The  animals  were  considered  sacred  and  
laws  protected  them  from  labour,  receiving  a  gift  of  a  white  
elephant  from  a  monarch  was  both  a  blessing  and  a  curse:  a  
blessing  because  the  animal  was  sacred  and  a  sign  of  the  
monarch's  favour,  and  a  curse  because  the  animal  had  to  be  
kept  and  could  not  be  put  to  practical  use  to  offset  the  cost  of  
maintaining  it.  
 White  Elephants  
Convention  over  Configuration  
  Bank  with  existing  data  warehouse  

  Extension  to  support  Balanced  Scorecard  

  Wanted  to  use  a  different  data  modelling  technique  in  the  same  data  
warehouse  for  the  new  elements    

  Justified  as  a  management  decision  because  “we  can’t  afford  to  re-­‐
develop  the    old  model  but  we  want  the  best  technique  for  the  new  parts”  

  Resultant  model  could  simply  not  be  used  

  ‘On  Rails’  
  Values  “DRY  -­‐  Don’t  repeat  yourself”  &  “Convention  over  configuration”  
  Use  each  component  in  a  consistent  way  
  Aids  Understanding  and  Reduces  Maintenance  
  Use  each  component  for  the  purpose  it  was  intended  
 White  Elephants:  
Not  evaluating  technology  
  Second  Generation  Data  Warehouse  Build  
  Existing  platform  major  RDBMS  
  Recommended  solution:  Componentised  architecture  with  high  
performance  database  engine  and  simple  ETL  architecture  
developed  by  small  in  house  team  
  Chosen  solution:  Existing  RDBMS  vendor  with  tightly  coupled  ETL  
tool  from  a  different  vendor  and  major  SI  doing  development  on  
site  –  “because  that’s  the  way  we  do  it  here  and  management  
won’t  consider  anything  else”  
  Current  issues:  
  Considered  too  costly  and  is  being  delivered  late  
  Already  having  to  review  technology  choices  because  of  
performance  issues  before  roll-­‐out  complete  
  SI  has  left  taking  all  the  knowledge  with  them  
 White  Elephants:  
Making  an  ETL  White  Elephant  
  Existing  data  warehouse  in  personal  finance  company  with  a  
ETL  load  built  from  SQL  scripts  and  a  shell  script  and  tight  
control  via  SVN  in  production  for  3  years  
  New  IT  Director  commissions  review  by  vendor  that  
recommends  the  vendors  ETL  tool  for  US$400K  
  Six  Months,  4    Consultants  @  US$1.5K  per  day    
(a  total  spend  >US$600K)    later  …  
  All  ETL  converted  
  Runs  20%  slower  
  Other  BI  developments  delayed  whilst  changes  made  

  Deemed  TOO  EXPENSIVE  to  revert  back    

  Eventually    reverted  back  12  months  later  when  vendor  quoted  for  
upgrade  to  system  
 White  Elephants:  
Breaking  an  ETL  White  Elephant  
  Interactive  Media  corporation  moves  from  traditional  RDBMS  to  
commodity  appliance  technology    
  Data  Volumes  doubling  every  six  months  

  Internal  Review  
  ETL  tools  can’t  handle  load  
  ETL  experts  too  expensive  for  long  term  engagements  
  SQL  scripts  can  be  developed  by  more  resources  
  SQL  scripts  allow  more  work  to  be  done  in  the  appliance  
  SQL  scripts  allow  more  agile  approach  
  SQL  scripts  allow  tighter  change  management  

  New  architecture  componentised,  commodity  based  with  simple  

script  engine  reduces  costs  by  90%  and  increased  productivity  by  
100%  
 White  Elephants:  
The  OPEX  trap  
  BI  Projects  command  headline  CAPEX  budget  figures  
  BI  Projects  are  widely  publicised  during  development  
  Most  of  the  money  is  spent  in  OPEX  
  It  comes  from  User  Support,  Training,  Changes  and  
Operational  Support  
  It  is  normally  2x-­‐3x  and  often  5x  more  than  the  cost  of  the  
build  over    the  lifetime  of  the  system  
  It  is  hidden  –  spread  over  multiple  budgets  in  such  a  way  that  
it  is  hard  to  evaluate  and  often  ignored  

  Any  action  (especially  tactical  actions)  in  the  build/test  

stage  that  will  result  in  increased  OPEX  is  deadly  to  ROI  
 White  Elephants  
The  New  Reporting  Tool  
  Data  Warehouse  Second  Generation  Build  
  New  project  –  new  interface  concept  
  Current  system  users  were  ‘un-­‐happy’  with  the  existing  tool  
  The  reality  was  largely  issues  with  the  data  model  and  data  
quality    of  the  current  system  
  Replaced  with  an  equivalent  reporting  tool  

  The  cost  
  Retraining  of  over  2000  staff  
  Failure  to  de-­‐commission  old  reporting  tool  
  50%  too  many  licences  bought  for  the  new  tool  
  User  dissatisfaction  with  the  (new)  reporting  tool  
 White  Elephants:  
Reporting  Tool  Penetration  
  Business  intelligence  vendors     Nigel  Pendse  of  BI  Survey  
like  to  talk  up  a  20/80  split:   shows  just  over  8  percent  of  
employees  are  actually  using  
only  20  percent  of  users  are   BI  tools.    
actually  consuming  BI  
technologies;  the  remaining     Even  in  industries  that  have  
80  percent  are   aggressively  adopted  BI  tools  
(e.g.,  wholesale,  banking,  and  
disenfranchised.     retail),  usage  barely  exceeds  
11  percent.  
  Reasons  
  Security  limitations     Hardware  cost    
  Data  availability  
  Slow  query  performance    
  Software  cost    
  Internal  politics  and     Software  was  too  hard  to  
(more  precisely)  internal   use  
power  struggles     User  scalability  
 White  Elephants:  
Not  testing  the  delivery  
  Any  delivery  has  stages  
  Requirements  /  Analysis  /  Design  /  Build  /  Test/  Deploy  etc.  

  Every  stage  can  have  tests  associated  with  it  

  Requirements:  Mind  Experiment  Method  
  Analysis:  Cross-­‐checking  with  other  sources  
  Design:  Algorithmic  Checks  
  Build:  Unit  Tests  /  Boundary  Checking  
  Testing:  System  Tests  /  Integration  Test  

  Test  early  and  often  

  Fixing  something  1  stage  later  =  2x  more  expensive  
  Fixing  something  2  stages  later  =  4x  more  expensive,  etc.  

  Many  more  small  tests  performed  early  will  save  huge  amounts  of  time  and  money  
  Have  to  handle  short  term  slippages  
  If  you  haven’t  got  time  to  test  then  you  are  planning  to  spend  OPEX  

  Massive  user  perception  impact  as  information  is  right  first  time  
 White  Elephants  
Garbage  In  ::  Garbage  Out  
‘On  two  occasions  I  have  been  asked  "Pray,  Mr.  Babbage,  if  you  put  into  the  machine  wrong  
figures,  will  the  right  answers  come  out?"  …  I  am  not  able  rightly  to  apprehend  the  kind  of  
confusion  of  ideas  that  could  provoke  such  a  question.’  –  Charles  Babbage  1791-­‐1871  

  Data  Quality  is  more  critical  than  ever  

  A  failure  to  address  Data  Quality  at  every  stage  will  always  
lead  to  additional  costs  

  Plan  for  your  BI  project  to  spin  off    dozens  of  data  quality  
projects  and  continue  to  do  so  throughout  its  life  

  Data  Quality  issues  drive  users  away  which  directly  

increases  cost  of  ownership  and  reduces  ROI    
 White  Elephants:  
De-­‐commission!  
  Data  Warehouse  Component  Delivered  
  De-­‐commission  the  previous  reporting  system  
  NO  -­‐  REALLY  -­‐    TURN  IT  OFF  !  

  If  it  is  left  on:  

  It  costs  money  to  run  (and  it  is  all  invisible  OPEX)  
  Users  will  compare  results  and  distrust  the  new  system  even  
if  it  can  be  proved  to  be  more  correct  than  the  old  system  
  Users  will  continue  to  use  the  old  one  because  it  is  familiar  
and  no-­‐one  likes  change  
  Users  that  do  not  migrate  but  have  been  trained  on  the  new  
system  will  have  to  be  re-­‐trained  when  they  start  to  use  it    
 The  Solution  
  The  proposition  suggests  that  doing  BI  has  not  delivered  
the  expected  ROI  for  most  organisations.    
  Does  this  mean  that    organisations  should  stop  developing  
BI  solutions?  
  Categorically  NO  –  BI  should  be  hugely  worthwhile  
  But    remember  you  can  only  succeed  if:  
  Your  organisation  fully  embraces  Business  Intelligence  
  You  expect  and  embrace  Black  Swans  
  You  avoid  and  mitigate  White  Elephants  
  You  make  your  own  luck  
 How  to  be  lucky  
  Lucky  people  frequently  happen  upon  chance  opportunities  

  There  are  positive  black  swans  in  every  organization,  it  is  just  a  matter  of  identifying  
them  as  they  occur  

  Lucky  people  listen  to  their  hunches  

  Members  of  your  team  will  have  insights  beyond  their  remit  based  on  their  
experience,  identify  these  people  and  exploit  their  insights  

  Lucky  people  persevere  in  the  face  of  failure  

  Every  project  will  face  set-­‐backs  -­‐  plan  for  them  

  Lucky  people  have  the  ability  to  turn  bad  luck  into  good  fortune  

  Every  project  will  face  set-­‐backs  -­‐  embrace  them  as  an  opportunity  and  change  your  
project/remit  and  reset  your  goals  

 Based  on  work  by  Prof.  Richard  Wiseman,  University  of  Hertfordshire,  2003  
 Improving  ROI  
  Black  swans  (handled  positively)  massively  increase    
Gain  from  investment  

  White  elephants  (eliminated)  significantly  reduce  the    

Cost  of  the  Investment  

  Whatever  the  CAPEX  investment  of  the  project  

the  OPEX  will  be  significantly  more  –  design  for  this  
 Black Swans &
White Elephants

Return On Investment
In Business Intelligence

THANK YOU