Burgeoning data volumes continue to place organisations' IT departments under greater strain than ever as they look to harness data effectively while overcoming a minefield of performance issues. The importance of the data warehouse in attaining the maximum benefit from data has grown exponentially, with its promise to provide an enterprise-wide view of business activities and increase the company's profitability through intelligent data handling. Increased sales, more effective and intelligent marketing, enhanced customer services, and streamlined business processes--the data warehouse is regarded in high esteem by organisations as being capable of paving the way towards the attainment of these business benefits.
Ever since the building of data warehouses began and the term was first coined by industry heavyweight Bill Inmon, the perceived wisdom has been to maintain a separation between the data warehouse and the company's operational systems. Today, while this 'separatist' thinking still dominates many data warehousing professionals, the approach is being questioned. A school of thought is emerging that challenges the division of the data warehouse from operational systems, citing the fact that the justification for separation is purely technical--as opposed to being a divide founded on the drivers of the business. The evolving wisdom, particularly acknowledging the continuous march of technological progress, is to consider alternatives to the implementation of a traditional warehouse solution, where operational systems are capable of living in harmony with the data warehouse.
Why accept a compromise?
Businesses want data warehouses to provide a complete and immediate understanding of the enterprise, offering the capability to react quickly to the marketing place and to out-manoeuvre their competitors. Business drivers include, for example, increased revenues through more effective marketing and cross selling to the existing customer base, again based on better understanding of customer activity and profiles. Similarly, by identifying inefficiencies and areas of strength, cost reductions can be achieved, while revenues can be increased. The benefits, in short, more than pay for the implementation of the data warehouse.
There are also the technical considerations that translate to business benefits for the organisation. Today, technology is such that separate systems--with the concomitant investment and ongoing operational costs--could be unnecessary. The same is true of compromise approaches, or "halfway houses", as discussed later, where an intermediate database is used to attempt to furnish intelligent data rapidly. Not only is this approach very much a poor cousin to the data warehouse, yielding inferior data results, but it also exposes the business to greater costs in terms of setting the solution up, maintaining it and then replacing it in the future.
Disharmony and differing demands
The need to cater for disparate demands is why the data warehouse has traditionally been implemented separately from operational systems: they each have different profiles and make different demands on hardware and applications. Technically, therefore, the IT department has faced a range of conflicts between performance, and the operational and user requirements of the respective systems.
For example, because an operational system is usually built for a transaction processing workload, it needs to cater for multiple concurrent short-lived transactions, mixing queries with updates. The data warehouse, in contrast...