Resources: Finding a home for all that data

Resources: Finding a home for all that data
By Stephen Pritchard

Published: November 26 2009 18:04 | Last updated: November 26 2009 18:04

When companies started to build the first enterprise data warehouse and knowledge management systems, in the late 1970s, there was little doubt that these were projects that demanded significant investment in both time and resources.

The early data warehouse systems certainly required mainframe resources, and running queries took days, if not weeks.

But advances in computing power, as well as improvements in programming, have done much to reduce the infrastructure demands of business intelligence (BI). It is now quite possible to run small-scale BI queries using little more than a data source, a laptop computer and a spreadsheet program.

Some businesses – especially smaller ones – do indeed manage their data analysis this way.

However, BI experts caution that this approach struggles to scale up to support the larger enterprise, and can raise real difficulties in areas such as data governance and lead to companies having multiple master data sets, or “multiple versions of the truth”.

“Many people start with something small in scope, and there is nothing wrong with that,” says Jeanne Harris, a BI specialist at Accenture’s Institute for High Performance Business.

“But if marketing, and finance, and sales have their own scorecards, based on their own data, it will be a Tower of Babel. Very few organisations have done a good job of creating a single view of their data.”

Nor is the hardware challenge one that chief information officers – or users of business data – can completely ignore.

Although processing power has increased in line with Moore’s Law and data storage has also fallen in price, the growth of business data is faster still. Volumes of data are reckoned to double every 12 to 18 months, twice as fast as just three years ago.

Some businesses are reacting by moving to grid-based supercomputers, or by offloading BI processing to private or public “clouds”. Others are deploying solid-state hard drives in their data warehouses, because of the superior data throughput they offer.

But such systems are expensive and large organisations, in particular, are beginning to struggle with the time it takes to load data into a warehouse or a BI system, especially if it comes from multiple sources.

“With data warehousing appliances [dedicated computers for data processing], the bottleneck is not the speed of the box or the quantity of storage but the time it takes to load the information, especially if you are dealing with demographic information,” says Bill Hewitt, president and chief executive of Kalido, a data management company.

“Even at data loading rates of 10 gigabytes an hour, there is one company that is looking at 39 weeks to load its data.”

This is leading some companies to consider alternative approaches to analytics, such as stream-based processing. It is also prompting businesses to look at BI tools, as well as broader-based technologies such as enterprise search, that can examine data in situ, rather than require them to be loaded into a warehouse and then processed.

Such technologies could also help businesses to overcome their reliance on data from operational systems, such as customer relationship management or enterprise resource planning. Such transactional data are almost always historic, and leads to BI acting as a “rear view mirror” for management, rather than as an accurate predictor of trends.

“Most organisations don’t use external data but rely on [data from] their operational systems to solve specific problems,” explains Earl Atkinson, a BI expert at PA Consulting Group. As a result, the data will only be as good – and as timely – as the information held in those underlying systems.

Before companies can build enterprise-wide knowledge management or BI systems, they also need to work on the quality of the data. Data can also be accurate but partial, or misleading, especially if they were originally gathered for a different purpose.

“A customer, for example, can exist in multiple IT systems,” points out Tony Young, CIO of Informatica, a data management technology vendor. “You need to have a common agreement on who the customer is, for example, if you want to look at their history.

“If I ask a financial person who the customer is, it is the person you bill. Marketing will say it’s the person who responds to a campaign. For sales it might be the person signing the cheque. These are all correct, but they are not common. You have to agree how you are going to treat that information.”

This, more than hardware assets, network capacity, or even the ability to write complex algorithms to analyse data, goes to the heart of the debate around the resources needed for advanced business intelligence.

Organisations need to decide, early on, which information they are going to use, and be honest about the completeness, or otherwise, of their data sets.

If they do not, the results can be disastrous.

“In the run up to the financial crisis, institutions knew that there were three categories of risk but they only had data for one. So that was the one they thought about,” says Accenture’s Ms Harris. “You need to understand all of the risk variables and how they relate to each other, and this needs different technologies and capabilities in modelling, and in experimental design.”

Organisations also need to consider whether conventional data sources, such as those produced by back-office IT applications, or by more specialist tools, such as a retail point-of-sale system or a supply chain management system, really give the full picture.

Increasingly, companies are looking for ways to mine the information held in “unstructured” data, such as e-mails, presentations and documents, or even video clips or recorded phone calls, to provide a basis for BI, and hence better decision making.

“As much as 80 per cent of the information in a company is unstructured, against just 20 per cent that is structured,” notes Bob Tennant, chief executive at Recommind, a company that specialises in using search technology for information risk management.

“Most business intelligence is focused on that 20 per cent of structured data, as it is pretty high value and easy to deal with. But there are a lot of useful, unstructured data that are not being taken advantage of.”

Tapping into that unstructured information might not be easy. But it is the best, and for some companies, probably the only way to make more use of existing resources, in order to make better business decisions.

..............................................................................................

Q&A: ING Lease UK

ING Lease UK is part of the ING Group – one of the largest financial companies in the world. In 2004, the company acquired three businesses from Abbey National Group.

With 300 employees and 100,000 customers, the company has to ensure its reporting and market perception is as accurate as it can be.

Dan Ilett, for Digital Business, questioned Chris Stamper, chief executive of ING Lease UK, about how it creates useful intelligence from its information.

Digital Business What did you do to improve internal reporting?

Chris Stamper We turned conventional wisdom on its head. We found a tool that allowed the business to assemble all information from disparate data sources into one platform. This allowed us to make decisions in real time.

We ignored the “start small and learn” approach and took the “start big and understand” approach by focusing on the most fundamental question we needed answering which was “where do we make our profit and why?”.

DB What has been your return?

CS As an example, analysis of secondary income opportunity has driven £600,000 ($997,091) of additional annual income.

DB How has using “internal” business intelligence helped?

CS First, it has given us the ability to make decisions based on fact rather than intuition or perception and has provided complete transparency when understanding profit and loss levers.

We have now moved to a “nowhere to hide from the facts” culture, the IT department has been removed from the critical path to information and everyone in the organisation has access to answers. This encourages collaboration and end-to-end thinking.

DB What lessons did you learn from this? What would you tell others to do?

CS That perception-based decision making is a characteristic of sales-led organisations. That culture can be very quickly moved with the right tools and environment.

We now have a strong focus on real data quality.

Copyright The Financial Times Limited 2009. Print a single copy of this article for personal use. Contact us if you wish to print more to distribute to others.