The pyramid of wisdom
The 'Big Data' notion of a huge ocean of Data with tiny little molecules of golden wisdom hidden inside, which people want to process in order to make better business judgements, is not a new one.
Indeed, for many years, data scientists have talked about the pyramid of wisdom that is comprised of four stages; data, information, knowledge and wisdom. Only with data as the building block, putting that data into context to extract information into knowledge comes wisdom. Wisdom comes from over time being able to use learned knowledge to make accurate business decisions and ultimately use the core data to drive strategic business opportunities.
The oil rig
However, the problem with Big Data comes with its size; the amount of data that needs to be put into context has increased exponentially over the past I 0 years, to the extent that it now poses a real issue.
Of course, there are tools out there which can help you dig and corral your data, such as Hadoop, Oracle and Mongo.
These are, in effect, the drilling platforms that get down and extract the raw material that is required to begin processing. But these big data handling tools only get the oil/data to the surface on a one-off basis.
Once you strike crude oil, therefore, the key is to refine...