How Big is Big Data?
Unfortunately ‘Big Data’ has now become a trendy term from a marketing perspective and everyone has jumped on the band wagon twisting the ‘Big Data’ message to whatever suits their cause. Ultimately we have had ‘lots of data’ for many years. Consider the amount of information that government, banks, insurance companies, pharma companies, healthcare sector etc. has collected in the last 40 years?
Transactional Data value
There is no doubt that the rate of creation of data has multiplied exponentially but one has to contrast this with the value of the data being created. Much of the volume of data being created is being created on social media and, if data is the new oil, is comparable to very crude oil as there are trends and indicators in there but arguable little factual data. When you consider transactional data that has been collected for many years, this is the premium refined oil because the vast majority of events in a transactional set of data reflects a real event that happened.
To a degree, we can state that the massive volumes of data being created is low value, however, because of the volume of that data, there is ultimately a high value once it has been processed. The low volume data that is of high quality can also provide massively valuable insights, however, it is highly secured, difficult to get to and has many data governance issues around it. It is clear though that if both sets of data can be combined, the sum of the insights they can bring is exponentially larger than the sum of the two parts.
It is also clear that more and more transactional activity is happening in an open way on the Internet so information is being gathered daily that does not have the access, security and data governance issues associated with it that the existing transactional data does. Governments have masses of data they could mine for more altruistic purposes.
This means that competitive institutions such as banks and insurance companies have three to five years to leverage the value in their existing legacy data sets. It remains to be seen if these institutions have woken up yet and smelt the coffee!