What is Big Data ?
We’ve all heard the statistic. Something like ‘every two days now we create as much information as we did from the dawn of civilization up until 2003’ in Google Chairman Eric Schmidt’s version of it.
Now where does all of this data come from and why do we need it?
Social media is one obvious example – millions of users record and share their thoughts and opinions electronically and instantly, in a way which was once the province of a small number of writers and journalists using books and newspapers with, by comparison tiny circulations and long publication delays. Social media creates tremendous amounts of rich data that, for example, can be used for improving marketing strategaies. The simple fact that you are reading this article contributes to the creation of data.
And the ability to capture data electronically extends into many other fields. As an example, cars 20 years ago gave their drivers a handful of parameters such as engine oil pressure, rpm, mileage, fuel in the tank at a given point in time. Today these cars have a battery of sensors constantly sampling and recording the vehicle’s performance, creating large amounts of data that can be used for improving the quality of the car or for instance promote a new model that corresponds the closest to the customers driving behaviour.
But much the largest element of the data explosion isn’t in the form of lists, files and structured statistics. It’s unstructured, consisting of articles, messages, social media postings, e-books, images, audio, video or web pages. ‘Big’ data is all about this unstructured data, which if it existed before at all, was mostly stored on paper and accessible only through the human eye. Now it’s accumulating electronically at extraordinary speed on private servers and public networks. The question is how to make it useful by bringing it into your analytical environment.
Why does it matter?
Because it reflects human, and specifically buyer, behaviour. Buyers and potential buyers explore and communicate, and as they do so, they leave their footprints behind in the form of email messages, comments on social media, web-site interactions, information downloads, edits on documents, etc., etc.
When paper (even photographs or film!) was where this unstructured data resided, very little of it was recorded at all, and any form of analysis involved many hours of human reading or viewing time. Furthermore, apart from the odd stamp on a library book or the purchase of an educational video, buyers and other investigators left no trace behind.
All that has changed in the last ten years. Now data is captured and stored electronically, there is an explosion in its volume, anyone using it leaves a trail and, most importantly of all, all the information available can be analysed efficiently by machine rather than man.
How do we get to it?
It’s already there – in your servers, your e-mail and telephone records, your tablets and mobile phone, possibly in ‘cloud’ services like DropBox or Salesforce.com.
What you need is an efficient tool to bring it together and to link it to your structured data such as your customer, orders, invoicing and prospect files.
What’s that ‘efficient tool’?
Oddly enough, there is a parallel here with the early days of web-sites. Once exotic, they’re now ubiquitous and everyone knows they have to have an effective one. Yet by far the most influential tool for running web-sites was Apache, an ‘open source’ collaborative software, which still today accounts for around 60% of the world’s web sites. Why? – because it works properly and because the basic software was and is free-of-charge.
Now, with the coming of Big Data (itself partially driven by all those websites), the same ‘Apache Foundation’ of collaborative software engineers has developed a set of programs known as ‘Hadoop’ to store both structured and unstructured data. And Hadoop too is rapidly moving from being an exotic newcomer to a standard database tool. It works, it’s under continuous development, the tools (although not all the software needed to implement and support them) are free-to-use and Hadoop runs on standard low-cost hardware, either your own or ‘in the cloud’. In many organisations, we see the Hadoop solution being implemented in parallel with standard relational database systems.
Furthermore, programmers and analysts can use standard business intelligence software like Business Objects, SAS and Tableau to analyse the data in a Hadoop system, as well as sophisticated statistical techniques, textual analysis, data mining and other advanced tools. Hadoop brings the data together, business intelligence tools do the rest.
I have enough problems trying to get my marketing, sales and prospect data linked correctly to my customer and revenue information without worrying about all this Big Data as well?
Hadoop is designed to store both structured and unstructured data – so combining different types of structured information (customer and revenue, say, with CRM systems like salesforce.com or Microsoft Dynamics) is pretty easy for it. By using Hadoop tools, you can not only link unstructured ‘big’ information to your data but also bring together structured systems such as CRM, customer orders and invoicing and business intelligence so that you have a real and comprehensive sales and marketing database.
What does it all buy me?
In B2B, it’s all about efficiently identifying sales opportunities, so maximising the RoI on your marketing budget by, isn’t it?
Somewhere, hidden in all that structured and especially the unstructured ‘big’ data is the intelligence which shows when somebody, somewhere, is in the market, and if you can get to him or her fast and accurately, you’ll (a) stand the best chance of getting the order (b) invest least marketing € in finding the opportunity.
That’s what ‘big data’ and Hadoop-based processing can do for you. By bringing together your structured information from CRM and accounting systems with unstructured data which captures the behaviour of potential buyers, you will be able to identify far more accurately those who are nearest to buying. If you can focus your lead generation and nurturing programs through conventional methods like e-mailing, postal DM and telemarketing on far fewer but much more ready-to-buy targets, your cost per qualified lead will drop substantially and your RoI will soar.
If you’re interested in a radical improvement in your marketing RoI, let’s talk.