Big Data has great promise – no doubt that we are on the brink of discovering innovative ways to make a lot of money due to the ability to consume and analyze a greater volume, velocity, and variety of data than ever before.
The question, however, is who will make these innovative discoveries?
Can the innovation be made by a kid in a garage, a college kid in his dorm room, or a talented IT professional dreaming of a start-up in his free time?
On the one hand, the answer is NO – the college kid in the dorm will NOT be changing the world via big data the way Mark Zuckerberg did with social networking. Why? The reason quite simply is because “Big Data” requires DATA and a lot of it and in potentially many different forms.
Data is not free
In fact, there are billion dollar businesses that are essentially data companies – they sell raw data (and analytics on it). I’ve even worked at one in my time. OK – Some of that raw data is free but much of the data is not. The sheer volume of data is prohibitively expensive to acquire for a college kid to start to experiment with. Additionally, some big data use cases revolve around social data that sites like Facebook can use to make money (data is why Facebook has such a high valuation). Did you think Facebook gives away their data for free? If the college kid can’t even get his hands on the data then the game is over before it even starts.
On the bright side, however, cloud computing infrastructure as a service and pay-as-you-go pricing provides affordable infrastructure for the college kid. Powerful infrastructure is also a pre-requisite for Big Data – but it’s not a barrier for the college kid.
It goes back – again – to access to data.
The majority of the big data discoveries will come from big established companies with the resources to acquire it.
Agree or disagree? Actually, proving me wrong would make me happy.
Eric Weinstein, MBA 
I don’t think that the actual problem is in the complexity of processing/searching anymore, to a large degree, this has been mediated by tools like graph databases, Hadoop, Solr-Lucene, and HPCC.
I believe that there is still plenty of ‘garage’ style entrepreneurial opportunity for developing specialty tools and analytics. This is evident by the emergence of a thriving BigData processing ecosystem that provides new solutions in areas like in machine learning (Mahout), multi-structured data analytics (Hadapt), etc.
Yaacov – That’s a very good point regarding speciality tools and big data. If able to pick up a slight bit of traction, those tools are good candidates for acquisition. Taken this into consideration then, while “garage style” kids may not have access to all the data like a big company will, developers still have opportunity for (specialty) tools. A glimmer of hope for the dorm room kid!