Big Data – The power to tell the future…and the past.

The 1992 film Sneakers features a scene where Sir Ben Kingsley and Robert Redford’s characters – both activist hackers whose lives have taken divergent paths – discuss the idea of power.

‘It’s not about who’s got the most bullets,’ Kingsley states, ‘It’s about who controls the information. What we see and hear, how we work, what we think… it’s all about the information!’

Over twenty years later those words are increasingly becoming a reality, but the real power isn’t merely the acquisition of information, instead it resides in being able to find the patterns hidden in its midst. Trend analysts have of course been doing this very thing for years, but due to  our ever expanding use of the internet for all aspects of modern life there is now so much data being created that it requires a new approach – this is the world of ‘Big Data’.

IBM recently stated that people create a staggering 2.5 quintillion bytes of data every day (that’s roughly equivalent to over half a billion HD movie downloads). This information is generated from a huge variety of sources including social media posts, digital pictures, videos, retail transactions, and even the GPS tracking functions of mobile phones. In fact IBM go on to say that ‘90% of the data created in the world today has been created in the last two years.’

Harnessing this vast array of information is no easy matter, requiring superfast computers that can process colossal amounts of data and produce results in time frames that would have been impossible only a few years ago. Even the way the data itself is stored and arranged is a far stretch from the kind of spreadsheets or databases we have used in the past. But the potential rewards for this substantial investment in cutting edge technology and software are so great that big data research is rapidly becoming the digital equivalent of the 19th century gold rush.


‘Big data marks the beginning of a major transformation,’ explain authors Kenneth Cukier and Viktor Mayer-Schonberger in their book Big Data. ‘Just as the telescope enabled us to comprehend the universe and the microscope allowed us to understand germs, the new techniques for collecting and analyzing huge bodies of data will help us make sense of our world in ways we are just starting to appreciate.’ Financial institutions now pay handsomely for analysis that allows them to predict the future movements of markets, while insurance companies use it to determine the potential risks of each policy holder. The medical profession is discovering how careful examination of large data sets can help them diagnose diseases such as breast cancer faster, and even law enforcement is making use of the predictive power that big data allows.

A recent episode of the Horizon programme on the BBC followed the work of Professor Jeff Brantingham, whose PredPol system was helping lower crime in districts of Los Angeles. The analytic software he and his team developed interrogated historical and current crime records, alongside wider data for the areas in question, looking for complex trends. It then produced a list of predictions on where and when crimes would be most likely to occur, which the police department used to strategically deploy officers in those areas. Now this might sound like something out of Minority Report, but at the end of the six month trial the police confirmed that burglaries were down by 27%, and the areas that used the software had seen a 13% overall decrease in crime as opposed to the 0.4% by those using traditional methods.

‘We all like to think that we’re in control of everything,’ said Brantingham, ‘but in fact all of our behaviour is very regular, very patterned in ways that is often frightening to us. Offenders are no different, they do exactly the same things over and over and over again, and their criminal offending patterns emerge right out of that regularity of their behaviour.’


PredPol has proven so successful that it is being rolled out across several other divisions of the LAPD, and the system has also reached these shores as Kent Police is now trialling the software. But the power of big data isn’t just in predicting the future, it is also a key element in what has become known as Big Science, an area that seeks to discover the very origins of life itself and how our bodies work today.

The Human Genome project, which set out to decode the genetic building blocks of life, took a team of dedicated scientists a decade of intensive work to complete. The same task now, using the new technologies behind big data, would take as little as a day. Astronomers are able to map the heavens with much higher powered telescopes than ever before and collate the terabytes of data that they produce in a fraction of the time. Particle physicists working on the Large Hadron Collider were able to record the behaviour of subatomic particles travelling at unbelievable speeds and process that data to potentially discover the Higgs Boson (or ‘God’) particle, which could change the future of science in a profound way.

All of these advances in data collection and manipulation are exciting and possibly life-changing. None more so than the recently announced ‘Brain Research through Advancing Innovative Nanotechnologies’ programme (or BRAIN for short), whose goal is to map the activity of the human brain. Using state of the art equipment the scientists will be attempting to monitor, record and analyse millions, even billions, of neurons simultaneously. The programme, which was revealed by President Obama in April, hopes to find cures to conditions such as Alzheimer’s, strokes, and autism, with additional research into other areas of mental illness.

There are those who argue that the nature of big data means discoveries can be made without the scientists truly understanding the origin or motive of the problem – so they’ll know how something happens, but not necessarily why. This argument opens up the philosophical nature of research, but some big data proponents respond in more pragmatic terms. ‘As humans we have been conditioned to look for causes,’ Kenneth Cukier and Viktor Mayer-Schonberger explain, ‘even though searching for causality is often difficult and may lead us down the wrong paths. In a big-data world, by contrast, we won’t have to be fixated on causality; instead we can discover patterns and correlations in the data that offer us novel and invaluable insights. The correlations may not tell us precisely why something is happening, but they alert us that it is happening. And in many situations that is good enough.’

This article originally appeared on the PC Advisor website and in the print magazine as part of a monthly section I write entitled News Viewpoint. To see that version please click here, or purchase a copy of the fine magazine from your local newsagent. 

The Fight to Keep the Internet Open

The recent London Olympic games had many poignant moments, but one in particular stood out for those with a keen interest in technology. During the opening ceremony Sir Tim Berners-Lee, inventor of the world wide web, tweeted a message from the mainstage which was then displayed in huge letters all around the stadium. It simply read ‘This is for Everyone’, and was a reminder of how important the modern internet has become to all of us. It’s a concern then that in the past few years a political storm has been brewing about the future of the web, or more precisely how we gain access to it.

The argument centres around the principle of ‘Net Neutrality’, and looks set to become one of the most important debates in technology for the foreseeable future. Tim Wu – Professor at Columbia Law School, author of the book ‘The Master Switch’, and the man who coined the phrase Net Neutrality – recently stated in an interview that ‘The internet was built on the principle that the carriers take your data where you want it to go, and that people are allowed to communicate over the internet without interference from the people in the middle. It’s a pretty profound principle’.

This is the essence of Net Neutrality. The fact that the companies that supply you with your gateway to the internet remain neutral to the content and sites that you visit, treating all as equal. It’s obvious, sensible, and exactly how the internet has essentially functioned up until now. But as we increase the amount of data we consume, due to services like Youtube and BBC iPlayer, some ISPs are beginning to talk about charging more to carry this content, or at least to ensure that the quality of its delivery remains high.

It can be easy to forget that the way we access the web is still controlled by only a few companies – those with the physical infrastructure that allows signals to pass between two points. This gives those companies a tremendous amount of power as our reliance on the internet increases. How would you feel if your provider decided that to access some of the more popular online sites you would be required to pay them more? You want Facebook as part of your package, then that’ll be an extra £5 a month. Youtube? £10. It might sound far fetched but there is growing concern that this behaviour could be on the horizon unless governments decide that Net Neutrality becomes enshrined in law.


It isn’t just a simple case of the amount of data we consume, that would seem reasonable, instead it’s more a case of where or from whom we receive it. Recently customers who bought an iPhone 5 on the AT&T network in the US were told that they wouldn’t be allowed to use the built in Facetime feature over their 3G service, even though it should come under the customers’ data plan. AT&T were basically telling their customers that they could only use the data which they were paying for in a manner that the company would decide. We’ve seen something similar here in the UK were several of the mobile carriers have disabled the Skype service on their networks, and also deny customers the option of tethering their laptops or tablets to their mobile phones to create internet hotspots. This selective attitude towards allowable services could theoretically be used to promote rival services with whom the carriers have favourable agreements.

Media reform advocacy groups such Free Press (, who started the Save The Internet campaign (, voice a chilling version of the future when they argue that the ISPs, and media companies that back them, ‘want to tax content providers to guarantee speedy delivery of their data. And they want to discriminate in favor of their own apps, services and content — while slowing down or blocking competitors’ services.’ This might sound like classic scaremongering, but it should be noted that back in 2005 AT&T caused outrage when it proposed charging certain web companies increased rates so they could receive preferential treatment for their web traffic. This would have effectively given the paying sites faster download speeds than their non-paying rivals, and made AT&T a tidy profit from this two-tiered approach. Then in 2007 Comcast, the largest cable TV and internet provider in the US, was found to be tampering with traffic across their network to certain file sharing sites in an attempt to make them unusable.

In fact a recent joint study between the Body of European Regulators for Electronic Communications (BEREC) and the European Commission found that at least 20% of mobile Internet users in Europe ‘have contracts that allow their Internet service provider (ISP) to restrict services like VOIP (e.g. Skype) or peer-to-peer file sharing.’ while the same was true for home broadband connections, especially in regards to peer-to-peer sites at peak times.

The main concern that advocates argue is one where the internet as we know it becomes more of a series of walled gardens where access is strictly controlled and regulated by a few large corporations each of whom offer their own services and media – almost a return to the days of AOL and Compuserve, with customers locked-in to their specific version of the web. ‘Big media companies want to be in charge again,’ states Tim Wu, ‘Most of the media for the last 150 years have been closed, that is if you look at NBC, or you look at the cable networks – they decide what goes on the network. So the threat comes from the fact that the broadcasters, phone companies, cable [companies], are used to that business model and want to go back’.


In 2012 the SOPA, PIPA, and ACTA bills were proposed in the US and Europe as a way of curtailing content on the web under the auspice of copyright protection. If passed they would have forced ISPs to regulate the content that went through their networks, which could have resulted in sites being delisted from search results, thus virtually disappearing from the internet, if the ISPs thought there was a possibility that they contained copyright material. The bills were defeated in their respective countries after widespread discontent from the online community, but the entertainment industry is thought to be preparing new ones, and their lobbying power means that they will certainly be heard.

The political level of involvement in the issue saw Holland become the first European country to make it law that no ISP can charge more for access to specific sites, while Chile has also created relevant legislation. The concern about Net Neutrality became so great that even President Obama spoke about it in a 2010 interview, during which he said ‘We’ve got to keep the internet open, we don’t want to create a bunch of gateways that prevent someone who doesn’t have a lot of money but has a good idea from being able to start their next Youtube, or their next Google on the internet’.

The battlelines are being drawn, and it looks like this is an issue that’s set to be hard fought over the next few years, and which could decide our online experience for decades to come.

A version of this post originally appeared as part of a new series of features called News Viewpoint that I write for the PC Advisor website and also appears in the February 2013 issue of the print magazine – yes, I know that’s in the future, but the way magazines work is a mysterious form of sorcery. To see the original click HERE or pop out to your local newsagent and purchase the rather splendid magazine itself.