Suspicions about “big data” have deepened since Edward Snowden, the former subcontractor for the US National Security Agency, leaked secret information about the government’s mass surveillance of telephone and email communications in America.

What exactly is being collected and who is collecting it? Nobody really knows.

From Facebook posts and tweets to credit card transactions and subway trips, almost everything you do creates data points that are being collected and potentially used by governments and organizations for their own use.

Big data is a term for data sets that are so large they can’t be handled by traditional database management systems or software. Often the people who are contributing this data have no idea what is happening with it.

It sounds creepy, and it is. But Steven Gray, a teaching fellow in big data analytics at the Centre for Advanced Spatial Analysis at University College London, says big data collection is here to stay – and it’s only going to get bigger.

“As we invite more technology into our homes and become more dependent on having this technology in our day-to-day lives, we are going to generate more data that can be mined and used in some sort of way,” says Gray.

The term “big data” only became part of the vernacular in the past five years, due to the smartphone-inspired explosion in user-generated information. PricewaterhouseCoopers (PwC) estimates 2.5 quintillion bytes of data are created every day, with 90 percent of the world’s data created in the past two years alone.

But the concept of big data, and the challenge of processing it, isn’t new. It has existed since mainframe computers came into use in the 1950s. The difference now is that technology is more powerful and can mine more data than ever before.

Governments and organizations have an ever-growing big data analytics toolkit of automated computer programs, special software and algorithms at their disposal. They can use these tools to scan, extract and analyse huge amounts of public data from the Internet, their own internal databases and, with the permission of operators, from telephone networks. The market value of that data has already topped 10 billion US dollars (8.6 billion euros), and PwC expects it to surpass 53 billion dollars (42.5 billion euros) by 2017.

For companies, big data has become as important as experience and intuition in the decision-making process, says John Studley, data analytics leader at PwC in Australia.

“The new software tools enable you to reach into your existing data and extract the elements that you need and then slice and dice them in different ways or combine them to do predictive analysis,” he says. “These software tools are relatively inexpensive compared with the money that you can generate. Around a third of the companies that we have surveyed are now using big data far more extensively in their decision making, and that is leading directly to three times the benefits they had before using the data.”

Data sets, such as sales transactions and customer details, offer companies a rich insight into their customers’ buying habits, which can be used to personalize advertising and promotions.

A good example is US-based Amazon, which uses a recommendation engine to make suggestions to customers based partly on their previous purchases.

But despite the increased use of big data, many chief executives still rely on gut feeling – and they are paying the price. PwC estimates Australia missed out on 48 billion Australian dollars (33.5 billion euros) in economic benefits in 2013 because companies failed to fully leverage big data.

The applications for big data go far beyond the corporate world. Big data is also being used by law enforcement agencies to spot possible crime before it happens, optimize public transport systems and even identify contagious disease outbreaks.

But do individuals get anything in return for their data?

The answer is yes, says Gray. He points to the Google Now virtual personal assistant as an example of how individuals directly benefit from giving up a bit of their privacy.

Among other things, Google Now can scan your calendar and location and then combine that data with the latest traffic information to recommend what time you should leave for a meeting.

“Users don’t mind giving up some personal data if they are getting some value in return,” Gray explains.

But since Snowden lifted the lid on the US government’s surveillance programme, concerns have grown about how much data is mined and how it is being used.

A survey by the Global Research Business Network found almost a third of UK and US citizens do not trust their governments with their data. At least 40 percent say they are “very concerned” about how it is used.

Gray acknowledges that data collection and, in particular, the combination of different data sets can be misused, but he maintains that many of the concerns are overblown.

“You have an image in your head that there’s a single person sitting behind a computer somewhere with all this data who can look at your private life,” Gray says. “Actually what’s happening is an automated computer system is fusing the data together to create some value for the user. What’s important is that companies are transparent with what they’re doing with your personal data.”

Emma Carr, director of UK-based Big Brother Watch, agrees, but says regulating big data collection is “incredibly difficult”.

In the absence of tighter regulation, consumers will have to take more responsibility for protecting their own privacy.

“I think people will have to wake up to the fact that data is one of the most valuable commodities in the 21st century, and we are handing that over readily,” says Carr. “People are going to have to become a lot more savvy about how that works and how to protect themselves and their information.”