I thought I would set the record straight about an ongoing myth in intelligence analysis. It goes something like this:

The flaw in current intelligence collection systems is that they gather too much data to possibly go over.

The myth stems from a misunderstanding about the nature of spying systems. Most people believe that the surveillance state exists to prevent immediate terrorist threats (or more specifically, threats to the state). This justification is pushed by the governments and security organisations that are involved in spying.

The PRISM program exists to collect communication over the Internet

If you think that it is an intelligence failure to collect all data, you assume that the data is intended to be processed today, and not, say, in 20 years. In 20 years, assuming Moores law continues at the current rate, computers will have 8192 times as much computer power. This is in addition to 20 years of algorithm development. Today's collected data will be trivial to process in 20 years.

Who cares about 20 year old information? The extended history of intelligence collection works in much the same way that a biography will tell the story of the formation of a great leader from the beginning. What their childhood was like. Their historical involvement with different movements over decades. Letters that they had received. Speeches that they had given. By using extended historical information, we can piece together the type of person they are. A current leader may give a nice speech today, but without historical context there is no way to tell what sort of a person they are, because that nice speech may have been written by a good speech writer. 

If a spy agency is looking at social media updates in the last hour, telling the difference between jokes about terrorism and real threats to the state is extremely hard. However, looking over many decades of communication and locations visited, a security agency can create fuller picture about the types of people they want to stop, whether they be real perpetrators of harm or a group that threatens the ongoing existence of the national security state.