WASHINGTON - The National Security Agency is collecting less than 30 percent of all Americans’ call records because of an inability to keep pace with the explosion in cellphone use, according to current and former U.S. officials.
The disclosure contradicts popular perceptions that the government is sweeping up virtually all domestic phone data. It is also likely to raise questions about the effectiveness of a program that is premised on its breadth and depth, on collecting as close to a complete universe of data as possible to make sure clues aren’t missed in counterterrorism investigations.
In 2006, the officials said, the agency was collecting nearly all records about Americans’ phone calls from a number of U.S. companies under a then-classified program, but as of last summer that share had plummeted to less than 30 percent.
The government is taking steps to restore the collection - which does not include the content of conversations - closer to previous levels. The National Security Agency is preparing to seek court orders to compel wireless companies that currently do not hand over records to the government to do so, said the current and former officials, who spoke on condition of anonymity to discuss internal deliberations.
That effort comes in the wake of President Barack Obama’s decision last month to find a way to move the data out of the government’s hands to ease concerns about intrusions on privacy. Obama has given the Justice Department and the intelligence community until March 28 to come up with a plan.
The actual percentage of records gathered is somewhere between 20 percent and 30 percent, and reflects Americans’ increasing shift from landlines to cellphones, as well as technical challenges in preparing the National Security Agency database to handle large amounts of new records, the current and former officials said.
That still likely represents tens of billions of records going back five years, a cause of concern to privacy and civil-liberties advocates.
One official said intelligence agencies have quietly chafed at assumptions that the National Security Agency was collecting all phone records. But they have been reluctant to correct the record because they did not want to draw attention to the gap.
In defending the program, administration officials have emphasized the need to gather all records.
“If you’re looking for the needle in the haystack, you have to have the entire haystack to look through,” Deputy Attorney General James Cole told Congress last July.
Edward Felten, a Princeton University computer scientist who has studied the program from a technological perspective, said the revelation “calls into question whether the rationale offered for the program is consistent with the way the program has been operating.”
Supporters, however, say the gap might undermine the argument that the program is ineffective because it has thwarted no attacks. “We should have a debate about how effective would it be if it were fully implemented,” one official said.
But collection of even a quarter of the records is valuable, officials said.
“It’s better than zero,” National Security Agency Deputy Director Rick Ledgett said Thursday, without describing the program’s exact scope. “If it’s zero, there’s no chance.”
One former senior official acknowledged that 100 percent was the goal, but asserted that as long as the collection “is fairly spread across the different vendors in the geographic area that you’re covering,” the collection provides value.
The National Security Agency, for instance, is still able to obtain the call records of some customers whose phone companies are not covered by the program. When the customers of a noncovered carrier call customers of a covered carrier, the latter’s records should reflect both ends of the call.
Some industry officials said the 20 percent to 30 percent figure can only be explained if the agency also is missing records from companies that provide Internet-based calls.
According to industry and government figures, the number of landlines in use fell from 127 million in 2009 to 96 million in 2012, a 24 percent drop. By contrast, the number of cellphones in use in the United States jumped from 255 million in 2007 to 326 million in 2012, a 28 percent rise. And Internet-based subscribers, according to the Federal Communications Commission, shot up from 26 million in 2009 to 42 million in 2012, a 62 percent rise.
The collection program began without court or congressional approval after the Sept. 11, 2001, terrorist attacks but was placed under court supervision in 2006 when American phone companies balked at providing the data solely at the request of the executive branch.
Under the program, the National Security Agency receives daily transfers of call “metadata” from several of the nation’s largest phone companies. Those records include the numbers called, the calls’ time and duration, but not the content of conversations, subscriber names or cell tower location data.
The bulk collection began largely as a landline program, focusing on carriers such as AT&T and Verizon Business Network Services. At least two large wireless companies are not covered - Verizon Wireless and T-Mobile U.S., which was first reported by the Wall Street Journal.
Industry officials have speculated that partial foreign ownership has made the agency reluctant to issue orders to those carriers. But U.S. officials said that was not a reason.
“They’re doing business in the United States, they’re required to comply with U.S. law,” said one senior U.S. official. “A court order is a court order.”
Rather, the official said, the drop in collection stems from several factors.
Apart from the decline in landline use, the agency has struggled to prepare its database to handle vast amounts of cellphone data, officials said.
“It’s not simply the ability to go to the court and order some vendor to give you more records, but you have to make sure that the [agency’s collection system] is prepared and ready to take the data and meet all the requirements of the court,” the former official said. “You don’t want to turn it on and get hundreds of millions of records, only to find out that you’ve got the moral equivalent of raw sewage spilling into the Chesapeake Bay.”
The process of preparing the system can take months, said the senior U.S. official, adding that mobile calls have data elements that are different from landline calls. “That’s a really detailed set of activities where we get sample data in, and we march it through our systems,” the official said. “We do that again and again and again. We put in auditing procedures to make sure it works. So before we turn on that mobility data, we make sure it works. … It’s very complex.”
Compounding the challenge, the agency in 2009 struggled with compliance issues, including what a surveillance court found were “daily violations of the minimization procedures set forth in [court] orders” designed to protect Americans’ call records that “could not otherwise have been legally captured in bulk.”
As a result, the National Security Agency’s director, Gen. Keith Alexander, ordered an “end-to-end” review of the program, during which additional compliance incidents were discovered and reported to the court. The process of uncovering problems and fixing them took months, and the same people working to address the compliance problems were the ones who would have to prepare the database to handle more records.
The agency fell behind, the former official said.
Last June, the program was revealed through a leak of a court order to Verizon by former National Security Agency contractor Edward Snowden, setting off a national debate over the wisdom and effectiveness of bulk collection.
The same agency personnel were tasked to answer inquiries from congressional overseers and others about how the program and its controls worked. “At a time when you’re behind, it’s hard to catch up,” the former official said.
Storage and implementing new features to comply with court requirements also costmoney, and that has been difficult in an era of budget cutbacks, the former official said.
The agency did not go to the court to seek new orders because it was not prepared, officials said.
“Until you are sure that you have an architecture that has the right features and the right capacity, you wouldn’t go to the time and trouble of getting the court to authorize the collection and retention of the data,” the former official said. “Because the court would want to know that you’ve followed through on that and you had a material intent to get it and use it.”
In a speech last month, Obama announced that he intended to find a way to get the government out of the business of holding onto the bulk records, but he also said its capabilities should be preserved.
Obama also announced that he wanted to immediately impose new limits on how the database is used, by requiring the National Security Agency to wait for a judge on the Foreign Intelligence Surveillance Court to sign off before querying records associated with a number that is suspected of links to terrorism - except in emergencies - and by limiting analysts to only pulling up records of people who are up to two levels removed from that number.
Previously, the surveillance court had allowed the agency to decide that a search was justified and had let analysts go up to three levels out - meaning an exponentially larger number of people’s calls would be scrutinized.
On Wednesday, according to a statement issued Thursday in the name of James Clapper, the director of national intelligence, the surveillance court issued an order amending the rules in line with Obama’s proposed changes.
Meanwhile, a judicial clerk announced Friday that Chief Justice John Roberts has made his first selection to the Foreign Intelligence Surveillance Court since Snowden’s revelations about the spy programs that had been secretly approved by the court.
The leaks have focused greater attention on how Roberts has used his unilateral authority to select judges to serve seven-year terms on the court. Of the 11 judges serving - all selected by Roberts - 10 had been appointed by Republican presidents.
But in May, when the term expires for Judge Reggie Walton of the District of Columbia, Roberts has selected an Obama appointee, Judge James Boasberg, also of the District of Columbia, to fill the position until 2021.
Several members of Congress have proposed changing the way judges are selected to serve on the court to achieve greater ideological diversity in light of its evolving role and growing power, and Obama’s surveillance review group also recommended doing so.
Information for this article was contributed by Ellen Nakashima of The Washington Post and by Charlie Savage of The New York Times.
Front Section, Pages 7 on 02/08/2014