U.S. Outlines N.S.A.’s Culling of Data for All Domestic Calls

Together, the new round of disclosures shed even more light on the scope of the United States government’s secret surveillance programs, which have been dragged into public view and debate by leaks from the former N.S.A. contractor Edward J. Snowden.

The Office of the Director of National Intelligence released the newly declassified documents related to the domestic phone logging program at the start of a Senate Judiciary Committee hearing on the topic. Simultaneously, The Guardian published a still-classified 32-page presentation leaked by Mr. Snowden that describes the N.S.A.'s XKeyscore program, which mines Internet browsing information that the agency is apparently vacuuming up at 150 network sites around the world.

The documents released by the government, meanwhile, include an April ruling by the Foreign Intelligence Surveillance Court that supported a secondary order — also leaked by Mr. Snowden — requiring a Verizon subsidiary to turn over all of its customers’ phone logs for a three-month period.

It said the government may access the logs only when an executive branch official determines that there are “facts giving rise to a reasonable, articulable suspicion” that the number searched is associated with terrorism.

The releases also included two formerly classified briefing papers to Congress from 2009 and 2011, when the provision of the Patriot Act that the court relied on to issue that order was up for reauthorization. The papers outlined the bulk collection of “metadata” logging all domestic phone calls and e-mails of Americans and are portrayed as an “early warning system” that allowed the government to quickly see who was linked to a terrorism suspect.

“Both of these programs operate on a very large scale,” the 2011 briefing paper said, followed by something that is redacted, and then: “However, as described below, only a tiny fraction of such records are ever viewed by N.S.A. intelligence analysts.”

Both programs traced back to the surveillance efforts the Bush administration secretly started after the terrorist attacks of Sept. 11, 2001, and which initially operated outside statutory authority or court oversight. The Bush administration later obtained orders from the Foreign Intelligence Surveillance Court to continue them.

The Obama administration has said it shut down the program that collected e-mail “metadata” in 2011, but it is not clear whether such collection has continued under a different program.

The newly disclosed XKeyscore presentation focuses in particular on Internet activities, including chats and Web site browsing activities, as intelligence analysts search for terrorist cells by looking at “anomalous events” like who is using encryption in Iran or “searching the web for suspicious stuff.”

In contrast to the domestic-call tracking program, the example cited in the XKeyscore presentation — which said it had generated intelligence that resulted in the capture of more than 300 terrorists — appeared to be focused on overseas activity.

A map showed 150 network sites around the world at which the N.S.A. is collecting that information; it is not clear whether the governments in those places are aware of the spying.

The volume of data is so vast that most of it is stored for only three days, the presentation said, although “metadata” — information showing log-ins and server activity, but not content — is stored for a month.

Several of the pages on the presentation were redacted by The Guardian.

But the presentation shows that while much of the focus from Mr. Snowden’s revelations so far has been on communications — whether calls or e-mails — that are linked, directly or indirectly, to a known suspect, the N.S.A. is also collecting and searching through massive amounts of Web-browsing activity.

“A large amount of time spent on the Web is performing actions that are anonymous,” the presentation explains, saying that the XKeyscore system can extract and store retrospective activity from “raw unselected bulk traffic.”

One example of how analysts might use the system is to search for whenever someone has started up a “virtual private network” in a particular country of interest; VPNs are pipelines that add greater security to online communications. N.S.A. analysts are able to use the system to extract the activity retrospectively from “raw unselected bulk traffic” and then decrypt it to “discover the users.” 

Ballmer outlines the road ahead for Microsoft Office, Skype, 'Bing Now'

According to Microsoft's Steve Ballmer, the future of Microsoft involves such things as "living" documents, the Microsoft equivalent of "Google Now," a blurring of email and chat, and the ability to add a gaming layer to everyday activities.

It's an ambitious vision, and Microsoft chief executive Steve Ballmer laid it all out in a strategy document that accompanied his "One Microsoft" memo outlining an ambitious company reorganization. While most of the early attention paid to Microsoft's strategy correctly focused on what the reorganization means for the short term, it's worth focusing on what the company has in store for the longer-term future of its product groups, too.

Ballmer made much of the fact that Microsoft is reorganizing around devices and services and moving away from being a purely software-driven business. But Microsoft was founded on identifying and meeting key needs that it can address, such as productivity, collaboration, and fun. What Ballmer's document appears to do is essentially remix those concepts, combining them in much the same way painters mix primary colors together, to develop new and profitable combinations of technologies.

Ballmer addresses five key areas: the future of documents, anticipatory data, the future of social, gaming, and the "shell" of the Windows interface.

"Documents are going from being printed to being experienced," Ballmer writes. "There are many high-value needs for personal creative expression — some just for fun, others at work or at school. We will reinvent the tools and form of expressing oneself (and expressing things as a group) from paper and slides to online. We will ensure that the tools handle multimedia (photos, videos, text, charts and slides) in an integrated way and natively online."

That's exactly the conclusion we reached on Wednesday, a day before Ballmer released his reorgnization memo. By injecting data into documents via Bing and the cloud, users are encouraged both to use Microsoft Office and to let documents "live," connected to the cloud. These "living documents" can't be easily reproduced by reader technology from Google and others, making Office that much more valuable.

"Our machine learning infrastructure will understand people’s needs and what is available in the world, and will provide information and assistance," Ballmer writes. "We will be great at anticipating needs in people’s daily routines and providing insight and assistance when they need it. When it comes to life’s most important tasks and events, we will pay extra attention. The research done, the data collected and analyzed, the meetings and discussions had, and the money spent are all amplified for people during life’s big moments. We will provide the tools people need to capture their own data and organize and analyze it in conjunction with the massive amount of data available over the Web."

If this doesn't sound eerily similar to what Google already does with Google Now, then you haven't been paying attention. Google Now works by sniffing your email, calendar, and location, among other things, and reminds you of what time your flight leaves and how long it will take to get there. Frankly, Google Now is one of those "Wow!" technologies that adds real value to Android -- and Google is busy adding them to Chrome, too. Microsoft offers very little that's comparable, and it seems intent upon changing that.

Using Bing as an "intelligence engine," Microsoft could begin collating information from Mail/Outlook/Outlook.com, the calendaring applications of both, location from Windows Phone, and traffic. Google's foundation is its massive wealth of data, and its ability to understand how that data interrelates to each other and to its users. But Microsoft needs to offer at least an outwardly competitive solution. It appears that the company intends to do so.

"Social communications are time-intensive, high-value scenarios that are ripe for digital re-imagination," Ballmer writes. "Such innovation will include new ways to participate in work meetings, PTA and nonprofit activities, family and social gatherings, and more. We can reimagine email and other communication vehicles as the lines between these vehicles grow fuzzy, and the amount of people’s digital or digitally assisted interaction continues to grow. We can create new ways to interact through hardware, software and new services. Next-gen documents and expression are an important part of online social communications. We will not focus on becoming another social network for people to participate in casually, though some may use these products and services that way."

In the beginning, there was email, and Outlook. Over the years, Microsoft has added calendaring (including the ability to share calendars); SharePoint; its enterprise videoconferencing and instant messaging client, Lync; Yammer; and Skype. All of these are different approaches to the same problem: how to easily communicate, collaborate, and share content both between consumers and business colleagues.

What Ballmer appears to be saying is that we should expect these products to coalesce into one another, although what the final form will be isn't clear. Microsoft already began integrating Lync and Skype in June, has previewed Skype for Outlook.com, and announced plans to integrate Yammer and SharePoint functionality, as well as Yammer and email. A planned Lync Room System will provide a simplified videoconferencing system, Microsoft executives have said, while Skype will be part of the Xbox One game console.

For now all of these products should remain separate applications, but look for these to blend together over time. Within the enterprise, the focus appears to be on "collaboration"; within the consumer space, Ballmer implies that "connection" appears to be the emphasis. But both include elements of the other.

"The experience we will deliver across all our devices centers on the idea of better connecting people with the things they care about most," Ballmer writes. "This includes their files, documents, photos, videos, notes, websites, snippets, digital history, schedules, tasks, and mail and other messages, combined with real-time information from our devices and services. It is more than what we think of as the shell today, and no current label really fits where we are headed. Neither the desktop nor the social graph describes this new experience, and neither does the search box, the pin board or the file system."

Ballmer's statement that "we will not focus upon another social network for people to participate in casually" also implies that its social network, So.cl, will remain a technology play, rather than a distinct product.

Microsoft raised eyebrows when its first introduction of the Xbox One console focused less on its capabilities as a game console and more as a TV set-top box. But Microsoft's announcement that it would combine live streams of NFL football games with fantasy-football updates in real time is an example of Microsoft's plans for gaming, which include using the Xbox One to provide what it calls "interactive participation".

"Although we will enable these activities effectively, our biggest opportunity is in creating the fun people feel most intensely, such as playing a game that lasts hours and takes real concentration, or immersing them in live events and entertainment (including sports, concerts, education and fitness) while allowing interactive participation," Ballmer writes. "Interactivity takes engagement and makes things serious; it really requires differentiated hardware, apps and services. People want to participate at home and on the go, and in gatherings with others."

Ballmer also writes that Microsoft also sees "opportunity in fitness and health," which probably implies, at the least, a pedometer app for Windows Phone as well as more opportunities for Microsoft's Kinect peripheral. One of the opportunities for Kinect for Windows, Microsoft executives have said, is assisting physical therapy by guiding patients through the proper movements.

Finally, there's Windows 8, which Ballmer has called the "most remarkable replatforming from Windows," based not in small part on its revolutionary UI. And that, apparently, will not go away.

"We will continue to reinvent the core 'shell' of our family of devices and build upon what we have started with Windows 8," Ballmer writes. "We will keep evolving our new modern look, expanding the shell so that it allows people and their devices to capture, store and organize their 'stuff' in new ways. Our UI will be deeply personalized, based on the advanced, almost magical, intelligence in our cloud that learns more and more over time about people and the world. Our shell will natively support all of our essential services, and will be great at responding seamlessly to what people ask for, and even anticipating what they need before they ask for it."

We've already seen some examples of this: The Live Tiles in the People app within Windows Phone and Windows 8, for example, already display the images of a user's contacts. Meanwhile, Microsoft "evolved" the Start page in Windows 8.1, sharing background images with the desktop and reworking the Apps screen.

What's clear is that Ballmer has a goal in mind for Microsoft, and this week's reorganization was simply the latest milestone on its journey. It's a rare thing for a company to outline its plans for the future, even conceptually. Time will tell, however, if it's headed in the right direction.