SkyDriveLike us on FacebookFollow us on Twitter (@skydrive)
HotmailLike us on FacebookFollow us on Twitter (@hotmail)
MessengerLike us on FacebookFollow us on Twitter (@messenger)
Service status
Live Connect Dev Center
Windows Live Help Center
Learn more about Windows Live
Ever wonder how Hotmail stores the billions of email messages we receive each day? Keeping our customers’ data safe and readily available is an immense responsibility that we take very seriously. And to do so efficiently at our scale is a sizeable engineering challenge. This post will discuss how we address some of these challenges and reveal some major improvements we’re making in our storage system. Kristof Roomp is an architect in the Hotmail team and has been working on our storage system for the last 6 years.
Chris Jones
Hotmail’s storage system supports over one billion mailboxes and hundreds of petabytes of data (one petabyte is a million gigabytes, or a million billion bytes). The system services hundreds of thousands of simultaneous transactions from across the world. Just like the rest of Hotmail, our storage system is built using Microsoft technology, including Windows Server and Microsoft SQL Server. These systems are the backbone of Hotmail and are crucial to meeting the high standards we’ve set for the reliability and availability of our service.
The folks who work on Hotmail storage have three main goals: keeping your emails safe, providing new functionality to the Hotmail service, and running the service as efficiently as possible. In many cases, safety and efficiency go together. For example, by automating routine maintenance tasks and providing monitoring to detect problems before they appear to our users, we can reduce the chance of human error and thereby significantly improve the reliability of our service.
Recently, we’ve been working on a major upgrade to our storage system. Starting at the beginning of this year, we’ve been running the new system on a pilot cluster, using personal accounts of Microsoft employees who have volunteered to be test pilots. We’ve now finished certifying this new system, and are satisfied that it provides better reliability to users at a significantly lower price.
I’ll describe some of the key technologies that we have developed at Hotmail to make this happen.
First, what is RAID?
RAID (Redundant Array of Inexpensive Disks) is a technology that allows several hard drives to be attached to a single controller board, which makes them look like a single larger and much more reliable hard drive (sometimes called a “Logical Unit”) to the software running the storage system. A RAID system stores data on multiple drives so that if a single drive fails, the data can be automatically recovered. Although this sounds great in theory, in practice losing an entire RAID set happens all the time, especially if you have thousands of machines.
In Hotmail, we’ve been using RAID for a long time. In order to avoid losing email messages when a RAID set fails, we keep your email on multiple RAID groups, so that even if an entire RAID set breaks, we can still restore your messages.
However, as we looked at deploying drives with capacity greater than a terabyte, we realized that we weren’t getting our money’s worth from a reliability perspective. The reason had to do with the idea of “correlated” as opposed to “independent” failures.
As an analogy, think about engines on an airplane: there are many failures (such as mechanical problems) that only affect a single engine. These are called independent failures, and having more than one engine is helpful in these situations. However, if you were to run into a big flock of birds or run out of fuel, all engines could fail at the same time. These are called correlated failures, since a single event causes multiple failures.
In a similar way, RAID systems can easily deal with problems that affect single (or two in some configurations) hard drives, but they don’t help if the whole machine or the RAID controller runs into problems. For larger drives, we found that having completely independent copies (on hard drives not sharing the same machine or controller) was much more reliable than a significantly more expensive RAID configuration.
The new system ensures that the copies of data reside on independent hard drives, controllers, and machines. This kind of system is nicknamed “JBOD,” which stands for “Just a Bunch Of Disks.” In a JBOD system, the hard drive controller almost completely gets out of the way, which means that the software must now worry about all the failures that the controller previously handled. These failures can range from firmware bugs on the hard drives themselves to issues such as “unrecoverable read errors” that previously were automatically fixed by the controllers. In addition, the software must now scrub the drives periodically to check the data for “bit rot” (i.e., data that has for some reason become unreadable or corrupt). So basically, we built a distributed "RAID" controller completely in software, which replaces the industry-standard firmware ones.
The software we developed for the JBOD system monitors the hard drives schedules repair actions, detects failures, and diagnoses repairs. This software consists of a number of “watchdogs” that constantly monitor for certain types of failures. If the watchdog detects the failure that it is looking for, it raises an alert, which automatically triggers a repair process. This repair process can range from rebooting a machine or restarting a process, to fixing data corruption or even involving a human if progress can’t be made. We'll talk more about our advanced platform for monitoring, deployment, and repair in a subsequent post.
A big advantage of managing the drives in software is that the system knows exactly how many good copies of an email message we have. In the case where it finds that there are too few copies, it can prioritize repair actions to avoid a potentially dangerous situation. In situations where repairs are taking too long, it is possible to move data to another location altogether. This is also possible in RAID in a limited fashion, but it requires that every RAID controller has an extra spare drive hooked up to it, which increases costs significantly.
Building our own distributed system to store replicated email messages was a significant development effort, although the replication itself was simplified by the fact that email messages in Hotmail stay exactly the same as they were when they were delivered (in fact you can see exactly what is stored in Hotmail if you do a “View message source”). Data about email messages that changes (such as read/unread, location in a folder, etc) is stored separately.
The storage system consists of a set of machines, each of which has its copy of an email message and a journal recording messages that have arrived, organized by arrival date. The machines talk to each other from time to time, compare their journals, and copy any messages that they realize haven’t been copied to all machines. This can happen for a variety of reasons, mostly due to machine, network, or hard drive failures. In some cases, the journals are too far out of sync, in which case the system does a full comparison/copy.
Although hard drives have gotten bigger and cheaper, the speed at which they can retrieve data hasn’t changed much. This means that although we can pack more data on larger hard drives, the hard drives would eventually be unable to handle the rate of requests.
One technology that is promising in this area is Flash Storage (also called SSD, or Solid State Drive). SSDs use technology similar to what you'd find on an SD card or USB stick, but with a faster internal chipset and a much longer lifespan. A normal hard drive can perform a little more than one hundred read/write operations per second, whereas some of the fastest SSDs can do over one hundred thousand operations per second. However, this comes at a hefty price, as these devices are 10 to 100 times more expensive than hard drives when you look at what you pay per gigabyte of storage.
To explain how SSDs could help us, I’ll first describe how Hotmail stores your mailbox. In addition to storing the email messages themselves, we also track information about these messages (called metadata), such as the list of messages in your inbox, read/unread status of your messages, conversation threading, mobile phone synchronization etc. This metadata takes up an extremely small fraction of our total storage space, but due to its constantly changing nature, it is responsible for most of the load on our hard drives.
By using SSDs for this small and rapidly changing set of data, and using the largest hard drives available for storing messages, we are able to take advantage of the trend in larger and cheaper hard drives without making any sacrifices in the performance of our system.
What happens if your account is still on one of our older machines? Well, don’t worry, since the older systems run on smaller hard drives, there are more than enough disk operations/sec available to handle your inbox.
We’re extremely excited about our new storage system. The rollout has already begun and all new clusters that we deploy going forward will use JBOD. We will also retrofit JBOD to our existing systems over time. We have about 30 million users on JBOD today, with another 100 million moving to the system over the next couple months.
Our team is already planning and doing early design work for the next set of innovations, which will include hardware architecture changes and low-level software improvements to further increase the efficiency of our storage. We’re looking at patterns of email content and how our users access their data to inform our future designs.
These advancements will ensure that we can scale our service as we continue to expand our features for organizing your inbox, making you more productive, protecting you from spam, and providing you with the fastest, most reliable email service on the planet. Thanks for using Hotmail.
Kristof Roomp
Last week we released the developer preview of Windows 8 (WDP) at the //BUILD conference. While we covered a lot in the keynote, I wanted to provide folks who’ve been following our services for some time with a quick summary of what we talked about and what’s coming next. Windows 8 provides us with an opportunity to reimagine our applications and services in a new context of user interaction and the power of connected apps. Across our teams we have been building applications and services with Windows 8 in mind. In particular, we took the bold steps of:
One of the great things about web-based services is that you can log in from any browser and start right where you left off. The service keeps track of your settings and profile, and you can get started just by typing in your ID. With Windows 8, you now have that same ability for the important parts of your Windows experience, if you choose to. You can sign in to a Windows 8 PC with your Windows Live ID, and then it gets your profile, application settings, and other important information right from the cloud. And it is all built on top of the Windows Live infrastructure we’ve built to support over 500 million active users a month. In fact, over 150,000 people have already logged in to Windows 8 with their Windows Live ID, and as they move across Windows 8 devices, their settings and preferences will go along with them. The Windows Live service also keeps track of all of your Windows PCs and Windows Phones, allowing you to easily connect back to any PC when you are away, or find your phone when it is lost.
We all create documents, photos, and other files, and it is increasingly important that there is a place to store them in the cloud, either so you can get to them from any device, or so you can share them with friends, co-workers, or family members. Now every person who uses Windows 8 or Windows Phone has access to a SkyDrive, where you can put your most important files. These are stored in the cloud, accessible from any browser, and connected to the apps we’ve built for Windows 8 and Windows Phone. And you can share things with anyone, over email or through networks like Facebook, LinkedIn, or Twitter.
Metro style represents a new way to write applications, and we have rewritten our applications from the ground up based on the new Windows 8 platform. Our new applications target the new Windows 8 runtime and embrace the Metro style design principles, and cloud powered by our Windows Live service. An important aspect of our applications is that they are connected to the services you use every day, so you can quickly get a view of what’s happening. We know most people have more than one mail address, and Mail connects to multiple mail accounts, at home or at work. Calendar connects to your work and personal calendar, as well as calendars from friends, co-workers, or family members who have a Windows Live ID. People is a connected address book, bringing together contacts from work, home, and popular networks including Facebook and LinkedIn. Messaging keeps track of all of your IM conversations across Messenger, Facebook, and other connected IM networks. And Photos lets you view your photos, whether on your PC, another PC, SkyDrive, or other popular services.
We'll have a lot more to say when we make the code available for these apps. At BUILD we just showed a short preview of what's on the way.
Of course, an important part of Live and SkyDrive is the opportunity for software developers to use these services to power their applications. As we mentioned in the conference, Metro style applications are automatically cloud powered because Windows takes care of roaming the important settings. If you want to go beyond the basics, we have an API for SkyDrive that enables Windows 8 developers to read and write files to SkyDrive, and allows device manufacturers to add SkyDrive to their devices. And we have an additional set of APIs for Live, Messenger, and Hotmail, allowing developers to use these services. These are all part of Live Connect and were covered in depth in our talk at //BUILD.
We’re super excited by the opportunity we have with Windows 8 and have had a lot of fun building the new applications and services on the platform. Of course, we’re very early in our work and there is a lot more to talk about – what we showed is just a preview of what’s to come. We still have the rest of Windows Live Essentials; updates to Hotmail and Messenger; and even more on SkyDrive, Live, and our applications for Windows 8. We’ll cover these and more in subsequent posts.
Chris Jones – Vice President, Windows Live engineering
This blog post is a follow-up on the outage that occurred on September 8th. Just before 8:00 PM PDT that day, we became aware of a Domain Name Service (DNS) issue that was causing a service interruption for some some Microsoft services, including Windows Live services such as Hotmail and SkyDrive. No customer data was lost or compromised during this outage. The team has investigated the root cause and has taken immediate steps to improve.
So, what happened? A tool that helps balance network traffic was being updated and the update did not work correctly. As a result, configuration settings were corrupted, which caused a service disruption.
At 10:23 PM PDT we began to see service restoration. We confirmed that the incident was resolved by 11:35 PM PDT, although it took some time for the changes to replicate around the world and reach all our customers.
We determined the cause to be a corrupted file in Microsoft’s DNS service. The file corruption was a result of two rare conditions occurring at the same time. The first condition is related to how the load balancing devices in the DNS service respond to a malformed input string (i.e., the software was unable to parse an incorrectly constructed line in the configuration file). The second condition was related to how the configuration is synchronized across the DNS service to ensure all client requests return the same response regardless of the connection location of the client. Each of these conditions was tracked to the networking device firmware used in the Microsoft DNS service.
After restoring service, we have identified two streams of work to drive specific service improvements around monitoring, problem identification, and recovery. Along with these service improvements, Microsoft is focused on further hardening the DNS service to improve its overall redundancy and fail-over capability.
We are also developing an additional recovery process that will allow a specific property the ability to fail over to restore service and then fail back when the DNS service is restored. In addition, we are reviewing the recovery tools to see if we can make more improvements that will decrease the time it takes to resolve outages.
We are determined to deliver the very best possible service to our customers and regret any inconvenience caused by this outage.
Arthur de Haan Vice President Windows Live Test and Service Engineering
ZDNet’s Ed Bott penned a very thorough comparison of Photo Gallery 2011 and iPhoto ‘11 in The Ed Bott Report this past week. While he doesn’t go so far as to declare a winner, he highlights some clear advantages to our design approach. In a lot of ways, his experience and analysis reflects what we set out to achieve with the design. We thought it would be worthwhile to provide a little back story on how we ended up where we did so that folks could compare Ed’s analysis with how it really unfolded.
Looking at Photo Gallery 2008, we knew it was a good candidate for the ribbon. The product has a lot to offer, so the trick was to expose capabilities in a way that allowed users to use them easily. We needed to understand which parts of the product were being used the most, observe and listen to people who couldn’t do what they wanted to do, and then design a product to accommodate both of these needs and desires. This is exactly what teams did in Office 2007 when the ribbon was first introduced.
We had four major design principles as we began the design of Photo Gallery 2011.
A big part of Photo Gallery is about organizing photo and videos, so it was important that we reflect the new file library investments made in Windows 7. Whatever file structure users might create in the Windows File Explorer is maintained inside of Photo Gallery. Vice versa, any new folder structures made inside of Photo Gallery are reflected in Explorer.
Ed touches on this a bit in the beginning of page two of his post where he mentions the left navigation pane as he compares it to iPhoto. The symmetry of folder structures between Photo Gallery and File Explorer makes it easy for users to access their photos and videos in a familiar and predictable way. We know from countless interactions with our customers that people want to have control over the locations of their files. With that control comes confidence that they can organize, find, and enjoy their photos.
Using telemetry data from Photo Gallery 2008 gave us a real leg up on how to structure and organize the overall design of the product and the ribbon in particular. This data basically told us which commands were used most frequently so that we could make sure the most useful features were just a click or two away. (You can read more about telemetry and how we use it in this post from Steven Sinofsky.) There was also a recent post on how the ribbon for the Explorer in Windows 8 leveraged telemetry to arrive at its new design.
This is telemetry data taken from Photo Gallery 2008. If we look at the top ten commands used, it becomes clear right away that navigating, viewing, and managing are prominent. This gave us good insights that these commands needed to be readily accessible in the Home tab. In fact, the usage of these commands is so prevalent that we thought it necessary to duplicate most of them in the status bar so that they’re available no matter which tab you might have selected in the ribbon.
Status bar when in list view.
Status bar when looking at a single photo.
Ed alludes to a bit of this at the end of page two of his analysis where he points out the number of options available at any one time in Photo Gallery is about one third of those available in iPhoto when the ribbon is collapsed. We’ve seen a lot of customers choose to work this way. But what’s more compelling than the sheer number is the commands themselves. It’s not only that there should be a relatively small number of commands always available, but that it’s the right set commands available when users need them.
A big focus point of our design was better enabling people to effectively narrow a large set of photos down to the few they care to share. We asked, “how can users get from 400 photos of their latest vacation to the 40 they care about to the 10 they want to share?” We knew the ribbon could help steer people through that process. This is represented most prominently on the Home tab as each chunk alludes to the flow of managing photos from left to right: New, Manage, Organize, Quick Find, and Share. Naming the chunks in the home tab is important, but it’s even more important to get the names of the top level tabs right. So it was good to see Ed give attention to the View tab.
He zeroed in on the View tab, stating, “Both programs give you the capability to sort and group the display of photos in the pane on the right. Changing the sort order, viewing details such as ratings and captions, organizing photos with tags/keywords… These are among the most common tasks you’re likely to do with either program.” Here again he points out the advantages of the ribbon over traditional cascading menus. On the Home tab we also introduced the notion of flagging files. This is a good example in which we used the ribbon not only to expose frequently-used capabilities, but to introduce new capabilities. One of the quickest ways to narrow a set of photos down is to quickly flag them. It takes the stress out of rating or even deciding to keep or delete a given file.
Using the 400-40-10 model, you can easily flag 40 or so favorite files of the 400 imported in the Home tab.
Then you can use Quick find on the Home tab to view those flagged items.
When I flag my photos in Photo Gallery, I just use Quick find and then use the Insert key on the keyboard to flag the files I care about. This way I can keep one hand on the Insert key and the other on the arrow keys and then simply arrow to the photo I like and flag it. As Ed points out, the number of clicks to do these kinds of operations is minimized with the ribbon versus cascading menus. But in this case, I only have to use the mouse once to view the files I care about! And I’m not the only one taking advantage of flagging. Taking a dip into our Photo Gallery 2011 telemetry on September 8th, 2011, we see Flag showing up among the most frequently used commands on the Home tab.
The organization of the Home tab really sets the tone for app, so making sure it was designed to reflect our “400-40-10” goal played into this. We made big investments in face recognition and face tagging in this release as well, so we wanted the Faces gallery control in the Quick Find chunk to have prominence.
And as you can see, while Flagging has become popular, so has using faces to find people.
Introducing face recognition in Photo Gallery 2011 has been a true delighter for our users. It provides another way for users to easily find, enjoy, and share the photos of the people they care the most about.
Ed’s article, the comments, feedback from customers, and our own telemetry data all remind us that our work is not done. While overall we’re happy with where we landed with the design, everyday usage patterns help to tell a more evolved story. We knew editing in Photo Gallery 2008 was not easily accessible, so we predicted that having it show up as the first tab after Home would result in a boost in usage. That much is true.
Again, looking at telemetry data from September 8th, 2011, we see that people very frequently double click on a photo to edit. The top ten most-used commands are edit commands. It’s not until you get to command number 18 that Delete shows up.
Some of these numbers are deceiving in that editing operations often involve experimentation. And with that iteration, comes a flood of data. The fact remains, editing is very popular. As Ed points out on page four of his post, the ribbon makes editing easy, too. With live previews, it’s easy for users to preview an effect before committing to it with a click.
Hovering over the various exposure adjustments will automatically preview various aspects of brightness, shadows and highlights in a photo without clicking.
As we look at the data and listen to customers, we ponder how the design of Photo Gallery could evolve. We could look at the data and decide to trade off some of the lesser-used organization features for editing features in the Home tab. Or we could look at the data and decide people are successful at doing what they want to do. After all, once they’ve opened a file to edit, the Edit tab is the Home tab!
Clearly there are lots of ways to look at this. And we do. With research studies, customer feedback, and telemetry data, our work doesn’t have to be guess work. Ed’s right: “Everyone has an opinion about user interfaces,” and it turns out his own opinions are backed not only by his own experience, but by data. Reading Ed’s reverse engineering of our design decisions bolsters our confidence that we’re moving in the right direction. We by no means have it all figured out, but we do learn as we go. So by all means keep the feedback coming.
Brad Weed Group Program Manager Windows Live Photo Gallery, Movie Maker, and Writer
If you've been trying to use Hotmail, SkyDrive, or our other Live properties in the last couple of hours you may have noticed problems accessing our services. We're aware of these issues and actively working to resolve them. We apologize for the inconvenience and appreciate your patience.
UPDATE 9:45 PM PT:
We believe we have restored service for all customers at this time. We will continue our investigation into the root cause of these issues and post an update following our investigation. Again we appreciate your patience and apologize for the inconvenience.
UPDATE 11:02 PM PT:
We're aware of reports including the comments posted below that some customers still are seeing issues. We are working on propagating the DNS configuration changes and so it will take some time to restore service to everyone. Again we appreciate your patience.
UPDATE 11:49 PM PT:
We have completed propagating our DNS configuration changes around the world, and have restored service for most customers. Depending on your location you may still experience issues over the next 30 minutes as the changes make their way through the network. Thank you for your patience as we have worked to address these issues.