Archive for October, 2015

Our Commitment to 99.99% Uptime for Adobe Primetime Authentication

Adobe Primetime Authentication (formerly Adobe Pass) is committed to 99.99% uptime. That’s only around 4 minutes of downtime per month or less. Every month this year Primetime Authentication achieved 99.99% uptime or better, except this month. This month, an external factor put our systems to the test. All our preparations paid off. Primetime Authentication will end October with a 99.95% uptime, which is .04% shy of our goal. This post is about how we keep our uptime commitment.

Architecting robust services

Infrastructure breaks. No matter how much care you take, it breaks. There is no question about it. The only question is when. And the really hard question is actually how prepared you are to deal with it when it breaks.
Any type of infrastructure. Things break in the public cloud and in the private cloud. A wide range of factors such as software bugs, hardware defects, or third-party services failures can put any system to the test. So it takes a robust architecture to keep a service up even when infrastructure breaks.

Here is a recent example. Primetime Authentication is architected as a distributed system running in an active-active configuration between the eastern and western United States. It relies on our DNS provider to geo-balance the traffic. If we encounter problems in the east, the traffic is automatically shifted to the west and vice-versa. This is a major factor in our high availability. Our DNS provider is reliable and not expected to fail. And still…

The outage

Last week our DNS provider experienced a major outage that affected multiple states in the eastern United States and impacted several high profile services. That hit us hard. Let’s see how we dealt with it.

Rapid response

Within minutes of the DNS outage, our team was in a virtual incident war-room scrambling to react. It was after midnight in our time zone, but we were online in minutes. This is where all those annoying little things paid off: the fine-tuned uptime monitoring, the automated alerting system, the discipline of 24×7 on-call, and the practiced incident response procedures. They all came together like a well-oiled machine to save valuable time.
Ok, so what’s next? First we thought that something went wrong with our eastern instance. So we brought that down and expected the traffic to shift automatically to our western instance. No time for deep investigation, you need to act quickly to restore the service, that’s the first priority in an incident. But the traffic didn’t shift and critical alerts kept pouring down. By the time we narrowed this to be a DNS issue, we got an email from our provider acknowledging the outage.

A DNS outage is bad. Your service is up, but Internet users cannot reach it because the name cannot be resolved to an actual IP address. Most services are just waiting for the outage to pass and thus restore user access. But we can’t afford that with a 99.99% commitment. So we have a backup DNS system.

The right precaution

Some time ago we migrated from our in-house operated DNS solution to a cloud provider. And we had the precaution of keeping our DNS settings with the in-house system as well in an inactive status. This proved to be the key to our recovery. We re-activated the DNS settings in our in-house system and made that our primary DNS provider. The fact that we kept the “dormant” records there allowed us to enforce aggressive propagation of the new DNS records. The technical details are less trivial than depicted here and involve a pre-defined hierarchy of domains and subdomains that allowed us to control our DNS in this way. The point is that you cannot just adopt a new DNS provider on the spot because this change will take hours to propagate for the first time.
So our precaution of keeping the backup DNS system saved us. We were back online and fully accessible in a short amount of time. It took us more than 4 minutes, for sure. So we failed our 99.99% for the month by a small margin. But we were back before everybody else was and we were one of the very few services to do so.

Continuing to protect Primetime Authentication uptime

In retrospect, this looks like a simple thing. Such things always do, hindsight is 20/20. Of course, another DNS provider, what could be simpler? You would be amazed how few services actually have that at the ready. Most of the affected services just waited in frustration of not being able to react until the DNS provider fixed their outage, 90 minutes later. A DNS outage at this scale doesn’t happen every day, it happens rarely. But when it happens, you can lose invaluable uptime minutes if you are not prepared.

We continue to remain prepared to protect the uptime of Primetime Authentication when the next challenge strikes.

Exciting trends in video monetization and TV connected devices in Q2 2015 Digital Video Benchmark report

The Digital Video Benchmark report by Adobe Digital Index can help you keep up-to-date with the changes taking place in how consumers view video. It covers both online video consumption and TV Everywhere trends. In the Q2 2015 report, we identified three key trends that will be important to Adobe Primetime customers. Here’s a summary of these trends.

  1. Video monetization is on an upward trend; viewers are now watching nearly 3 ads per video view
    Video ads per video view are up 30% over Q2 2014. Monetizing video appears the easiest in reality TV, where viewers see 3.9 ads per video view, and the hardest in comedy, where viewers see 2.1 ads per video view.
    global monthly average ad start per video start
  2. Authenticated TV is moving back to the living room; 21% of TV Everywhere authentications are happening from TV connected devices
    The share of authenticated viewing via a TV connected device rose from 10% in Q2 2014 to 21% in Q2 2015, an increase of 110% year over year (YoY). This, combined with a 21% decline YoY in tablet video viewing suggests that viewers are ditching their tablets in favor of TV connected devices. Kids swapping out iPads for Apple TV and Roku are leading this charge.
    share of TV everywhere users by device type
  3. Viewing frequency is now the highest on TV connected devices
    In Q2 2014, TV connected devices trailed iOS and Android in the average number of authenticated video views per visitor. By Q2 2015, viewing frequency jumped from 7.2 authenticated video views per visitor to 9.7 authenticated video views per visitor. This makes TV connected devices beat out all other access types on viewing frequency. In comparison, iOS has a viewing frequency of 9.2 authenticated video views per visitor, Android has a viewing frequency of 7.9, and the browser has a viewing frequency of 4.8.
    tv everywhere viewing frequency by access type

You can read the full Q2 2015 Digital Video Benchmark by Adobe Digital Index by downloading a free copy here.

How Adobe Primetime is Incorporating DASH

In March, I wrote about how Adobe Primetime is transitioning to MPEG-DASH, or DASH for short, which is a standard for adaptive bitrate (ABR) delivery of media to any device. This month, Abhinav Kapoor, Sr. Manager of Software Development at Adobe Primetime, shared the latest update at IBC 2015 regarding our plans to incorporate DASH and expand the use of DASH for streaming video.

In his presentation, Abhinav talks about Adobe Primetime’s support for DASH on HTML5, on our native clients across all the devices that we support, and about how we’re using DASH on both the client- and server-side. For example, Adobe Primetime has several server-side components that are ready for DASH, which has several benefits, including reducing the complexities around inserting ads on the server side. He also covers how we’re working on extensions to the standard in order to bring in support for features like WebVTT and common encryption for the rotation of licenses and keys.

Adobe Primetime looks forward to actively pursuing to integrate standards like DASH to engage audiences with premium TV experiences online. Interested in the full story? Watch Abhinav’s presentation in this video.