On Sat­ur­day, Decem­ber 10, we expe­ri­enced a data col­lec­tion inci­dent in our Dal­las data cen­ter. The impact for most of the affected cus­tomers was approx­i­mately 5%-10% of their data col­lec­tion requests were not answered for approx­i­mately 80 min­utes. A smaller set of cus­tomers with very spe­cial­ized and cus­tomized data col­lec­tion imple­men­ta­tions were more sig­nif­i­cantly impacted. We are work­ing directly with impacted cus­tomers. If you are a cus­tomer and feel you may have been impacted but have not yet been con­tacted by your account man­ager, please con­tact 1−800−497−0335 and to talk to some­one live, press 1 and then 1 again. Your sup­ported user should make this call.

While we con­tinue to inves­ti­gate the root cause of the event, we have ascer­tained that a key net­work device in the Dal­las data cen­ter was not pass­ing traf­fic cor­rectly. We are inves­ti­gat­ing, in part­ner­ship with our ven­dor, why the sec­ondary device (this was part of a redun­dant pair) did not detect the prob­lem and fail over and what the ini­tial trig­ger or cause of the event may have been.

An event like this, while rare, often trig­gers a larger dis­cus­sion about redun­dancy, a crit­i­cal ingre­di­ent for any enter­prise ser­vice deliv­ered via the cloud. At Adobe, we have numer­ous redun­dan­cies imple­mented through­out our archi­tec­ture. At each site, we have mul­ti­ple data col­lec­tion ‘net­works’. This event impacted a sin­gle one of these net­works. At each layer within the net­work, we also have redun­dan­cies (mul­ti­ple fire­walls, routers, switches, load bal­ancers, servers, etc.).

This brings us to a much larger redun­dancy strat­egy we began rolling out sev­eral years ago: RDC (regional data col­lec­tion). Adobe cus­tomers can par­tic­i­pate in imple­ment­ing a multi-site redun­dancy strat­egy by mov­ing to RDC. While we have pri­mar­ily touted the per­for­mance ben­e­fits of RDC, cus­tomers also inherit multi-site redun­dancy, which min­i­mizes their impact should any site be affected, but also allows us to take a prob­lem site offline much more quickly. Addi­tional infor­ma­tion can be found here:
http://​blogs​.omni​ture​.com/​2​0​0​9​/​1​2​/​1​4​/​s​p​e​e​d​i​n​g​-​u​p​-​y​o​u​r​-​s​i​t​e​-​w​i​t​h​-​r​e​g​i​o​n​a​l​-​d​a​t​a​-​c​o​l​l​e​c​t​i​on/

We deeply regret the incon­ve­nience this has caused and are tak­ing steps to pro­tect against sim­i­lar events going for­ward. We are com­mit­ted to deliv­er­ing the prod­ucts and ser­vices our dig­i­tal mar­ket­ing cus­tomers need to drive pos­i­tive busi­ness impact and ROI.

0 comments