Automation Journey in the world of L10n!!

Automation Journey in the world of L10n!!

Feb’14, Reetika Ghai

Automate

The blog talks about the importance of automation in the world of localization and its increased need in the world of Agile

Paradigm Shift from Waterfall to Agile in the World of Localization

Do you know which is the fastest land animal in the world reaching speeds up to 113km/h?

Over the last two years, there’s been a gradual shift in the Software Development Life Cycle (SDLC) methodology for most of the Adobe flagship products. Product management has moved from yearlong waterfall product development life cycle to sprint-based Agile methodology (based on iterative and incremental development, where requirements and solutions evolve through collaboration between self-organizing, cross-functional teams).

As a result of changing market trends, we need to reinvent our approach to localization testing to meet the changing requirements of Agile methodology. In Agile, development and test cycles are shorter with quick turnaround time. In localization, test volumes have spikes considering the duration of the sprint cycle of 2-3 weeks. Features require frequent validation across multiple locales and platforms before certifying for a release in a simultaneous release model (sim-GM). In the Agile framework, it’s important to be cognizant of the business goals from localization perspective. I would categorize these in three broad areas:

  1. Time boxed frequent releases to market: With agile most of the Adobe products have at least one release every quarter to a frequency as high as weekly/monthly releases
  2. Increased test scope leading to increased localization efforts: With each sprint the legacy scope to certify LOC build increases
  3. Higher focus on rapid new feature development with simultaneous release to market: Certifying features on N Locale and M platforms in a sprint of 3-4 weeks

These goals create the following challenges for an International Quality Engineering (IQE) team while deciding the scope on the localized builds for localization testing:

    • Ensuring increased test coverage on the new features while balancing the coverage for legacy feature areas
    • Ensuring re-usability of tests across various platform variants
    • Ensuring test accuracy across repetitive scenarios on multiple languages
    • Ensuring faster execution to uncover defects early on
    • Ensure all the features work as expected on all supported platforms and locales
    • Ensure co-existence with different versions of the released product/patches
    • Ensure shipping the product simultaneously in all supported locales across geographies (sim-GM)
    • Ensure optimized test coverage on all the supported locales and platform variants

 

Automation in Agile & Localization

Why automation testing in Localization?

- Multiple Releases in an year

- High volume of testing

- Complexity of Platform: Locale combination

- Improved test coverage leading to better quality

- Scalability

- Faster time to market

- Cost Effectiveness

With these initial thoughts, we proposed to expand the automation coverage of Dreamweaver product features from English language to localized languages in September 2012. Our initial Goal was to attain 45% feature automation coverage on localized builds with respect to coverage on English build on Mac platform.

Gradually we built the feature automation capabilities in the next six months, starting from enabling the automation framework for localization (i.e., added support to the automation framework to run scripts on the localized operating system) to running daily smoke tests on all the 15 supported languages, and eventually having good feature level automation coverage.

Automation is a great way to overcome the above challenges and effectively achieve optimized test coverage on localization builds. With automation, it would be possible to certify incremental creative cloud releases for all the supported operating systems and language combinations supporting the time-bound releases.

With multiple releases to market in a year, manual execution of the repeatable test scope by the localization vendors leads to increased test efforts. The major part of the increased test effort can be attributed to incrementally increasing legacy test scope, i.e., legacy scope is cumulative the sum of all the deliverables in the past of a product and would increase with each sprint. On the other hand, automated tests can be run over and over again ensuring defined coverage across platforms and language combinations, thereby contributing to the overall product quality for the time boxed release. This not only eliminates the test redundancy but also helps in producing faster test results.

Having the legacy area automated would help the localization tester focus manually on the current sprint deliverable, hence uncover defects early in the test cycle.The IQE needs to be cautious in deciding the scope of automation on localized builds. Prioritizing the automation coverage is very important.

With each quarterly release to market, the certification scope of the legacy features set for a product is increasing, leading to amplified repeatable test effort across multiple sprints compared to one time validation in the yearly releases model

Legacy Automation Coverage

Journey into Dreamweaver Automation

DW Localization team has 88% Functional Coverage & 86.5% of conditional coverage w.r.t core coverage of 50% conditional and functional coverage in CC release for MAC!

For adopting product automation in localized languages, our journey stared by answering a few of our initial questions:

  • What features do we need to automate?
  • What will be the sequence of feature automation?
  • What locales should we consider to start with, based on data from prerelease and bug history?
  • What would be the best approach for optimized test coverage in the different locales?
  • In the automation framework, where should the locale specific strings (used in Test scripts) be placed? Or should we pull the strings directly for comparison from the Adobe Localization Framework (ALF) at runtime?
  • How much effort is required for adopting automation on locales?
  • What would be the initial setup required to start automation in the different locales?
  • How much additional effort is required for running automation scripts in localized builds regularly?
  • What will be the hardware needs and the challenge to meet them?
  • What should be the frequency of automation runs (daily smoke, basic feature plan, and new feature plan)?
  • How to have the best execution turnaround time on all locales? What should be the optimization matrix considering fast turnaround time in agile?

Initial 6 months journey into adoption of automation framework for Dreamweaver localization

Time chart

Dreamweaver Automation Framework

Dreamweaver automation is based on the Adobe homegrown automation framework called ‘Jerry’. The framework was developed by the Dreamweaver English QE team. It was written in Core Java, supported by apple scripts and java scripts in the backend, making use of the Dreamweaver’s API exposed by the developers.

DW framework


The diagram depicts the automation workflow:

Step 1: A job ticket (contains configuration details like TC #, Platform, Machine details, language information etc.) is fed into the Pulpo server.

Step 2: Pulpo server primary purpose is machine management and result logging. Pulpo server invokes the test machine based and executes the test automation based on the plan mentioned in the job ticket.

Step3: Once the execution is completed the log/results are copied to the Pulpo server for further analysis.

Step 4: Results are logged to the automation dashboard “AutoDash”

The Jerry framework contains automated test cases grouped under various test plans:

Daily Smokes – Basic test for validation of daily build

Basic Features Plan – Contains test cases of the legacy and new areas covering feature smoke in Test Studio

Acceptance Plan – Contains acceptance and full test pass coverage for features developed in legacy and present release cycle in Test Studio

We started with one iMAC machine dedicated to Dw automation. However, soon after proof of concept was successful, we added one more dedicated machine for automation on localized builds. The above test plans got executed on a pre-scheduled basis across all 15 locales on the predefined execution plan. Job tickets distributed across 15 locales were fed to the Pulpo server either manually or automatically and were triggered on the arrival of new build in Codex. Typically, by the time we arrived at the office, build sanity was completed on all the locales and we were good to share the builds with our vendor partners.

For monitoring and optimization of test coverage across 15 languages, a dedicated execution calendar was followed. Based on the calendar, different automation test plans were executed on various locales/platform combinations on a daily basis. Daily smoke test for build validation were executed, followed by dedicated full feature test pass on the weekends. The execution was pre-scheduled and the test coverage was distributed across locales for optimal results given the time and machine constraints.

Accomplishments & Learnings

Accomplishments

In the Creative Cloud (CC) release, we benefitted from having automated test passes on localized builds across 15 languages:

  • Overall test coverage efficiency improved four folds compared to manual test execution
  • Quick sanity test for acceptance of localization build before passing the build to vendor partners increased efficiency
  • Achieved quick turnaround time for basic feature testing by automation scripts
  • Parallel certification on multiple builds (Patch and Main line builds)
  • More focus on new features development part of the current sprint by the localization functional testers
  • Prerelease build certification completely through automation
  • Built blueprint and Code Sign verification through automation on all locales in 2 hours compared to 32 hours of manual verification

Learnings

  • Support from the core team: It is essential to have the automation blessed from the English team for optimal support. In case of Dreamweaver, we got immense support from the team, especially from Kiran Patil (Quality Manager) and Arun Kaza (Sr. Quality Lead) for driving the automation efforts on localized builds
  • Pilot on automation framework for one Tier 1/double-byte/Cyrillic locales to ensure the framework was robust and would support automation on most of the locales
  • Always document the issues /challenges you face during setting up automation, they always act as a reference point later
  • Ensure core scripts are independent of English strings. In Dreamweaver, updating the legacy automation scripts to make these scripts run on localization was a big challenge, as automation scripts were failing at string comparisons. Aishvarya Suhane (Localization Automation QE) was a great help for writing functions in automation framework and creating a few new scripts for resolving localization-specific issues.

Cheetahs in the world of localization …

Special thanks to Guta Ribeiro for inspiring & mentoring me to write my first Blog & Rakesh Lal for his support.

GALA Webinar on Localization Project Management

This article was originally written in English. Text in other languages is provided via machine translation.

Manish Kanwal, International Program Manager at Adobe will be conducting a webinar at GALA (Globalization and Localization Associates), which is the largest non profit standards organization within the language industry. The webinar will present insights into the best practices for managing a complex localization project. Additionally, it will elucidate with a case study of a comprehensively large project with engineering teams spread across the globe including, linguistics, reviewers, legal, supply-chain, marketing, customer-support and many more.

Join this webinar to acclimatize what it takes to project manage and localize in demanding conditions, right from the point the product is envisaged until its public launch. Event details are available here, it will be broadcasted on 26 July 11:00 EDT

InDesign CS6…. Welcome to India!

This article talks about the overall objective of Localization in a new market or in business terms an “Emerging Market”. You might wonder, “why that specific word Emerging?” Because of the business opportunity it presents by taking a product to a new market where the demand exists, but somehow the product was not made available.

In the publishing domain, India is still one of the few countries where Print has seen a steady growth. Excerpts from one of the famous research site below:

“Contrary to most other markets in the world that continue to witness an erosion of the print media industry, in India, the sector witnessed a growth of ten percent in 2010 and is expected to continue to grow at a similar pace over the next five years. Rising literacy levels and low print media penetration offer significant headroom for growth, says a FICCI-KPMG report, recently released at FICCI FRAMES 2011 event…………”[Source All About Newspaper, publish date March`2011]

Does this present an opportunity for Adobe to expand in the Print Media space leveraging its one of the most popular Desktop publishing software InDesign®. Yes, but at what cost? Let’s weigh in the cost and benefits.

  1. Over the course of last few years, Adobe India sales force has been meeting Indian customers to understand how InDesign can be made ‘India ready’.
  2. In India, English is quite close to as being the second most spoken language just behind Hindi, giving a leeway to probably still hit the market with an English user interface (UI).
  3. The most talked about area in the frequent customer meetings was the support of Indic scripts in Print and Desktop Publishing Adobe applications. The current World-Ready composers for middle-eastern text included partial support for several Indic scripts. However, a number of bug fixes and product support requirements were needed for Adobe to officially certify and launch the product in India.

The specifics listed above did carve a path for InDesign to see support for Indic scripts in CS6 release. Based on input from the Product Management, the following 10 Indic scripts ranked highest on the priority list to support:

Each of the locales above have a good percentage of Print Media in the Indian market ranging from Newspaper, Magazines, Journals, etc. To support these locales was a tough road ahead since most of these locales use complex character combination, glyphs, hyphenation rules, dictionary support.

Phase 1 of this project included adding dictionary support in InDesign for these locales. We integrated the locale-specific open source dictionaries, evaluated them against competing products (with similar support) spanning a series of script specific test data hand-picked by linguists. The test criteria being:

  • Test maturity and quality of the dictionaries embedded
  • Misspell words intentionally and compare the corrected words
  • Ensure the words in InDesign when copied maintain their sanctity
  • Validating a few language rules, as applicable, such as hyphenation, matras, spellings, etc

Dictionary evaluation did show quite impressive results, allowing us to move to second phase of this endeavor of analyzing InDesign for Indic scripts. After a significant number of complex workflows, a few engineering tweaks along the way, we were able to achieve what we set our eyes at initially.

  • Added dictionaries and spell checkers for the 10 scripts
  • Added Hyphenation for the 10 scripts
  • Bundled 1 Indic font family: Adobe Devanagari
  • Included a script that users can run to set relevant defaults and correctly handle imports from Word docs etc.

Even though we started off this effort as a seed project, codenamed as InDesign Indic 1.0, we were able to achieve more than we shot for. InDesign proved not just compatible for the majority of the locales listed above but offered notable support for even the most complex glyphs.

Switch to the World-Ready Composer, an alternate composition engine, with a single click of indicPreferences.js in Window > Utilities > Scripts panel to explore the Indic world in InDesign. By virtue of basic Indic script support in InDesign CS6, you can now type in these languages and characters would shape and render correctly. And yes, there will be more refinements to the Indic Script support in future releases to come.

Let us know what you think and how you plan to use these features. Please visit here for the complete list of Language support in InDesign CS6.

Contributed by Harpreet Singh (Adobe India)

The Localization Wall

This article was originally written in English. Text in other languages was provided by machine translation.


Des Oates
Localization Solutions Architect

I first got involved in the localization industry when I joined Aldus Corporation in Scotland in early 1994 shortly before it became part of Adobe. Kurt Cobain was still rockin ‘n rollin. Bill Clinton had just completed his 1st year of his 1st term and D:Ream were top of the UK music charts with ‘Things Can Only Get Better’. A prophetic anthem for todays article.

Back then Aldus’ European localization team comprised of a group of around 40-50 in-house staff comprising of Localization Engineers, QE, Linguists, Graphics/DTP Professionals, Planners and Researchers. A grand assembly for sure. But as I recall our delivery capabilities were not quite so grand: For a typical software release, a localization project would:

  • Target no more than 10 target languages in total
  • Have no more than 2 or 3 languages actively worked on at any time
  • Be the only major software release worked on at that time
  • Employ little or no external partners
  • Take up to 9 months to complete large projects.

Nine months to localize one product in 10 languages. Seriously? NASA can get a robots to Mars faster!

Contrast this to today. In Spring 2010 we released Adobe Creative Suite 5.0 :

  • 5 Suite Versions.
  • 15 individual products
  • 24 languages

Over 600 localized applications simshipped* with English, with 50% bug reduction over the previous release. I think you’ll agree it’s an incredible step up from the old days.

Nowadays Adobe Globalization group is slightly larger than it was back then. We focus mostly on Program Management, Globalization/Engineering Leadership and International QE. Almost everything else is handled by trusted partners. We are always looking to improve our productivity, quality, and global reach. As such we’ve made a lot of changes over the years to our processes our staff and our technology. It’s hard to capture all the changes we’ve made succinctly in a article like this, but based on this experience, I thought I’d share some lessons we’ve learned along the way.

The biggest changes we have made are in these interdependent areas: Architectural, technical, and cultural. Here’s some key points:

  • Internationalization. If done well initially, the localization benefits (financial and time-to-market) will outweigh up front the costs by an order of magnitude. Evangelizing best I18n practices for your technology is also a worthwhile endeavour. Internationalization support should be a key criterion when deciding on your development platform for your project.

 

  • Automation. We are always striving to improve localization automation in our business. Don’t think of localization as a human process. It doesn’t have to be. It could be a series of automated steps, one or more of which may require some human translation input. As a rule of thumb, the more manual steps you have in your localization process, the costlier it will be. Whether you use a GMS, a bespoke system, or just a bunch of scripts- it doesn’t matter. You will reap productivity rewards and reduce costs if you employ reliable, maintainable and repeatable automation.

 

  • Release/Build Integration. In the old days, our Localization Engineers built every component of the localized software that went on the CD manually on their own workstation. It was error-prone, and labor-intensive and required a lot of QE. Now all application language versions are built as part of a unified process. Localization has become simply a release engineering sub-process, allowing us to scale up our efforts dramatically. If you first optimize your automation, it makes sense to integrate the process into a single multilingual release configuration.

 

  • Trusted/Trusting Partners: The final area of change was the way we interacted with other groups. We identified cultural and communication barriers between us and the groups we work with. Ultimately you need to establish trusted effective partnerships with the stakeholders in your localization processes. It may be internal teams such as development teams or business units that you need to reach out to, or external partners such as LSPs or translation providers.

 

Here at Adobe we started the ‘World Readiness’ programme: An initiative lead by my colleague Leandro Reis which provides an assessment framework to evaluate the global-readiness of our products. Along with highlighting the problems it offers advice and expertise on how to fix them. Our internal ‘customers’ were compelled by this approach, and our internal localization walls began to fall.

Similarly if you use external partners, they should be willing and capable of integrating with your business – not vice versa. That may require some initial training and ongoing mentoring. It’s easy decide not to do this, to keep the localization wall high between you and your partners, throw localization work back and forth over it but that model is ultimately more costly. The lack of transparency can lead to project overruns, increased defect rates, and occasionally chaos. However if you streamline your own localization processes, lower your localization walls and select competent partners willing to embrace your business processes, then you will gain a trusted capable partner, and your partners will gain a high-value, repeat-business client. A win-win situation.

Just for fun I looked up the number 1 song in the UK charts when Adobe customers across the globe started receiving their localized copies of Creative Suite 5 in May 2010…

…”Good Times” by Roll Deep.

 

* simship: No more than 5 days after English