Posts in Category "Best Practices"

Automation Journey in the world of L10n!!

Automation Journey in the world of L10n!!  

Feb’14, Reetika Ghai

Automate

The blog talks about the importance of automation in the world of localization and its increased need in the world of Agile

Paradigm Shift from Waterfall to Agile in the World of Localization

Do you know which is the fastest land animal in the world reaching speeds up to 113km/h?

Over the last two years, there’s been a gradual shift in the Software Development Life Cycle (SDLC) methodology for most of the Adobe flagship products. Product management has moved from yearlong waterfall product development life cycle to sprint-based Agile methodology (based on iterative and incremental development, where requirements and solutions evolve through collaboration between self-organizing, cross-functional teams).

As a result of changing market trends, we need to reinvent our approach to localization testing to meet the changing requirements of Agile methodology. In Agile, development and test cycles are shorter with quick turnaround time. In localization, test volumes have spikes considering the duration of the sprint cycle of 2-3 weeks. Features require frequent validation across multiple locales and platforms before certifying for a release in a simultaneous release model (sim-GM). In the Agile framework, it’s important to be cognizant of the business goals from localization perspective. I would categorize these in three broad areas:

  1.   Time boxed frequent releases to market: With agile most of the Adobe products have at least one release every quarter to a frequency as high as weekly/monthly releases
  2. Increased test scope leading to increased localization efforts: With each sprint the legacy scope to certify LOC build increases
  3.  Higher focus on rapid new feature development with simultaneous release to market: Certifying features on N Locale and M platforms in a sprint of 3-4 weeks

These goals create the following challenges for an International Quality Engineering (IQE) team while deciding the scope on the localized builds for localization testing:

    • Ensuring increased test coverage on the new features while balancing the coverage for legacy feature areas
    • Ensuring re-usability of tests across various platform variants
    • Ensuring test accuracy across repetitive scenarios on multiple languages
    • Ensuring faster execution to uncover defects early on
    • Ensure all the features work as expected on all supported platforms and locales
    • Ensure co-existence with different versions of the released product/patches
    • Ensure shipping the product simultaneously in all supported locales across geographies (sim-GM)
    • Ensure optimized test coverage on all the supported locales and platform variants

 

Automation in Agile & Localization

Why automation testing in  Localization?

- Multiple Releases in an year

- High volume of testing

- Complexity of Platform: Locale combination

- Improved test coverage leading to better quality

- Scalability

 - Faster time to market

- Cost Effectiveness

 With these initial thoughts, we proposed to expand the automation coverage of Dreamweaver product features from English language to localized languages in September 2012. Our initial Goal was to attain 45% feature automation coverage on localized builds with respect to coverage on English build on Mac platform.

Gradually we built the feature automation capabilities in the next six months, starting from enabling the automation framework for localization (i.e., added support to the automation framework to run scripts on the localized operating system) to running daily smoke tests on all the 15 supported languages, and eventually having good feature level automation coverage.

 

Automation is a great way to overcome the above challenges and effectively achieve optimized test coverage on localization builds. With automation, it would be possible to certify incremental creative cloud releases for all the supported operating systems and language combinations supporting the time-bound releases.

With multiple releases to market in a year, manual execution of the repeatable test scope by the localization vendors leads to increased test efforts. The major part of the increased test effort can be attributed to incrementally increasing legacy test scope, i.e., legacy scope is cumulative the sum of all the deliverables in the past of a product and would increase with each sprint. On the other hand, automated tests can be run over and over again ensuring defined coverage across platforms and language combinations, thereby contributing to the overall product quality for the time boxed release. This not only eliminates the test redundancy but also helps in producing faster test results.

Having the legacy area automated would help the localization tester focus manually on the current sprint deliverable, hence uncover defects early in the test cycle.The IQE needs to be cautious in deciding the scope of automation on localized builds. Prioritizing the automation coverage is very important.

With each quarterly release to market, the certification scope of the legacy features set for a product is increasing, leading to amplified repeatable test effort across multiple sprints compared to one time validation in the yearly releases model

Legacy Automation Coverage

Journey into Dreamweaver Automation

DW Localization team has 88% Functional Coverage & 86.5% of conditional coverage w.r.t core coverage of 50% conditional and  functional coverage in CC release for MAC!

For adopting product automation in localized languages, our journey stared by answering a few of our initial questions:

  • What features do we need to automate?
  • What will be the sequence of feature automation?
  • What locales should we consider to start with, based on data from prerelease and bug history?
  • What would be the best approach for optimized test coverage in the different locales?
  • In the automation framework, where should the locale specific strings (used in Test scripts) be placed? Or should we pull the strings directly for comparison from the Adobe Localization Framework (ALF) at runtime?
  • How much effort is required for adopting automation on locales?
  • What would be the initial setup required to start automation in the different locales?
  • How much additional effort is required for running automation scripts in localized builds regularly?
  • What will be the hardware needs and the challenge to meet them?
  • What should be the frequency of automation runs (daily smoke, basic feature plan, and new feature plan)?
  • How to have the best execution turnaround time on all locales? What should be the optimization matrix considering fast turnaround time in agile?

Initial 6 months journey into adoption of automation framework for Dreamweaver localization

Time chart

Dreamweaver Automation Framework

Dreamweaver automation is based on the Adobe homegrown automation framework called ‘Jerry’. The framework was developed by the Dreamweaver English QE team. It was written in Core Java, supported by apple scripts and java scripts in the backend, making use of the Dreamweaver’s API exposed by the developers.

DW framework


The diagram depicts the automation workflow:

Step 1: A job ticket (contains configuration details like TC #, Platform, Machine details, language information etc.) is fed into the Pulpo server.

Step 2:  Pulpo server primary purpose is machine management and result logging. Pulpo server invokes the test machine based and executes the test automation based on the plan mentioned in the job ticket.

Step3: Once the execution is completed the log/results are copied to the Pulpo server for further analysis.

Step 4: Results are logged to the automation dashboard “AutoDash”

The Jerry framework contains automated test cases grouped under various test plans:

Daily Smokes – Basic test for validation of daily build

Basic Features Plan – Contains test cases of the legacy and new areas covering feature smoke in Test Studio

Acceptance Plan – Contains acceptance and full test pass coverage for features developed in legacy and present release cycle in Test Studio

We started with one iMAC machine dedicated to Dw automation. However, soon after proof of concept was successful, we added one more dedicated machine for automation on localized builds.  The above test plans got executed on a pre-scheduled basis across all 15 locales on the predefined execution plan. Job tickets distributed across 15 locales were fed to the Pulpo server either manually or automatically and were triggered on the arrival of new build in Codex. Typically, by the time we arrived at the office, build sanity was completed on all the locales and we were good to share the builds with our vendor partners.

For monitoring and optimization of test coverage across 15 languages, a dedicated execution calendar was followed. Based on the calendar, different automation test plans were executed on various locales/platform combinations on a daily basis. Daily smoke test for build validation were executed, followed by dedicated full feature test pass on the weekends. The execution was pre-scheduled and the test coverage was distributed across locales for optimal results given the time and machine constraints.

Accomplishments & Learnings

Accomplishments

In the Creative Cloud (CC) release, we benefitted from having automated test passes on localized builds across 15 languages:

  • Overall test coverage efficiency improved four folds compared to manual test execution 
  • Quick sanity test for acceptance of localization build before passing the build to vendor partners  increased efficiency
  • Achieved quick turnaround time for basic feature testing by automation scripts
  • Parallel certification on multiple builds (Patch and Main line builds)
  • More focus on new features development part of the current sprint by the localization functional testers
  • Prerelease build certification completely through automation
  • Built blueprint and Code Sign verification through automation on all locales in 2 hours compared to 32 hours of manual verification

Learnings

  • Support from the core team: It is essential to have the automation blessed from the English team for optimal support. In case of Dreamweaver, we got immense support from the team, especially from Kiran Patil (Quality Manager) and Arun Kaza (Sr. Quality Lead) for driving the automation efforts on localized builds
  • Pilot on automation framework for one Tier 1/double-byte/Cyrillic locales to ensure the framework was robust and would support automation on most of the locales
  • Always document the issues /challenges you face during setting up automation, they always act as a reference point later
  • Ensure core scripts are independent of English strings. In Dreamweaver, updating the legacy automation scripts to make these scripts run on localization was a big challenge, as automation scripts were failing at string comparisons. Aishvarya Suhane (Localization Automation QE) was a great help for writing functions in automation framework and creating a few new scripts for resolving localization-specific issues.

Cheetahs in the world of localization …

Special thanks to Guta Ribeiro for inspiring & mentoring me to write my first Blog & Rakesh Lal for his support.

Join Us: 3rd Adobe Experience Manager Multilingual Content Special Interest Group Meeting

The Multilingual Content Intelligence team at Adobe is excited to host our third Adobe Experience Manager Multilingual Content Special Interest Group (SIG) meeting on Thursday, Nov. 7, at our headquarters in San Jose. Our program this time is on Digital Asset Management (DAM), and we plan to focus on how DAM can be used for multilingual purposes. During the meeting, we’ll also have a Multi Site Manager (MSM) review session to share feature enhancement plans for future releases.

This is a great opportunity to understand basic concepts of DAM and related best practices in a multilingual context. Attendees will also rub elbows with our Adobe experts, share their experiences and challenges, and network with peers from various industry leading companies that are putting Adobe Experience Manager to use.

The details:

Date: Thursday, Nov. 7, 2013

Time: 8:30 AM – 4:30 PM PST
Address:  Adobe San Jose headquarters
345 Park Avenue
San Jose, CA, 95110

The event is free, but you must register in advance to attend. For registration, please visit: https://adobesig.eventbrite.com

For more information or any questions, you’re welcome to ping me at: seunlee at adobe.com

We hope to see you next week!

Seungmin Lee

Sr. Program Manager

Five Golden Rules to Achieve Agile Localization

Illustrations by Julia Feng (OhISeeItNow.com)

Agile_Loc_LogoOver the past decade, many software development teams have switched their development methodology from a waterfall model to something much more agile such as Scrum. Through this transition, their expectations towards other teams such as Localization have changed and these teams have had to improve their agility too.

At Adobe, we have a centralized Localization group that currently supports 135 product and functional teams. Most of these teams have adopted some form of agile development methodologies and have reduced their development cycle from 18-24 months to yearly, quarterly, monthly and, these days, bi-weekly releases. A couple of Adobe product teams and other companies are even releasing updated versions of their product multiple times per day, making it imperative for Localization to keep improving its agility.

Drawn from our experience, this article presents Five Golden Rules that need to be satisfied in order to achieve optimal agile localization.

Rule 1 – “We are one Team!”

Within many companies, Adobe included, Localization is a centralized function serving all product and functional (e.g. Marketing, Sales, Legal, etc.) teams. This structure makes sense because localization is a specialized field, therefore resources (people, tools) and processes can be leveraged across the company. Nonetheless, localization should still be everyone’s concern. The Localization team can come up with many solutions, but the best ones originate when there is a true partnership between the product/functional and localization teams.

We're one team!

We’re one team!

The most agile teams treat Localization staff as if it were part of their own team.

A core aspect of Scrum is to include all skill sets, including localization, required to deliver a product to users. Therefore, Localization team members must be included in all development aspects – from backlog review to retrospective – so they could plan and address international issues early on. Strong partnerships also need to be established with localization vendors when companies, such as Adobe, engage with partners and vendors for their translation and testing activities.

Customer engagement is a key aspect of agile methodologies as it validates the quality and usefulness of the work performed thus far. We recommend engaging with international customers too, because their issues increase awareness around internationalization.

In summary, all stakeholders (development teams, functional teams including localization, vendors and customers) need to collaborate closely in order to achieve great agility.

Rule 2 – “Internationalization is King”

In our Globalization Myth Series, we defined Internationalization (commonly abbreviated as i18n) as an engineering exercise focused on generalizing a product so that it can handle multiple languages, scripts and cultural conventions (currency, sorting rules, number, date and time formats…) without the need for redesign.

Internationalization is King

In other words, the better internationalized an application is, the easier it will be to localize.

In the waterfall model, teams could possibly work around some of the internationalization deficiencies because of longer development cycles. Unfortunately, in the agile world, there is not enough time to look for work-around solutions anymore. The code needs to be internationalized from the get-go.

There are various approaches to improve internationalization in a company, which includes the following:

  • I18n_EducationEducation: Training core developers is an effective way to reduce the number of internationalization issues in a product. By exposing engineers to localization and internationalization issues, they gain a broader perspective on the impact of their code and avoid some of the classic internationalization pitfalls.

 

  • I18n_LibrariesInternationalization Libraries: Leveraging Open Source internationalization libraries, such as ICU or JavaScript i18n, is another good practice. Instead of re-inventing the wheel, engineers can reuse code that has already been validated by others. Also, i18n libraries usually support 100+ locales, which require a significant amount of research and development time. It will be hard for a single team or even company to support so many locales.
  • Peer_ReviewCode review: Practicing peer reviews  is an effective method to reduce internationalization defects in a product. Since developers know their code will be reviewed, they pay more attention to its quality (peer pressure effect) so it also benefits internationalization. Some companies even automate this review process using tools such as Globalyzer.
  • Globalization Report Card: Benchmarking products against an ideal architecture helps to improve internationalization too. As part of our World-Readiness program, we created a Globalization Report Card system to assess the degree of world-readiness in each Adobe product. This scorecard measures products against a set of internationalization criteria (ability to input international characters, display date formats, translate the user interface, and so on…). It is an efficient way to track progress made by each team over time and can even create some healthy competition among product teams. These teams are motivated to be on top of the i18n charts!

Globalization Report Card

Rule 3 – “Integrate Localization into the Development Process”

To release a new product, development teams have many high-priority tasks and usually prefer not to have to worry about localization until necessary. As a consequence, localizability issues are often discovered too late and encounter the risk of being deferred to a future release. Product teams don’t always anticipate the impact of a particular task/feature on localization, and quite often, the Localization team isn’t able to influence design or development until the feature is already implemented.

In an agile process, features and development tasks are tracked in a backlog and reviewed at the beginning of every sprint. To eliminate the side effects of the “throw-over-the-wall” model described above, it is critical to include Localization representatives during these sprint-planning meetings so more visibility and importance are given to the localization tasks. This also provides great educational value to all stakeholders who can then understand the impact of their decisions on the localization process. Localization or a proxy should also attend the daily sprint meetings to keep up with the development pace and decisions. By attending these meetings, Localization team members can be much more proactive and influent.

Localization Process Integration

Adobe Revel and Photoshop.com are examples of teams that integrate localization into their development process. They also prioritize localization intensive features/tasks upfront – carving enough time for the Localization team to run its process and deliver high-quality localized releases.

In a recent Localization World event, Amrit Singh (International Program Manager for our Installation technologies) presented LocBan (Kanban applied to Localization). Just like in a Toyota factory, the Localization team maintains a board of “To Do”, “Work In Progress” and “Done” tasks which provides great visibility on the localization “conveyor belt”. Similarly, it would be beneficial to maintain Kanban boards for each translator. In the waterfall model, translators used to receive large localization kits, which they had to scramble to complete within the deadline. In the agile world, translators are now able to “pull” work as their bandwidth opens up.

Localization Kanban

By using an integrated Kanban board, everyone has a clear understanding of all the various dependencies and accountabilities, resulting in stronger collaboration and higher success rate.

Rule 4 – “Reduce, Reuse and Recycle”

Localization can generate a lot of waste if not planned properly. So, it is key to become “green” in order to become more “agile”.

Reduce, Reuse, Recycle

Reduce

It is clear that reducing the localization effort will have a positive impact on a team’s agility. This could be achieved in 2 ways: by validating the localization scope and by reducing the translation waste generated during the localization process.

  • Reducing Localization Scope

The Localization Manager’s job is to ensure the company localizes the right product and content into the right language set. At Adobe, we have had situations in which we were localizing too much content. For example, using Adobe’s Digital Marketing Suite, we discovered that Russian customers prefer reading Development documentation (such as API descriptions) in English rather than in Russian. We were able to save a lot of time and cost by removing this component from our localization requirements.

Similarly, through market research, we discovered that most Middle-Eastern Creative Suite customers prefer to use an English user interface with Arabic or Hebrew documentation. This combination makes English content such as videos and tutorials more accessible to them.

In short, tracking web analytics and engaging with customers, power users, pre-release testers and geos constitute a great way to validate the localization requirements and improve agility.

  • Reducing Localization Waste

Once the localization requirements are confirmed, it is key to limit the translation waste generated during the localization process. This obviously impacts the translators’ work but also the bandwidth of the localization staff.

Sweep Away Waste!An effective way to reduce localization waste is by understanding its root cause. At Adobe, we categorize all localization defects through a common set of keywords, which provides us with a good picture of the issues faced across products. We can then develop solutions to reduce, if not eliminate, these defects.

Localization waste sometimes originates from English strings -assuming English is the source language. Indeed, translations created before English strings get finalized will need to be revisited and will likely generate some waste.

In the agile world, we can’t afford that extra time, so it is important to validate the English content before handing it off to the translators. Doing something as simple as spell checking can help to reduce a lot of localization waste. In a product such as InDesign, about 3% of the English user interface strings are updated once they get reviewed for spelling and grammatical mistakes. For a product that is localized into 25 languages, this represents a waste equivalent to 75% of a single language scope!

Also, many of the software localization testing activities are necessary because localization is happening out of context. Solving that problem can tremendously speed up the localization process. In an ideal world, localization should be a product feature that allows translators to translate the user interface in-context. Facebook did a great job in this area by enabling translators (in this case its user community) to translate and provide feedback within the application itself. Alternatively, translators should be provided context information through builds, screenshots or meta-data information (e.g. developer comments, feature name, expected delivery time, etc.).

To reduce waste, it is also recommended that localizers develop glossaries, style guides and tools that leverage previous localizations.

Ultimately, it’s critical for translators to validate their work as they translate. That way, activities down the production line can be eliminated or reduced, which makes the entire process more agile.

Reuse

Reuse when it makes sense!Reusing strings can sometimes be a source of challenging defects in software localization, so it has to be handled carefully. For example, the English string “none” could be translated as “aucun” or “aucune” in French based on the gender of the noun to which it refers.

That said, reusing strings – in the same context – could also help to improve agility, since these strings won’t need to be translated multiple times.

An area where Adobe has experienced positive results with reducing and reusing English content is in our instructional content. In documentation, Adobe relies on Acrolinx to control the quality of the English (source) content. Authors need to use a certain authoring style (e.g. shorter sentences) and are encouraged to leverage existing paragraphs (e.g. legal disclaimers). This improves consistency in the English documentation and has the great benefit of reducing the localization workload too.

Similarly, DITA (read Reduce, Reuse and Recycle: Developing a Reuse Strategy for DITA) and Content Management Systems such as Adobe Experience Manager (formerly known as Day CQ) are designed to reuse/share content across multiple channels and publications.

Recycle

Recycling is the process of transforming existing materials (or waste) such that they could be reused again – sometimes for a totally different purpose. Creating polar fleeces from used plastic bottles or isolating walls using old denim jeans are classic examples of recycling.

Such transformations can apply to translations too. Translators don’t need to translate every sentence from scratch. Translation technologies such as Translation Memories and Machine Translation engines can help translators recycle previous translations and speed up the translation process. At Adobe, we have experienced dramatic productivity gains when we used these technologies. In general, a translator supported by these technologies will deliver in an hour what other translators would deliver in a day. These are impressive gains that contribute to localization agility too.

Rule 5 – Automate, Automate, Automate

The last requirement to achieve agile localization is automation. With agile, you can’t afford to send translation requests through e-mails or cut and paste strings from a spreadsheet to a source file. All translation hand-offs should be automated and managed through a centralized system. Over the years, Adobe’s Globalization team has developed such a platform. This system is able to connect with various source control systems, manage translation jobs, leverage existing translations across projects and content types and provide machine translation engines. In the Globalization Myth 4 article, Guta Ribeiro introduced Airport, our new system to automatically connect with our vendors and help us march towards our lofty one-hour translation goal.

 Automate

Beyond translation, it’s also important to automate other aspects of the localization process, such as build, quality assurance, bug fixing, screenshots and distribution of the localized releases. We can only go as fast as our slowest component, which is why it’s critical to automate all aspects of the localization process.

Conclusion

Localization agility can be achieved as long as all stakeholders work as a unified team. It is critical for core engineers to develop well-internationalized code from the get-go. This can be achieved via training, code reviews, usage of (Open Source) internationalization libraries and globalization report cards.

The localization process should be fully integrated within the overall development process so that all dependencies and accountabilities are clear. We recommend using Kanban boards (via tools such as Trello) to raise this visibility. To become agile, it’s also important to act “green” (i.e. reduce, reuse and recycle). Doing so represents an effective way to control waste generation before and during the localization process. Finally, all efforts should be made to automate all parts of the streamlined localization process.

At Adobe, not all localization projects are handled with great agility – yet! Some projects are more agile than others. However, based on our experience, we believe agility can be achieved by adopting these Five Golden Rules:

  1. “We are One Team!”
  2. “Internationalization is King”
  3. “Integrate Localization into the Development Process”
  4. “Reduce, Reuse and Recycle”
  5. “Automate, Automate, Automate”

 

Special thanks to Rob Jaworski, Amrit Pal Singh, Ashish Saxena, Janice Campbell,  Leandro Reis, Peter Green, Julia Feng and Quynn Le for their invaluable feedback on this article.