AEM Rockstar Tips & Tricks Blog Series

Posted on Monday, March 20, 2017 By

Our First AEM Rockstar Tips & Tricks Guest Blog!

Late last year we launched our first AEM Rockstar competition, where we asked for tips & tricks from you, the Rockstar AEM marketers and technologists who create and innovate within Adobe Experience Manager. Winners won a free pass to Summit and will present their tips & tricks to an audience of over 500 attendees and a panel of judges, who will award the best tip & trick winner with a cool prize! 

Thanks again to all the Rockstars who competed this year. If you are at Summit and want to attend the Rockstar session, on Wednesday, March 22, click here:  http://bit.ly/2kbauhJ!   We’ll see you at Summit!

We were rocked by the response to the Rockstar contest, but ultimately we could only choose five finalists to present at the 2017 Adobe Summit. However, we received so many impressive submissions from the AEM community that we decided to feature a series of guest blog posts, written by the runners-up. These tips & tricks range from favorite ways to use Forms, to more technical examples that can help you better synchronize content, use new tools, and much more.

Our first guest post comes from Sagar Sane, who is a Technical Architect at iCiDIGITAL (www.icidigital.com), a digital agency that specializes in AEM. He has over four years of experience working with AEM / CQ5; primarily focused on server-side development and integrations of AEM with other systems, and over five years of web development experience. He is based at the iCiDIGITAL location in Raleigh, North Carolina.

Grabbit – A Rockstar  Tool for Content Sync in AEM

Suppose you have an enterprise-scale AEM implementation with an author and multiple publishers in production. There is a staging environment mirroring closely to production from an infrastructure point of view, and you might even have development and UAT environments used for development and testing, respectively. The business stakeholders of your AEM implementation prefer that you keep the content synchronized as much as possible between these environments. Generally, you would use tools like Package Manager or vlt rcp to achieve this.

However, these tools do not scale well enough for large implementations where your AEM environments are geographically far away, or that have very large amounts of content to be synchronized. At the core, both these tools use the WebDAV protocol. WebDAV uses XML for serialization and deserialization and uses HTTP handshakes for every node that is synchronized between two AEM instances. This means that any latency on the network will hurt the content sync performance, in terms of time and space efficiency.

Grabbit: A Different Approach

The name Grabbit refers to this grabbing of content from one CQ/AEM instance and copying it to another. However, it also refers to Jackrabbit, the reference JCR 283 implementation that the content is being copied to and from. Grabbit is installed as any other AEM application and is available on bintray for anyone to download.

Grabbit takes a different approach to solve the problem of content synchronization. Grabbit’s goal is to make it easier and faster to sync content from one AEM instance (Grabbit Server) to another (Grabbit Client). Grabbit was developed as a content sync solution for one of our clients, with iCiDIGITAL members the primary contributors. Upon the success of the initial releases of the tool, Grabbit was ultimately open sourced by our client couple years ago.

How does it Work?

Unlike the other tools mentioned above that use HTTP handshakes for each node to be synced, Grabbit creates a stream of data over HTTP(s) from the Grabbit Server (source) to the Grabbit Client (destination). For serialization, Grabbit uses Google’s Protocol Buffers. Better than XML- based serialization, Protocol Buffers is extremely space efficient in transferring the data over the network. As its core underlying technology, Grabbit uses Spring Batch. Some of the core features of Spring Batch that Grabbit uses are Chunk-Based processing, Declarative I/O, start/stop a job and job processing statistics.

Grabbit needs to be installed on both the AEM instances; the instance that requests content to be copied to (the Grabbit Client) and the other one from which the content is copied from (the Grabbit Server). You initiate the request to copy content from Grabbit Server to Grabbit Client by providing a Configuration File to the client. For each path in the configuration file that needs to be synced, a new Grabbit Job is created on the Grabbit Client. Each job then opens only one HTTP(s) connection to the server and uses Protocol Buffers for data transfer marshaling / unmarshalling. There are several configuration parameters that can be set in the configuration file. You can find details about that here.

Grabbit also allows you to monitor the jobs on the client. The monitoring API provides basic information like the job start time/end time, current path, the number of nodes written, etc. Under the covers, Grabbit uses Spring Batch’s querying features for this. The job status is represented in JSON format as below:

To summarize, the basic sequence of Grabbit usage can be illustrated as below:

Why Use Grabbit?

Grabbit is supported for most of the newer AEM versions and is under active development. We have already received and continue to receive a lot of feedback to improve Grabbit. In our experience, Grabbit has proved to be 2 to 10 times faster than other content sync alternatives in AEM, depending on the network conditions and the amount of content being synchronized. I encourage the users of Grabbit to submit issues, questions, or comments on GitHub. We hope that you give Grabbit a shot and that it proves to be useful for everyone in the AEM Community. Post a comment if you have any questions!

1:18 PM Permalink

Do you consider yourself an AEM Rockstar?

Posted on Monday, November 7, 2016 By

Its time to make your AEM knowledge get you the fame and recognition you deserve!


 

aem_rockstar

EDIT (1/23/2017) : The submission for 2017 AEM Rockstar is now closed. If you have submitted an entry, we will soon be in touch. If you missed out, please consider submitting next year. Please register for the session at Summit to see the top 5 tip. Session details are http://bit.ly/2kbauhJ!

At Adobe Summit 2017, we’re introducing a brand new session where you get to be a rock star! We’re looking for the top tips & tricks from AEM marketers and technologists that we can feature in a fun, fast-paced, and informative breakout session.

Do you want to be on a Summit stage and share your own AEM tip? If you’re chosen as one of our presenters, you’ll receive a free pass (valued at $1895) to Summit on March 19-23, 2017 in Las Vegas*, and the opportunity to win cool prizes like a Drone, Surface Book, or who knows what! And don’t forget about the bragging rights.

To be considered, all you have to do is submit your tip using this online form. We’re looking for all things AEM. Sites, Assets, Mobile, Forms, anything and EVERYTHING! We want tips that you believe will show your fellow AEM marketers and technologists new things about AEM and help them to do their jobs more effectively and efficiently. Ultimately, your judges will be the session attendees, including leaders from the AEM Product and Marketing teams, as they vote for their favorite tips. Don’t be on the fence about submitting a tip — you might be surprised how useful things can be to others.

We’ll be screening your tips based on how innovative, practical, and valuable they are as well as how broadly they could be used by AEM marketers and technologists at other companies in different industries. We’ll select up to 5 finalists who will come to Las Vegas and present their tip to an audience. This is a great opportunity to attend Summit and showcase your talents to the world!

We are giving away some swag to everyone who submits an idea as well. Every person who submits an idea gets an AEM Rockstar sticker!

aem-rockstar-sticker

All submissions must be submitted by 1/22/2017. Soon after that we’ll reach out to all the applicants with next steps.

Go to AEM Rockstars Application Form

*You will be responsible for travel expenses. 
12:15 PM Permalink

config vs cfg – Battle royale

Posted on Thursday, June 9, 2016 By

If you’ve been around AEM (and CQ) long enough, you’ve probably seen OSGi configurations serialized out in the form of a file. Its the easiest way to pre-configure various services and components in AEM. A common example is configuring a node and/or data store.

https://docs.adobe.com/docs/en/aem/6-2/deploy/platform/data-store-config.html

You’ll often find examples listing a files that have .cfg or .config extensions. Which leads me to the question:

Which format is the right one?

That is a bit tougher to answer. They both share generally the same patterns but have a few subtle differences.

CFG format:

  • Standard properties file format (java.util.Property)
  • Comments allowed
  • Only string values supported
    • A class name PropertiesUtil can convert (cast) values to their proper types. All common configurable classes available (OOTB) should use this utility and therefore should not be a big issue.

CONFIG format:

  • Introduced in the Felix framework in 2010 (see: https://issues.apache.org/jira/browse/FELIX-2513)
    • But not used directly by Sling until later (2011-ish+); CQ/AEM uses the Sling Launchpad Installer vs. the Felix FileInstall.
  • Supports typed values (boolean, Int, Float, Char, etc)
  • first line can be a comment (#example), but inline comments are not allowed
  • Various things need escaping
  • Multivalue properties are easier

A good explanation of the format can be found here:
https://sling.apache.org/documentation/bundles/configuration-installer-factory.html

The configuration handler code which sets up the formatting:
https://github.com/apache/felix/blob/trunk/configadmin/src/main/java/org/apache/felix/cm/file/ConfigurationHandler.java

You can see examples of the CONFIG format persisted under the <install dir>/crx-quickstart/launchpad/config in a typical AEM instance.

Conclusion

The good news is that both formats are functional and supported. You don’t have to choose one versus the other. Although I’d advise against doing both interchangeably. Therefore, the guidance that I’d suggest is:

If you’re starting a new deployment/implementation, use CONFIG. Its the newer standard and provides more flexibility with regards to property config design.

If you’ve already deployed and have existing CFG files, stick with CFG. There appear to be no plans to remove/deprecate this format.

10:31 AM Permalink

AEM 6 – migrating to FileDataStore from embedded segmentstore binaries

Posted on Wednesday, March 16, 2016 By

Have you ever come into a situation where you’re up and running on AEM 6 using the default setup with the segment store containing everything – content & large binaries – and you wish you would have thought to split them out into a FileDataStore (or S3) after the fact? Most of the time you probably have architected the solution this way from the start. However, there are probably times you want to eek out every bit of performance optimizations that you can after initial rollout.

Enter oak-upgrade (and its Adobe sibling, crx2oak)

Background

As most should know already, the default Oak architecture you get with AEM 6 stores the binary content (files like images, videos, pdfs, etc) within the repository tar files. Look in crx-quickstart/repository/segmentstore for examples or read up on the structure here: http://jackrabbit.apache.org/oak/docs/nodestore/segment/tar.html or check out Francesco Mari’s analysis here: http://francescomari.github.io/2015/05/04/looking-into-tar-files-in-oak.html.

Now for most Sites use cases where your authors create alot of content and occasionally upload a pdf or image, this setup is just fine. However, if you want the best performance and the ability to share your data store with other instances, then you should go with an external data store like FileDataStore which we’ll focus on here.

Getting Started

disclaimer: not responsible for any damage you cause to your instances. Please test thoroughly on non-production instances before considering attempting this on a working production instance.

Before we start, you should update your Oak version on AEM 6 to the latest Adobe supplied version. As of this writing, oak-upgrade (1.6) needed either 1.0.12+ (AEM 6.0) or 1.1.7+ (AEM6.1) to migrate segment format V_11 repositories. This is important.

You’re going to need alot of disk capacity during the migration as the process creates a new datastore directory with all the data as well as a new repository directory and segment store. You are doubling your disk needs temporarily as the end result is basically two copies of your repo.

Depending on the size of your repo, the migration may take a considerable amount of time. In addition to the binary data being copied, some of the Oak indexes will most likely need to be re-indexed (automatically).

The Steps

Backup all data before starting.

  1. Download the latest version of oak-upgrade from Apache. I built my version (1.6-SNAPSHOT) from source using maven.
  2. Create a parallel directory to house your new repository directory (eg. “repository_new”)
  3. Copy your oak-upgrade jar to the base install directory – parallel to the main aem quickstart jar.
  4. Run this command:
    java -jar oak-upgrade.jar –copy-binaries –src-datastore=crx-quickstart/repository/segmentstore –datastore=crx-quickstart/datastore crx-quickstart/repository crx-quickstart/repository_new –mmap[your paths may vary]
  5. The output of this command should be bunch of RepositorySidegrade INFO messages and a new repo directory with a segmentstore with tar files and a datastore directory with a bunch of two character long directories.
  6. Rename the existing repository directory to “repository.migrated” (or something else).
  7. Rename the newly populated repository to “repository” from whatever you set in step 4.
    Note: I realize that you can point AEM to whatever repository.home you wish via OSGi configuration, but this just seems easier (and faster).
  8. Create a text file named org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore.cfg in crx-quickstart/install. Add the following content to the file:
    path=crx-quickstart/datastore

    (or whatever you set your path to)

  9. Create a text file named org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.cfg in crx-quickstart/install. Add the following content to the file (Note: this is also where you can change the repository.home path (not shown here)):
    customBlobStore=true
  10. Start up AEM like usual and monitor error.log for any anomalies.

Conclusion

I ran the migration on a pretty vanilla AEM 6 (Oak 1.0.27) instance (with OOTB content) and the whole process went very quickly. The oak-upgrade command took less than 30 seconds. Startup took around 2 minutes to complete and I have yet to find anything wrong with the new system.

 

9:27 AM Permalink

MongoDB + AEM Production Servers

Posted on Tuesday, September 16, 2014 By

 

I response to my previous post, here is the basics of a sharded cluster MongoDB setup. As you can see, if you were to place each element on its own server you’d end up with alot of servers.

Sharded Mongo

After a bit of research into how each mongo element is used and its performance characteristics, I arrived at the following sleek(er) configuration of servers. This is arrangement is symmetric to allow for cross data center architectures.

Mongo-Production

Servers

This post was heavily influenced by a concise Will Burton post: http://two4seven.me/sharding/

12:48 PM Permalink