AEM Compose is a new automation tool for AEM developers. Check out this comprehensive guide to see what it's about, why and how we built it, and what use cases it caters to. Whether you're new to AEM or an experienced user, this guide will provide you with valuable insights and practical knowledge to help you get the most out of AEM Compose.
AEM Compose (AEMC) is a tool designed to solve the challenges that AEM developers face when it comes to automation. While there are some existing automation tools available in the market (like WCM.IO Ansible, AEM Design, our sunsetted CQ Chef Cookbook), they often lack the lightweight "one-click" automation feature that is critical for efficient development. AEM development is often a domain of larger companies which tend not to share their AEM-related tools publicly, which makes it harder for a shared standard to emerge. Such tools usually aim to enable the repetitive creation of an AEM environment, so that each new AEM developer or QA coming to the project can use it on their machine. This approach is much more comfortable to work with than the old approach, which typically involved a long article on some Wiki/Confluence page that quickly became out-of-date, lacked some crucial OS-specific details, and so on. Machines and automation can remember such details perfectly, while people cannot. On the other hand, Adobe focuses on AEM as a Cloud Service, which creates room for automation tools related to both local development and operations in classic AEM.
Previously, our go-to approach for complete AEM automation was to use Gradle. We have shared our impressions of this journey multiple times, including at the AdaptTo 2020 conference (AEM Developer's best friend - Gradle AEM Plugin). However, this approach proved to be less than ideal due to the steep learning curve involved with Gradle and Kotlin, which are not standard in AEM development. Gradle-based setup was primarily used for development (local) environments mainly because it has many dependencies and is heavy in general. A next step in our journey towards light-weight and simple AEM automation was Docker, but AEM's heavy-weight and mutable state made it difficult to maintain in the long run when containerized. Therefore, we've decided to lean towards Ansible. However, setting up local AEM instances using Ansible automation is doable only on a Unix OS, which is not always feasible for many AEM developers who work in corporations that primarily use Windows machines.
AEMC was created to address these challenges. By building automation using a CLI application written in the Go programming language, we were able to achieve native AEM performance. This automation runs smoothly on Unix, Mac, and Windows operating systems. Developers can streamline their development process and improve efficiency, regardless of their operating system. Additionally, we implemented an Ansible Collection based on the AEM Compose CLI tool to handle the Ansible case for higher environments.
Compared to the old approach powered by Gradle and AEM Plugin, AEMC has several key advantages. First, it offers superior runtime performance due to the lightweight nature of Go and its lack of dependencies. Setting up the tool only requires downloading a few megabytes of the binary program, rather than hundreds of JARs needed for Gradle.
The idea of using a CLI tool for AEM is not new. The AEMC project was initially inspired by aemCLI tool, but from our perspective, a few things were missing. While using a CLI tool is generally straightforward for humans, integrating it with other tools can be challenging. That's why AEMC supports multiple input and output formats (plaintext, JSON, YML) when running commands, making it easier to integrate with other DevOps tools.
Another crucial consideration is idempotency, especially when using Ansible and the Shell module, which can invoke any CLI commands. However, the problem with this module is that it lacks idempotency. Ansible will execute the command regardless of whether it has already been executed. To achieve the best possible automation that is resource and time-efficient, we need smarter tools that can determine if an action or resource is up-to-date and requires a change. Fortunately, most AEMC CLI commands are idempotent, and we can benefit from this optimization both with and without Ansible. Here are a few examples:
- An AEM instance will not be unpacked if it is already unpacked.
- An AEM instance will not be started if it is already running.
- A CRX package will not be deployed if it is already deployed.
- A JCR repository node (or OSGi configuration) will not be saved again if the desired properties are already saved on the instance.
Also, a key aspect of the AEMC tool design is the reuse of the same core automation for both developer/local environment setups and higher environment setups. This concept is illustrated in the logical diagram below:
Handling the AEM runtime can be challenging, particularly when determining the health of an instance after package deployment. Service packs and complex AEM application installations may cause unexpected instabilities, which can be a challenge for automation scripts that are implemented from scratch over and over again. By using the same core automation tool (AEM Compose Core) on a daily basis on a local instance through AEM Compose CLI, confidence can be improved when deploying code to production using AEM Compose Ansible Modules. The approach is not optimal, as managing higher environments requires the use of a different tool, typically Ansible, compared to the one used for local development, namely Task or Bash scripts. Nevertheless, this approach ensures that these two types of environments are no longer completely isolated from each other.
In this section, we will explore the various ways in which AEM Compose can be utilized in AEM projects. It is important to note that the numbers referenced in this section correspond to the edges depicted in the diagram above.
AEM developers, QA testers, technical leads, and sometimes even technical managers or business analysts may need to experiment with an AEM instance. To quickly setup AEM on local machine, the AEM Project Quickstart can be used to get started. Additionally, the next section of this blog post contains a more detailed Quickstart Guide that is worth reading.
It is possible to run any of the available commands from the shell, such as:
- Controlling AEM instances (
sh aemw instance launch).
- Deploying AEM packages (
sh aemw package deploy --url ...).
- Checking AEM instance status (
sh aemw instance status)
However, this approach should only be used for simple, one-time operations, troubleshooting, or similar use cases. Please review the accompanying screenshots to gain a general understanding of CLI functionality:
Simply calling the CLI directly might not be practical. As humans, we typically want to start, stop, and perform other actions on AEM instances, which can involve running multiple shell commands. To execute a series of commands, it's beneficial to organize them into tasks. Bash scripts may be sufficient for some people, but the Task tool is a lightweight and straightforward option worth exploring. It allows you to declaratively define dependencies between tasks, run them serially or in parallel, and provides other useful features. Conceptually, Task tool might be a neat and lightweight replacement for Gradle when treated as a task runner.
Consider reviewing the Taskfile defined
in the automation preset
app_cloud (explained later). Part of it is presented below:
desc: start and provision AEM instances then build and deploy AEM application
- task: aem:start
- task: aem:provision
- task: aem:deploy
desc: start AEM instances
- sh aemw instance launch
The need of setting up multiple AEM environments, including shared ones (DEV, INT, TEST), stage and production, can pose a significant challenge. Finding an efficient way to manage this problem could have a profound impact on the overall success of an AEM project.
Unfortunately, there is no one-size-fits-all solution that can accommodate all AEM project needs, and AEMC does not attempt to tackle this complex issue. However, it provides examples that can serve as a useful starting point for developing customized, project-specific automation.
If the Ansible tool cannot be used, it may still be beneficial to consider implementing other integrations similar to those done for Ansible but for Puppet, Terraform, etc., based on AEM Compose Core or CLI. Contributing any integration made back to the AEM Compose project on GitHub will be highly appreciated.
Sometimes radically simple provisioning used as a part of some other tooling in the form of shell scripting might be good enough. For such use cases, consider using AEM Compose CLI directly, for example in:
Most existing projects and almost all new ones aiming to be AEMaaS-ready have source code based on the structure defined by the AEM Project Archetype. Official Adobe documentation regarding the AEM environment setup (1), (2) is in the form of a manual guide. While it ensures that a tutorial can be followed without extra tools and that each step needs to be followed and understood before the reader can proceed, it comes with its host of challenges:
- Repeating the same steps for both author and publish instances (launching Quickstart JAR, cURL commands to deploy AEM packages, etc.)
- Frustration caused by the need to follow the manual guide again after unsuccessful code deployments or experiments
- No idempotence - missing a step, or executing one in the wrong order, may mean that the whole setup procedure needs to be performed again
- All team members must perform all the steps manually, which contributes to overall time waste and increased cost of setup
That's why AEM Compose aims to automate everything to achieve the best possible developer experience. A happy developer is a productive developer! :)
Adding AEM Compose automation to AEM projects is very easy and consists of only three simple steps.
This step is optional. Skip if you have an existing AEM project.
At the time of writing this post,
the current archetype version is
the most recent version of the AEM service pack
6.5.16. Correct these values accordingly to the most up-to-date values at the time of reading this post. Note that
sometimes the most recent service pack version may not result in building AEM application properly as Adobe is not
regularly releasing API (Uber JAR) in Maven repository
after each SP release. Thus, sometimes a lower version needs to be specified here.
Run the command:
mvn -B org.apache.maven.plugins:maven-archetype-plugin:3.2.1:generate \
-D archetypeGroupId=com.adobe.aem \
-D archetypeArtifactId=aem-project-archetype \
-D archetypeVersion=41 \
-D appTitle="My Site" \
-D appId="mysite" \
By default, the AEM archetype generates code for cloud AEM (AEMaaCS). To generate code for classic AEM (on-prem),
consider appending to command e.g
Open your terminal (Console/iTerm on Mac, Git Bash on Windows) and copy and paste the following snippet:
curl https://raw.githubusercontent.com/wttech/aemc/main/project-init.sh | sh
This command creates the AEM Compose wrapper script (aemw) in your project. Conceptually, its responsibility is the same as that of a wrapper for Maven (mvnw) or Gradle (gradlew), which installs a tool locally in a version defined in the project. This approach eliminates the need for users to install the tool widely on their OS, and allows for multiple versions of the tool, which is especially useful when switching from one AEM project to another without worrying about compatibility issues.
Let's run the suggested command:
sh aemw init
The typical result should look as follows:
At this stage, AEM Compose tries to detect the project type based on the file
Cloud AEM projects have an AEM version in date-based format, such as
using classic AEM use AEM with a version starting with
6.x, such as
6.5.15. If there is no such file available in
the root of the project, you need to specify the
--project-kind option explicitly.
Acceptable values for
app_cloudfor AEMaaCS (
app_classicfor AEM on-prem (
The purpose of the
--project-kind option is to select the proper automation preset of files that should be unpacked in
your existing AEM project. Don't worry! The idea of AEM Compose is to keep the number of files as small as possible,
while still providing a fully operational AEM environment. There are meaningful differences between the file presets.
App presets - designed to build and deploy an AEM application (using AEM all package).
- Cloud AEM automation preset (
app_cloud) uses the AEM Dispatcher Docker image coming from the SDK.
- Classic AEM automation preset (
app_classic) builds its own AEM Dispatcher Docker image to mimic the AMS setup of an HTTPD server. It also installs an AEM service pack.
- Cloud AEM automation preset (
- The last project kind (
instance) is designed to only set up AEM instances (author and publish) but with no AEM dispatcher. It assumes no AEM application building and deployment. Such a project could be initialized even in an empty folder to play with any type of AEM very quickly.
That's it! A full-blown AEM environment consisting of AEM author, publish, and dispatcher is now ready to be set up. It is true from the code perspective - code changes could now be saved in VCS (Git). However, the last thing needs to be prepared. It will be discovered soon :)
At the beginning, let's review what automation options we have so far. In detail, what type of commands and tasks can we use now? The AEM Compose CLI allows us to do a lot of things with AEM. It can manage as many AEM instances as we want, manipulate JCR nodes and OSGi bundles from the command line, and it contains pretty complex commands that help with doing advanced AEM provisioning, like configuring Crypto keys, configuring replication agents, and more.
What if we want to manage the state of the entire AEM environment? In such cases, using the tasks executed by this tool may be more convenient:
After reviewing and analyzing the "setup" task, which is responsible for setting up the entire AEM environment, let us proceed to run it using the following command:
sh taskw setup
However, as expected, AEMC reports a problem because we are missing the AEM source files (SDK ZIP or Quickstart JAR, SP, and license file), which need to be placed in the appropriate library directory (aem/home/lib).
The highlighted entries in the AEM project file structure shown below are the ones that were added by running the init
sh aemw init) with AEM Compose.
The explanation of the file tree layout (Unix inspired):
- aem/default - This is the VCS-tracked directory that contains the AEM Compose configuration.
- aem/home - This is the VCS-ignored and IDE indexing-excluded directory that holds source AEM files, the unpacked AEM instances, downloaded AEM packages, and AEMC temporary files.
- local.env - This is the DotEnv file with variables that are shared by both AEM Compose and Task tool. The purpose is to define AEM URLs, IP, and ports in a single place.
- aemw and taskw - These are tool wrapper scripts used to initialize project files, see initialize project files.
Once the AEM source files are provided in the appropriate library directory, rerun the setup task. This time, we should expect the following:
- The AEM instances will be created, started, and provisioned.
- The AEM application will be built and deployed.
- The AEM dispatcher will be deployed on Docker.
- Health checks will confirm successful setup at the end.
In the beginning, the setup task automatically sets up the JDK to ensure that the correct version of Java is used when launching AEM instances. This eliminates the common issue of "works on my machine", as different developers often have different JDKs installed on their workstations. If necessary, this feature can be easily disabled to use a pre-installed Java version (e.g. Oracle). OakRun is also set up to manage AEM instances when they are offline - to update admin passwords when they are changed. Although this is quite an advanced AEM dev-ops operation, AEMC allows users to forget about such difficulties.
The screenshot below shows that AEMC utilizes advanced AEM health checking to execute subsequent commands at the appropriate time. Furthermore, certain configuration changes require restarting instances, such as JVM options, Sling run modes, Sling properties, environment variables, and secrets. The tool operates intelligently in this regard, restarting instances only when absolutely necessary.
Finally, after completing the setup task, our AEM environment is up and running.
It is worth noting that essential information required to diagnose potential problems with AEM instances and dispatcher is also listed here. Health checks are performed to confirm the operational status of our AEM environment, which includes:
- Communication between AEM author and publish instances through replication, indicating their responsiveness,
- Accessibility of the AEM dispatcher virtual host through a domain and correct serving of AEM pages, suggesting the proper functioning of caching.
To fix or improve the health checks or tailor any other aspect of the setup process, you can easily modify the Bash/cURL commands found in the Task tool file.
This is all well and good, but what if the more advanced AEM customization is necessary? This may lead to questions such as:
- How can we work solely with the AEM author instance?
- How can we add an AEM publish preview instance?
- How can we configure Sling run modes?
The answers to these questions will be addressed in the next section of this guide.
AEM is a complex platform and, depending on the project's use case, it often requires customization. To simplify this process, the AEMC configuration is explicitly designed, listing all the necessary options in a single YAML file, namely aem/defaults/etc/aem.yml. The following snippet shows some of the meaningful options:
# AEM instances to work with
# Full details of local or remote instances
active: [[.Env.AEM_AUTHOR_ACTIVE | default true]]
http_url: [[.Env.AEM_AUTHOR_HTTP_URL | default "http://127.0.0.1:4502"]]
user: [[.Env.AEM_AUTHOR_USER | default "admin"]]
password: [[.Env.AEM_AUTHOR_PASSWORD | default "admin"]]
- -Djava.io.tmpdir=[[canonicalPath .Path "aem/home/tmp"]]
- -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=[[.Env.AEM_AUTHOR_DEBUG_ADDR | default
active: [[.Env.AEM_PUBLISH_ACTIVE | default true]]
http_url: [[.Env.AEM_PUBLISH_HTTP_URL | default "http://127.0.0.1:4503"]]
# State checking
# Time to wait before first state checking (to avoid false-positives)
# Time to wait for next state checking
# Number of successful check attempts that indicates end of checking
# Managed locally (set up automatically)
# Current runtime dir (Sling launchpad, JCR repository)
# Archived runtime dir (AEM backup files '*.aemb.zst')
# Source files
# AEM SDK ZIP or JAR
# AEM License properties file
# CRX Package Manager
# Force re-uploading/installing of snapshot AEM packages (just built / unreleased)
# Use checksums to avoid re-deployments when snapshot AEM packages are unchanged
# OSGi Framework
# Require following versions before e.g running AEM instances
version_constraints: '>= 11, < 12'
# Pre-installed local JDK dir
# a) keep it empty to download open source Java automatically for current OS and architecture
# b) set it to absolute path or to env var '[[.Env.JAVA_HOME]]' to indicate where closed source Java like Oracle is installed
# Auto-installed JDK options
# Source URL with template vars support
Expanding your AEM environment with additional instances is a simple task in AEM Compose. By copying, pasting, and modifying the relevant parts of the configuration in the YAML file, you can quickly scale your setup. AEMC offers numerous advanced options that affect the behavior of health checks, the deployment of AEM snapshot packages, and OSGi framework and JCR repository settings. Additionally, automatic JDK download is entirely customizable.
AEM Compose simplifies the process of setting up and managing AEM environments. With its streamlined approach, developers can focus on building solutions and not on the intricate details of configuring the system. In this post, we have covered the following:
- The purpose and benefits of AEM Compose
- How to install and set up AEM Compose
- An overview of the AEM Compose project structure
- How to customize AEM Compose configurations
Idempotency, both human and machine readability, and performance of the tool were key factors when building the tool, making it highly practical for use in real-world scenarios.
In future blog posts, we will dive deeper into the features and capabilities of AEM Compose. Stay tuned!
Thank you for investing your time in reading this article. I sincerely hope that AEM Compose will be a valuable addition to your toolkit in achieving your goals.
Hand image by ThisisEngineering
Hero image by Patrick Fore, opens in a new window
Featured image by Patrick Fore, opens in a new window