About

AEM Compose (AEMC) is a tool designed to solve the challenges that AEM developers face when it comes to automation. While there are some existing automation tools available in the market (like WCM.IO Ansible, AEM Design, our sunsetted CQ Chef Cookbook), they often lack the lightweight "one-click" automation feature that is critical for efficient development. AEM development is often a domain of larger companies which tend not to share their AEM-related tools publicly, which makes it harder for a shared standard to emerge. Such tools usually aim to enable the repetitive creation of an AEM environment, so that each new AEM developer or QA coming to the project can use it on their machine. This approach is much more comfortable to work with than the old approach, which typically involved a long article on some Wiki/Confluence page that quickly became out-of-date, lacked some crucial OS-specific details, and so on. Machines and automation can remember such details perfectly, while people cannot. On the other hand, Adobe focuses on AEM as a Cloud Service, which creates room for automation tools related to both local development and operations in classic AEM.

In the past

Previously, our go-to approach for complete AEM automation was to use Gradle. We have shared our impressions of this journey multiple times, including at the AdaptTo 2020 conference (AEM Developer's best friend - Gradle AEM Plugin). However, this approach proved to be less than ideal due to the steep learning curve involved with Gradle and Kotlin, which are not standard in AEM development. Gradle-based setup was primarily used for development (local) environments mainly because it has many dependencies and is heavy in general. A next step in our journey towards light-weight and simple AEM automation was Docker, but AEM's heavy-weight and mutable state made it difficult to maintain in the long run when containerized. Therefore, we've decided to lean towards Ansible. However, setting up local AEM instances using Ansible automation is doable only on a Unix OS, which is not always feasible for many AEM developers who work in corporations that primarily use Windows machines.

Challenge accepted

AEMC was created to address these challenges. By building automation using a CLI application written in the Go programming language, we were able to achieve native AEM performance. This automation runs smoothly on Unix, Mac, and Windows operating systems. Developers can streamline their development process and improve efficiency, regardless of their operating system. Additionally, we implemented an Ansible Collection based on the AEM Compose CLI tool to handle the Ansible case for higher environments.

Compared to the old approach powered by Gradle and AEM Plugin, AEMC has several key advantages. First, it offers superior runtime performance due to the lightweight nature of Go and its lack of dependencies. Setting up the tool only requires downloading a few megabytes of the binary program, rather than hundreds of JARs needed for Gradle.

The idea of using a CLI tool for AEM is not new. The AEMC project was initially inspired by aemCLI tool, but from our perspective, a few things were missing. While using a CLI tool is generally straightforward for humans, integrating it with other tools can be challenging. That's why AEMC supports multiple input and output formats (plaintext, JSON, YML) when running commands, making it easier to integrate with other DevOps tools.

Another crucial consideration is idempotency, especially when using Ansible and the Shell module, which can invoke any CLI commands. However, the problem with this module is that it lacks idempotency. Ansible will execute the command regardless of whether it has already been executed. To achieve the best possible automation that is resource and time-efficient, we need smarter tools that can determine if an action or resource is up-to-date and requires a change. Fortunately, most AEMC CLI commands are idempotent, and we can benefit from this optimization both with and without Ansible. Here are a few examples:

  • An AEM instance will not be unpacked if it is already unpacked.
  • An AEM instance will not be started if it is already running.
  • A CRX package will not be deployed if it is already deployed.
  • A JCR repository node (or OSGi configuration) will not be saved again if the desired properties are already saved on the instance.

Also, a key aspect of the AEMC tool design is the reuse of the same core automation for both developer/local environment setups and higher environment setups. This concept is illustrated in the logical diagram below:

A diagram showing how AEM Compose Core and AEM Compose CLI fit into a local and a cloud-based instance setup. The process is explained in details below.

Handling the AEM runtime can be challenging, particularly when determining the health of an instance after package deployment. Service packs and complex AEM application installations may cause unexpected instabilities, which can be a challenge for automation scripts that are implemented from scratch over and over again. By using the same core automation tool (AEM Compose Core) on a daily basis on a local instance through AEM Compose CLI, confidence can be improved when deploying code to production using AEM Compose Ansible Modules. The approach is not optimal, as managing higher environments requires the use of a different tool, typically Ansible, compared to the one used for local development, namely Task or Bash scripts. Nevertheless, this approach ensures that these two types of environments are no longer completely isolated from each other.

When to use AEM Compose

In this section, we will explore the various ways in which AEM Compose can be utilized in AEM projects. It is important to note that the numbers referenced in this section correspond to the edges depicted in the diagram above.

Developer/Local Environments

AEM developers, QA testers, technical leads, and sometimes even technical managers or business analysts may need to experiment with an AEM instance. To quickly setup AEM on local machine, the AEM Project Quickstart can be used to get started. Additionally, the next section of this blog post contains a more detailed Quickstart Guide that is worth reading.

Using the AEM Compose CLI Directly or through a Wrapper Script (1)

It is possible to run any of the available commands from the shell, such as:

  • Controlling AEM instances (sh aemw instance launch).
  • Deploying AEM packages (sh aemw package deploy --url ...).
  • Checking AEM instance status (sh aemw instance status)

However, this approach should only be used for simple, one-time operations, troubleshooting, or similar use cases. Please review the accompanying screenshots to gain a general understanding of CLI functionality:

Command - Launching instance

Command - Checking instance status

Organizing Multiple AEM Compose CLI Shell Commands into Task Files and/or Bash Scripts (2)

Simply calling the CLI directly might not be practical. As humans, we typically want to start, stop, and perform other actions on AEM instances, which can involve running multiple shell commands. To execute a series of commands, it's beneficial to organize them into tasks. Bash scripts may be sufficient for some people, but the Task tool is a lightweight and straightforward option worth exploring. It allows you to declaratively define dependencies between tasks, run them serially or in parallel, and provides other useful features. Conceptually, Task tool might be a neat and lightweight replacement for Gradle when treated as a task runner.

Consider reviewing the Taskfile defined in the automation preset app_cloud (explained later). Part of it is presented below:

aem:setup:
  desc: start and provision AEM instances then build and deploy AEM application
  cmds:
    - task: aem:start
    - task: aem:provision
    - task: aem:deploy

aem:start:
  desc: start AEM instances
  aliases: [aem:up]
  cmds:
    - sh aemw instance launch

Higher Environments

The need of setting up multiple AEM environments, including shared ones (DEV, INT, TEST), stage and production, can pose a significant challenge. Finding an efficient way to manage this problem could have a profound impact on the overall success of an AEM project.

Unfortunately, there is no one-size-fits-all solution that can accommodate all AEM project needs, and AEMC does not attempt to tackle this complex issue. However, it provides examples that can serve as a useful starting point for developing customized, project-specific automation.

For guidance on setting up an AEM environment in the AWS cloud, check out the Packer example. If you prefer simpler use cases, you can start by reviewing the Local example.

Using Ansible with AEM Compose Extension (3)

For more advanced use cases, including AEM production environment setups, it may be worth considering the usage of Ansible together with AEM Compose Ansible Collection.

Future and Custom Integrations (4)

If the Ansible tool cannot be used, it may still be beneficial to consider implementing other integrations similar to those done for Ansible but for Puppet, Terraform, etc., based on AEM Compose Core or CLI. Contributing any integration made back to the AEM Compose project on GitHub will be highly appreciated.

Using AEM Compose CLI Directly (5)

Sometimes radically simple provisioning used as a part of some other tooling in the form of shell scripting might be good enough. For such use cases, consider using AEM Compose CLI directly, for example in:

Main use case: AEM Project Archetype

Most existing projects and almost all new ones aiming to be AEMaaS-ready have source code based on the structure defined by the AEM Project Archetype. Official Adobe documentation regarding the AEM environment setup (1), (2) is in the form of a manual guide. While it ensures that a tutorial can be followed without extra tools and that each step needs to be followed and understood before the reader can proceed, it comes with its host of challenges:

  • Repeating the same steps for both author and publish instances (launching Quickstart JAR, cURL commands to deploy AEM packages, etc.)
  • Frustration caused by the need to follow the manual guide again after unsuccessful code deployments or experiments
  • No idempotence - missing a step, or executing one in the wrong order, may mean that the whole setup procedure needs to be performed again
  • All team members must perform all the steps manually, which contributes to overall time waste and increased cost of setup

That's why AEM Compose aims to automate everything to achieve the best possible developer experience. A happy developer is a productive developer! :)

Quickstart

Adding AEM Compose automation to AEM projects is very easy and consists of only three simple steps.

1. Initialize AEM project code base

This step is optional. Skip if you have an existing AEM project.

At the time of writing this post, the current archetype version is 41, and the most recent version of the AEM service pack is 6.5.16. Correct these values accordingly to the most up-to-date values at the time of reading this post. Note that sometimes the most recent service pack version may not result in building AEM application properly as Adobe is not regularly releasing API (Uber JAR) in Maven repository after each SP release. Thus, sometimes a lower version needs to be specified here.

Run the command:

mvn -B org.apache.maven.plugins:maven-archetype-plugin:3.2.1:generate \
  -D archetypeGroupId=com.adobe.aem \
  -D archetypeArtifactId=aem-project-archetype \
  -D archetypeVersion=41 \
  -D appTitle="My Site" \
  -D appId="mysite" \
  -D groupId="com.mysite"

By default, the AEM archetype generates code for cloud AEM (AEMaaCS). To generate code for classic AEM (on-prem), consider appending to command e.g -D aemVersion=6.5.15.

2. Initialize wrapper scripts

Open your terminal (Console/iTerm on Mac, Git Bash on Windows) and copy and paste the following snippet:

curl https://raw.githubusercontent.com/wttech/aemc/main/project-init.sh | sh

This command creates the AEM Compose wrapper script (aemw) in your project. Conceptually, its responsibility is the same as that of a wrapper for Maven (mvnw) or Gradle (gradlew), which installs a tool locally in a version defined in the project. This approach eliminates the need for users to install the tool widely on their OS, and allows for multiple versions of the tool, which is especially useful when switching from one AEM project to another without worrying about compatibility issues.

Quickstart - Init Wrappers

3. Initialize project files

Let's run the suggested command:

sh aemw init

The typical result should look as follows:

Quickstart - Init Project

At this stage, AEM Compose tries to detect the project type based on the file archetype.properties. Cloud AEM projects have an AEM version in date-based format, such as 2020.02.2265.20200217T222518Z-200130. Projects using classic AEM use AEM with a version starting with 6.x, such as 6.5.15. If there is no such file available in the root of the project, you need to specify the --project-kind option explicitly.

Acceptable values for --project-kind are:

  • app_cloud for AEMaaCS (202x.yy....)
  • app_classic for AEM on-prem (6.x)
  • instance

The purpose of the --project-kind option is to select the proper automation preset of files that should be unpacked in your existing AEM project. Don't worry! The idea of AEM Compose is to keep the number of files as small as possible, while still providing a fully operational AEM environment. There are meaningful differences between the file presets.

  • App presets - designed to build and deploy an AEM application (using AEM all package).

    • Cloud AEM automation preset (app_cloud) uses the AEM Dispatcher Docker image coming from the SDK.
    • Classic AEM automation preset (app_classic) builds its own AEM Dispatcher Docker image to mimic the AMS setup of an HTTPD server. It also installs an AEM service pack.
  • The last project kind (instance) is designed to only set up AEM instances (author and publish) but with no AEM dispatcher. It assumes no AEM application building and deployment. Such a project could be initialized even in an empty folder to play with any type of AEM very quickly.

That's it! A full-blown AEM environment consisting of AEM author, publish, and dispatcher is now ready to be set up. It is true from the code perspective - code changes could now be saved in VCS (Git). However, the last thing needs to be prepared. It will be discovered soon :)

AEM commands

At the beginning, let's review what automation options we have so far. In detail, what type of commands and tasks can we use now? The AEM Compose CLI allows us to do a lot of things with AEM. It can manage as many AEM instances as we want, manipulate JCR nodes and OSGi bundles from the command line, and it contains pretty complex commands that help with doing advanced AEM provisioning, like configuring Crypto keys, configuring replication agents, and more.

AEM Compose commands

Tasks

What if we want to manage the state of the entire AEM environment? In such cases, using the tasks executed by this tool may be more convenient:

Task tool commands

After reviewing and analyzing the "setup" task, which is responsible for setting up the entire AEM environment, let us proceed to run it using the following command:

sh taskw setup

However, as expected, AEMC reports a problem because we are missing the AEM source files (SDK ZIP or Quickstart JAR, SP, and license file), which need to be placed in the appropriate library directory (aem/home/lib).

Task Setup - failed

The highlighted entries in the AEM project file structure shown below are the ones that were added by running the init command (sh aemw init) with AEM Compose.

Project structure tree

The explanation of the file tree layout (Unix inspired):

  • aem/default - This is the VCS-tracked directory that contains the AEM Compose configuration.
  • aem/home - This is the VCS-ignored and IDE indexing-excluded directory that holds source AEM files, the unpacked AEM instances, downloaded AEM packages, and AEMC temporary files.
  • local.env - This is the DotEnv file with variables that are shared by both AEM Compose and Task tool. The purpose is to define AEM URLs, IP, and ports in a single place.
  • aemw and taskw - These are tool wrapper scripts used to initialize project files, see initialize project files.

Once the AEM source files are provided in the appropriate library directory, rerun the setup task. This time, we should expect the following:

  • The AEM instances will be created, started, and provisioned.
  • The AEM application will be built and deployed.
  • The AEM dispatcher will be deployed on Docker.
  • Health checks will confirm successful setup at the end.

In the beginning, the setup task automatically sets up the JDK to ensure that the correct version of Java is used when launching AEM instances. This eliminates the common issue of "works on my machine", as different developers often have different JDKs installed on their workstations. If necessary, this feature can be easily disabled to use a pre-installed Java version (e.g. Oracle). OakRun is also set up to manage AEM instances when they are offline - to update admin passwords when they are changed. Although this is quite an advanced AEM dev-ops operation, AEMC allows users to forget about such difficulties.

Task Setup - starting

The screenshot below shows that AEMC utilizes advanced AEM health checking to execute subsequent commands at the appropriate time. Furthermore, certain configuration changes require restarting instances, such as JVM options, Sling run modes, Sling properties, environment variables, and secrets. The tool operates intelligently in this regard, restarting instances only when absolutely necessary.

Task Setup - checking

Finally, after completing the setup task, our AEM environment is up and running.

Task Setup - finished

It is worth noting that essential information required to diagnose potential problems with AEM instances and dispatcher is also listed here. Health checks are performed to confirm the operational status of our AEM environment, which includes:

  • Communication between AEM author and publish instances through replication, indicating their responsiveness,
  • Accessibility of the AEM dispatcher virtual host through a domain and correct serving of AEM pages, suggesting the proper functioning of caching.

To fix or improve the health checks or tailor any other aspect of the setup process, you can easily modify the Bash/cURL commands found in the Task tool file.

This is all well and good, but what if the more advanced AEM customization is necessary? This may lead to questions such as:

  • How can we work solely with the AEM author instance?
  • How can we add an AEM publish preview instance?
  • How can we configure Sling run modes?

The answers to these questions will be addressed in the next section of this guide.

Configuration File

AEM is a complex platform and, depending on the project's use case, it often requires customization. To simplify this process, the AEMC configuration is explicitly designed, listing all the necessary options in a single YAML file, namely aem/defaults/etc/aem.yml. The following snippet shows some of the meaningful options:

# AEM instances to work with
instance:
  # Full details of local or remote instances
  config:
    local_author:
      active: [[.Env.AEM_AUTHOR_ACTIVE | default true]]
      http_url: [[.Env.AEM_AUTHOR_HTTP_URL | default "http://127.0.0.1:4502"]]
      user: [[.Env.AEM_AUTHOR_USER | default "admin"]]
      password: [[.Env.AEM_AUTHOR_PASSWORD | default "admin"]]
      run_modes: [local]
      jvm_opts:
        - -server
        - -Djava.awt.headless=true
        - -Djava.io.tmpdir=[[canonicalPath .Path "aem/home/tmp"]]
        - -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=[[.Env.AEM_AUTHOR_DEBUG_ADDR | default
          "0.0.0.0:14502" ]]
        - -Duser.language=en
        - -Duser.country=US
        - -Duser.timezone=UTC
      start_opts: []
      secret_vars:
        - ACME_SECRET=value
      env_vars:
        - ACME_VAR=value
      sling_props: []
    local_publish:
      active: [[.Env.AEM_PUBLISH_ACTIVE | default true]]
      http_url: [[.Env.AEM_PUBLISH_HTTP_URL | default "http://127.0.0.1:4503"]]
      # ...

  # State checking
  check:
    # Time to wait before first state checking (to avoid false-positives)
    warmup: 1s
    # Time to wait for next state checking
    interval: 6s
    # Number of successful check attempts that indicates end of checking
    done_threshold: 5
    # ...

  # Managed locally (set up automatically)
  local:
    # Current runtime dir (Sling launchpad, JCR repository)
    unpack_dir: 'aem/home/var/instance'
    # Archived runtime dir (AEM backup files '*.aemb.zst')
    backup_dir: 'aem/home/var/backup'

    # Source files
    quickstart:
      # AEM SDK ZIP or JAR
      dist_file: 'aem/home/lib/{aem-sdk,cq-quickstart}-*.{zip,jar}'
      # AEM License properties file
      license_file: 'aem/home/lib/license.properties'

  # CRX Package Manager
  package:
    # Force re-uploading/installing of snapshot AEM packages (just built / unreleased)
    snapshot_patterns: ['**/*-SNAPSHOT.zip']
    # Use checksums to avoid re-deployments when snapshot AEM packages are unchanged
    snapshot_deploy_skipping: true
    # ...
  # OSGi Framework
  osgi:
    # ...

java:
  # Require following versions before e.g running AEM instances
  version_constraints: '>= 11, < 12'

  # Pre-installed local JDK dir
  # a) keep it empty to download open source Java automatically for current OS and architecture
  # b) set it to absolute path or to env var '[[.Env.JAVA_HOME]]' to indicate where closed source Java like Oracle is installed
  home_dir: ''

  # Auto-installed JDK options
  download:
    # Source URL with template vars support
    url: 'https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.18%2B10/OpenJDK11U-jdk_[[.Arch]]_[[.Os]]_hotspot_11.0.18_10.[[.ArchiveExt]]'
    # ...

Expanding your AEM environment with additional instances is a simple task in AEM Compose. By copying, pasting, and modifying the relevant parts of the configuration in the YAML file, you can quickly scale your setup. AEMC offers numerous advanced options that affect the behavior of health checks, the deployment of AEM snapshot packages, and OSGi framework and JCR repository settings. Additionally, automatic JDK download is entirely customizable.

Furthermore, you have the flexibility to utilize variables from DotEnv files such as local.env at any point in the configuration file.

Conclusion

AEM Compose simplifies the process of setting up and managing AEM environments. With its streamlined approach, developers can focus on building solutions and not on the intricate details of configuring the system. In this post, we have covered the following:

  • The purpose and benefits of AEM Compose
  • How to install and set up AEM Compose
  • An overview of the AEM Compose project structure
  • How to customize AEM Compose configurations

Idempotency, both human and machine readability, and performance of the tool were key factors when building the tool, making it highly practical for use in real-world scenarios.

Next steps

In future blog posts, we will dive deeper into the features and capabilities of AEM Compose. Stay tuned!

P.S. Check out our slides and video from our talk at the AdaptTo 2023 conference :)

Thank you for investing your time in reading this article. I sincerely hope that AEM Compose will be a valuable addition to your toolkit in achieving your goals.

Robotic blue hand

Hand image by ThisisEngineering