Merge pull request #177 from wttech/pkg-install-logged

Package install with HTML reports

README.md

@@ -40,9 +40,11 @@ AEMC is a versatile tool for managing Adobe Experience Manager (AEM) instances.
     * [Improving performance](#improving-performance)
     * [Increasing verbosity](#increasing-verbosity)
     * [Installing content packages](#installing-content-packages)
+    * [Installing packages with troubleshooting](#installing-packages-with-troubleshooting)
 * [Examples](#examples)
-  * ['SSL by Default' support](#ssl-by-default-support)
-  * [Global Trust Store management support](#global-trust-store-management-support)
+  * [Replication agents](#replication-agents)
+  * [SSL by Default](#ssl-by-default)
+  * [Global Trust Store](#global-trust-store)
 * [Contributing](#contributing)
 * [Authors](#authors)
 * [License](#license)
@@ -366,6 +368,17 @@ instance:
     snapshot_deploy_skipping: true
     # Disable following workflow launchers for a package deployment time only
     toggled_workflows: [/libs/settings/workflow/launcher/config/asset_processing_on_sdk_*,/libs/settings/workflow/launcher/config/update_asset_*,/libs/settings/workflow/launcher/config/dam_*]
+    # Also sub-packages
+    install_recursive: true
+    # Use slower HTML endpoint for deployments but with better troubleshooting
+    install_html:
+      enabled: false
+      # Print HTML directly to console instead of writing to file
+      console: false
+      # Fail on case 'installed with errors'
+      strict: true
+      # Directory to place HTML report files
+      dir: aem/home/var/log/package/install
 
   # OSGi Framework
   osgi:
@@ -490,7 +503,7 @@ By default, the timeout is set to `10m`, but you have the option to increase it
 To set the timeout for a single AEMC command, use the following syntax:
 
 ```shell
-AEM_INSTANCE_HTTP_TIMEOUT=0 sh aemw package deploy --url my-package.zip
+AEM_INSTANCE_HTTP_TIMEOUT=0 sh aemw package deploy --file my-package.zip
 ```
 
 It's important to be aware that AEMaaCS also has its own timeout for requests made to the Package Manager UI. For detailed information, please refer to the [documentation](https://experienceleague.adobe.com/docs/experience-manager-cloud-service/content/implementing/developer-tools/package-manager.html?lang=en#aemaacs-packages).
@@ -500,14 +513,63 @@ To skip the instance health check after running a command that would normally tr
 To skip the instance health check for a single AEMC command, use the following syntax:
 
 ```shell
-AEM_INSTANCE_CHECK_SKIP=true sh aemw package deploy --url my-package.zip
+AEM_INSTANCE_CHECK_SKIP=true sh aemw package deploy --file my-package.zip
 ```
 
-# Examples
+### Installing packages with troubleshooting
+
+Starting from version 1.4.0 (see [#177](https://github.com/wttech/aemc/pull/177)), AEMC now supports AEM package installations using an HTML report serving endpoint, similar to CRX Package Manager's. While this method may result in slightly extended installation times, it provides a comprehensive HTML report detailing the package installation process.
+
+This new feature offers two distinct modes for leveraging its benefits:
+
+1. Saving HTML report to file:
+
+   ```shell
+   AEM_INSTANCE_PACKAGE_INSTALL_HTML_ENABLED=true sh aemw package deploy --file my-package.zip
+   ```
 
-## 'SSL by Default' support
+2. Direct console output of HTML report:
 
-AEM Compose supports 'SSL by Default' feature of AEM.
+   ```shell
+   AEM_INSTANCE_PACKAGE_INSTALL_HTML_ENABLED=true AEM_INSTANCE_PACKAGE_INSTALL_HTML_CONSOLE=true sh aemw package deploy --file my-package.zip
+   ```
+
+# Examples
+
+## Replication agents
+
+1. Configuring publish agent on AEM author:
+
+    ```shell
+    PROPS="
+    enabled: true
+    transportUri: http://localhost:4503/bin/receive?sling:authRequestLogin=1
+    transportUser: admin
+    transportPassword: admin
+    userId: admin
+    "
+    echo "$PROPS" | sh aemw repl agent setup -A --location "author" --name "publish"
+    ```
+
+2. Configuring flush agent on AEM publish:
+
+    ```shell
+    PROPS="
+    enabled: true
+    transportUri: http://localhost/dispatcher/invalidate.cache
+    protocolHTTPHeaders:
+    - 'CQ-Action: {action}'
+    - 'CQ-Handle: {path}'
+    - 'CQ-Path: {path}'
+    - 'Host: flush'
+    "
+    echo "$PROPS" | sh aemw repl agent setup -P --location "publish" --name "flush"
+    ```
+   If needed, update `localhost` to the value on which AEM dispatcher is available, e.g.`localhost:8080`.
+
+## SSL by Default
+
+AEM Compose supports *SSL by Default* feature of AEM.
 
 This feature requires:
 - certificate file in PEM format
@@ -517,7 +579,7 @@ This feature requires:
 - hostname for HTTPS connector (used by AEM to check if the setup was successful; has to be reachable by AEM)
 - port for HTTPS connector
 
-To set up 'SSL by Default', run:
+To set up *SSL by Default*, run:
 
 ```shell
 sh aemw ssl setup \
@@ -545,36 +607,39 @@ See the reference documentation: [AEM 6.5 > Administering Guide > SSL by Default
 
 For local environment remember to set different port numbers for author and publish instances.
 
-## Global Trust Store management support
+## Global Trust Store
 
 AEM Compose supports managing the trust store of AEM instances.
 
 This feature supports:
+
 - creation of the general trust store if it does not exist
-```shell
-sh aemw gts create --password PASSWORD_HERE -A
-```
+    ```shell
+    sh aemw gts create --password PASSWORD_HERE -A
+    ```
+
 - getting the general trust store status
-```shell
-sh aemw gts status -A
-```
+    ```shell
+    sh aemw gts status -A
+    ```
 
 - adding a certificate to the general trust store
-```shell
-sh aemw gts certificate add --path <path> -A
-```
+    ```shell
+    sh aemw gts certificate add --path <path> -A
+    ```
+
 This command will add a certificate to the general trust store only if not exists in trust store and will return the alias of the certificate.
 Command `certificate add` supports certificates in PEM and DER formats.
 
 - reading a certificate from the general trust store (by alias)
-```shell
-sh aemw gts certificate read --alias <alias> -A
-```
+    ```shell
+    sh aemw gts certificate read --alias <alias> -A
+    ```
 
 - removing a certificate from the general trust store (by alias)
-```shell
-sh aemw gts certificate remove --alias <alias> -A
-```
+    ```shell
+    sh aemw gts certificate remove --alias <alias> -A
+    ```
 
 # Contributing
 

examples/docker/src/aem/default/etc/aem.yml

@@ -139,6 +139,17 @@ instance:
     snapshot_deploy_skipping: true
     # Disable following workflow launchers for a package deployment time only
     toggled_workflows: [/libs/settings/workflow/launcher/config/asset_processing_on_sdk_*,/libs/settings/workflow/launcher/config/update_asset_*,/libs/settings/workflow/launcher/config/dam_*]
+    # Also sub-packages
+    install_recursive: true
+    # Use slower HTML endpoint for deployments but with better troubleshooting
+    install_html:
+      enabled: false
+      # Print HTML directly to console instead of writing to file
+      console: false
+      # Fail on case 'installed with errors'
+      strict: true
+      # Directory to place HTML report files
+      dir: aem/home/var/log/package/install
 
   # OSGi Framework
   osgi:

pkg/cfg/defaults.go

@@ -81,6 +81,12 @@ func (c *Config) setDefaults() {
 	v.SetDefault("instance.status.timeout", time.Millisecond*500)
 
 	v.SetDefault("instance.package.install_recursive", true)
+
+	v.SetDefault("instance.package.install_html.enabled", false)
+	v.SetDefault("instance.package.install_html.strict", true)
+	v.SetDefault("instance.package.install_html.console", false)
+	v.SetDefault("instance.package.install_html.dir", common.LogDir+"/package/install")
+
 	v.SetDefault("instance.package.snapshot_deploy_skipping", true)
 	v.SetDefault("instance.package.snapshot_patterns", []string{"**/*-SNAPSHOT.zip"})
 	v.SetDefault("instance.package.toggled_workflows", []string{})

pkg/instance.go

@@ -316,5 +316,4 @@ func (i Instance) LockDir() string {
 	}
 	log.Panicf("%s > lock files for remote instances are not yet supported", i.ID())
 	return "" // TODO dir should reflect url or name? or configurable? for remote instances and features that are locked via local files like: pkg deploy skipping, SSL, ...
-
 }

pkg/package_manager.go

@@ -1,6 +1,7 @@
 package pkg
 
 import (
+	"bufio"
 	"fmt"
 	"github.com/samber/lo"
 	log "github.com/sirupsen/logrus"
@@ -9,8 +10,11 @@ import (
 	"github.com/wttech/aemc/pkg/common/osx"
 	"github.com/wttech/aemc/pkg/common/pathx"
 	"github.com/wttech/aemc/pkg/common/stringsx"
+	"github.com/wttech/aemc/pkg/common/timex"
 	"github.com/wttech/aemc/pkg/pkg"
+	"os"
 	"path/filepath"
+	"strings"
 	"time"
 )
 
@@ -19,6 +23,10 @@ type PackageManager struct {
 
 	SnapshotDeploySkipping bool
 	InstallRecursive       bool
+	InstallHTMLEnabled     bool
+	InstallHTMLDir         string
+	InstallHTMLConsole     bool
+	InstallHTMLStrict      bool
 	SnapshotPatterns       []string
 	ToggledWorkflows       []string
 }
@@ -30,6 +38,10 @@ func NewPackageManager(res *Instance) *PackageManager {
 		instance: res,
 
 		SnapshotDeploySkipping: cv.GetBool("instance.package.snapshot_deploy_skipping"),
+		InstallHTMLEnabled:     cv.GetBool("instance.package.install_html.enabled"),
+		InstallHTMLDir:         cv.GetString("instance.package.install_html.dir"),
+		InstallHTMLConsole:     cv.GetBool("instance.package.install_html.console"),
+		InstallHTMLStrict:      cv.GetBool("instance.package.install_html.strict"),
 		InstallRecursive:       cv.GetBool("instance.package.install_recursive"),
 		SnapshotPatterns:       cv.GetStringSlice("instance.package.snapshot_patterns"),
 		ToggledWorkflows:       cv.GetStringSlice("instance.package.toggled_workflows"),
@@ -188,6 +200,13 @@ func (pm *PackageManager) Upload(localPath string) (string, error) {
 }
 
 func (pm *PackageManager) Install(remotePath string) error {
+	if pm.InstallHTMLEnabled {
+		return pm.installHTML(remotePath)
+	}
+	return pm.installJSON(remotePath)
+}
+
+func (pm *PackageManager) installJSON(remotePath string) error {
 	log.Infof("%s > installing package '%s'", pm.instance.ID(), remotePath)
 	response, err := pm.instance.http.Request().
 		SetFormData(map[string]string{"cmd": "install", "recursive": fmt.Sprintf("%v", pm.InstallRecursive)}).
@@ -199,7 +218,7 @@ func (pm *PackageManager) Install(remotePath string) error {
 	}
 	var status pkg.CommandResult
 	if err = fmtx.UnmarshalJSON(response.RawBody(), &status); err != nil {
-		return fmt.Errorf("%s > cannot install package '%s'; cannot parse response: %w", pm.instance.ID(), remotePath, err)
+		return fmt.Errorf("%s > cannot install package '%s'; cannot parse JSON response: %w", pm.instance.ID(), remotePath, err)
 	}
 	if !status.Success {
 		return fmt.Errorf("%s > cannot install package '%s'; unexpected status: %s", pm.instance.ID(), remotePath, status.Message)
@@ -208,6 +227,74 @@ func (pm *PackageManager) Install(remotePath string) error {
 	return nil
 }
 
+func (pm *PackageManager) installHTML(remotePath string) error {
+	log.Infof("%s > installing package '%s'", pm.instance.ID(), remotePath)
+
+	response, err := pm.instance.http.Request().
+		SetFormData(map[string]string{"cmd": "install", "recursive": fmt.Sprintf("%v", pm.InstallRecursive)}).
+		Post(ServiceHtmlPath + remotePath)
+	if err != nil {
+		return fmt.Errorf("%s > cannot install package '%s': %w", pm.instance.ID(), remotePath, err)
+	} else if response.IsError() {
+		return fmt.Errorf("%s > cannot install package '%s': '%s'", pm.instance.ID(), remotePath, response.Status())
+	}
+
+	success := false
+	successWithErrors := false
+
+	htmlFilePath := fmt.Sprintf("%s/%s/%s-%s.html", pm.InstallHTMLDir, pm.instance.ID(), filepath.Base(remotePath), timex.FileTimestampForNow())
+	var htmlWriter *bufio.Writer
+
+	if !pm.InstallHTMLConsole {
+		if err := pathx.Ensure(filepath.Dir(htmlFilePath)); err != nil {
+			return err
+		}
+		htmlFile, err := os.OpenFile(htmlFilePath, os.O_RDWR|os.O_CREATE, 0666)
+		if err != nil {
+			return fmt.Errorf("%s > cannot install package '%s': cannot open HTML report file '%s'", pm.instance.ID(), remotePath, htmlFilePath)
+		}
+		defer htmlFile.Close()
+		htmlWriter = bufio.NewWriter(htmlFile)
+		defer htmlWriter.Flush()
+	}
+
+	scanner := bufio.NewScanner(response.RawBody())
+	for scanner.Scan() {
+		htmlLine := scanner.Text()
+		if !success && strings.Contains(htmlLine, pkg.InstallSuccess) {
+			success = true
+		}
+		if !successWithErrors && strings.Contains(htmlLine, pkg.InstallSuccessWithErrors) {
+			successWithErrors = true
+		}
+		if !pm.InstallHTMLConsole {
+			_, err := htmlWriter.WriteString(htmlLine + osx.LineSep())
+			if err != nil {
+				return fmt.Errorf("%s > cannot install package '%s': cannot write to HTML report file '%s'", pm.instance.ID(), remotePath, htmlFilePath)
+			}
+		} else {
+			fmt.Println(htmlLine)
+		}
+	}
+	if err := scanner.Err(); err != nil {
+		return fmt.Errorf("%s > cannot install package '%s': cannot parse HTML response: %w", pm.instance.ID(), remotePath, err)
+	}
+
+	failure := !success && !successWithErrors
+	if failure || (successWithErrors && pm.InstallHTMLStrict) {
+		if pm.InstallHTMLConsole {
+			return fmt.Errorf("%s > cannot install package '%s': HTML output contains errors", pm.instance.ID(), remotePath)
+		}
+		return fmt.Errorf("%s > cannot install package '%s': HTML report contains errors '%s'", pm.instance.ID(), remotePath, htmlFilePath)
+	}
+	if successWithErrors {
+		log.Warnf("%s > installed package '%s': HTML response contains errors: %s", pm.instance.ID(), remotePath, err)
+		return nil
+	}
+	log.Infof("%s > installed package '%s'", pm.instance.ID(), remotePath)
+	return nil
+}
+
 func (pm *PackageManager) DeployWithChanged(localPath string) (bool, error) {
 	if pm.instance.IsLocal() && pm.IsSnapshot(localPath) { // TODO remove local check; support remote as well
 		return pm.deploySnapshot(localPath)

pkg/pkg/constants.go

@@ -0,0 +1,6 @@
+package pkg
+
+const (
+	InstallSuccess           = "<span class=\"Package imported.\">"
+	InstallSuccessWithErrors = "<span class=\"Package imported (with errors"
+)

pkg/project/app_classic/aem/default/etc/aem.yml

@@ -138,6 +138,17 @@ instance:
     snapshot_deploy_skipping: true
     # Disable following workflow launchers for a package deployment time only
     toggled_workflows: [/libs/settings/workflow/launcher/config/update_asset_*,/libs/settings/workflow/launcher/config/dam_*]
+    # Also sub-packages
+    install_recursive: true
+    # Use slower HTML endpoint for deployments but with better troubleshooting
+    install_html:
+      enabled: false
+      # Print HTML directly to console instead of writing to file
+      console: false
+      # Fail on case 'installed with errors'
+      strict: true
+      # Directory to place HTML report files
+      dir: aem/home/var/log/package/install
 
   # OSGi Framework
   osgi:

pkg/project/app_cloud/aem/default/etc/aem.yml

@@ -138,6 +138,17 @@ instance:
     snapshot_deploy_skipping: true
     # Disable following workflow launchers for a package deployment time only
     toggled_workflows: [/libs/settings/workflow/launcher/config/asset_processing_on_sdk_*,/libs/settings/workflow/launcher/config/dam_*]
+    # Also sub-packages
+    install_recursive: true
+    # Use slower HTML endpoint for deployments but with better troubleshooting
+    install_html:
+      enabled: false
+      # Print HTML directly to console instead of writing to file
+      console: false
+      # Fail on case 'installed with errors'
+      strict: true
+      # Directory to place HTML report files
+      dir: aem/home/var/log/package/install
 
   # OSGi Framework
   osgi: