Skip to content

Latest commit

 

History

History
86 lines (68 loc) · 4.59 KB

migrating_to_ams.md

File metadata and controls

86 lines (68 loc) · 4.59 KB
Raw

Migrating from using EGI ActiveMQ Message Brokers to using EGI ARGO Messaging Service

Prerequisites for using AMS

  • A valid host certificate from an IGTF Accredited CA.
  • A GOCDB 'Site' entry flagged as 'Production'.
  • A GOCDB 'Service' entry of the correct service type flagged as 'Production'. The following service types are used:
    • For Grid accounting use 'gLite-APEL'.
    • For Cloud accounting use 'eu.egi.cloud.accounting'.
    • For Storage accounting use 'eu.egi.storage.accounting'.
  • The 'Host DN' listed in the GOCDB 'Service' entry must exactly match the certificate DN of the host used for accounting. Make sure there are no leading or trailing spaces in the 'Host DN' field.
  • Messages sent via AMS must be below 1 Megabyte in size, and the messaging service is optimised around this limit. If your messages start hitting this limit when using SSM, see the advice at the bottom of this document.

Software requirements

Migration requires upgrading APEL SSM to at least version 2.4.0, installing the ARGO AMS Library, and adding new values to your configuration.

The ARGO AMS Library is available in UMD as python-argo-ams-library. Versions above 0.5.0 are recommended.

Configuration changes

Sender

The sender configuration is usually found under /etc/apel/sender.cfg. Follow the steps below to migrate.

  1. Comment out bdii and network.
  2. Uncomment host and set it to msg.argo.grnet.gr.
  3. Add the following as a new section at the top of your configuration: 
    [sender]
# Either 'STOMP' for STOMP message brokers or 'AMS' for Argo Messaging Service
protocol: AMS
  4. Add the following to the [messaging] section of your configuration: 
    # If using AMS this is the project that SSM will connect to. Ignored for STOMP.
ams_project: accounting
  5. To send to the central APEL Accounting server, change destination to one of the following depending on your type of accounting:
    • gLite-APEL for Grid Accounting
    • eu-egi-cloud-accounting for Cloud Accounting
    • eu-egi-storage-accounting for Storage Accounting

The next time ssmsend runs it should be using the AMS. You can check this by looking in the logs for a successful run, which should look like this:

2018-09-19 14:18:06,423 - ssmsend - INFO - ========================================
2018-09-19 14:18:06,424 - ssmsend - INFO - Starting sending SSM version 2.4.0.
2018-09-19 14:18:06,424 - ssmsend - INFO - Setting up SSM with Dest Type: AMS, Protocol : HTTPS
2018-09-19 14:18:06,424 - ssmsend - INFO - No AMS token provided, using cert/key pair instead.
2018-09-19 14:18:06,424 - ssmsend - INFO - No server certificate supplied.  Will not encrypt messages.
2018-09-19 14:18:07,061 - ssm.ssm2 - INFO - Found 1 messages.
2018-09-19 14:18:07,860 - ssm.ssm2 - INFO - Sent 5ba24c88/5ba24c8f0f129d, Argo ID: 18
2018-09-19 14:18:07,861 - ssm.ssm2 - INFO - Tidying message directory.
2018-09-19 14:18:07,862 - ssmsend - INFO - SSM run has finished.
2018-09-19 14:18:07,862 - ssmsend - INFO - SSM has shut down.
2018-09-19 14:18:07,862 - ssmsend - INFO - ========================================

Receiver

This is only used for the central Accounting Repository, Accounting Portal, and regional accounting servers.

  1. Follow the steps 1 to 4 as per the Sender documentation but editing your receiver configuration instead, usually found under /etc/apel/receiver.cfg, naming the section [receiver] rather than [sender].
  2. Change destination to be the subscription you are using to pull messages down.
  3. Add your token to the [messaging] section of your configuration: 
    token: your_token_here

Issues

Messages too large

  • Cloud sites using cASO should ensure they are using at least version 1.4.0 of cASO as this version limits the number of records in a message.
  • Grid sites using the APEL accounting client need to be using APEL 1.9.0 and SSM 3.2.0. They can then modify their APEL client script, usually located at /usr/bin/apelclient. At the moment, this requires a manual change, but will become a configuration option in the next version of APEL. For example, to halve the number of records per message from the default of 1000, add the line unloader.records_per_message = 500 after the call to DbUnloader: 
    @@ -233,6 +233,7 @@ def run_client(ccp):

         unloader = DbUnloader(db, unload_dir, include_vos, exclude_vos,
                               local_jobs, withhold_dns)
+        unloader.records_per_message = 500
         try:
             if interval == 'latest':
                 msgs, recs = unloader.unload_latest(table_name, send_ur)