Page tree
Skip to end of metadata
Go to start of metadata

The RabbitMQ module bundle provides an alternative way of publishing content and synchronizing instances in Magnolia. RabbitMQ is open source message broker software (a.k.a message-oriented middleware) that implements the Advanced Message Queuing Protocol (AMQP). 

The Magnolia RabbitMQ modules are suitable for: 

  • Environments consisting of more than three public instances.  
  • Environments that need to synchronize with other environments.

The RabbitMQ alternative gives you full control over activation and synchronization, while reducing the load on your author environment. Activation is persisted into queues, allowing for easier setup of continuous deployment environments.

Status INCUBATOR
TypeConnector
SubJar
JIRAQAARQ
Gitrabbitmq-publication

Installation

Maven is the easiest way to install the modules. Add the following dependencies to your bundle:

<!-- Integrates RabbitMQ with Magnolia -->
<dependency>
  <groupId>info.magnolia.rabbitmq</groupId>
  <artifactId>magnolia-rabbitmq-connector</artifactId>
  <version>${rabbitmqVersion}</version>
</dependency>

<!-- Allows you to synchronize and activate over RabbitMQ -->
<dependency>
  <groupId>info.magnolia.rabbitmq</groupId>
  <artifactId>magnolia-rabbitmq-activation</artifactId>
  <version>${rabbitmqVersion}</version>
</dependency>

Installing RabbitMQ

Follow the instructions at RabbitMQ to download and install the service. Instructions and packages are provided for all operating systems. It's also a good idea to enable the management plugin.

To get started even quicker consider using CloudAMQP to get RabbitMQ as a service.

Versions

1.7Magnolia 5.6

Usage

RabbitMQ can be used for publication and/or synchronization. There exists a single exchange sever with multiple queues. RabbitMQ exchanges are configured in the connector module.

Configuration

The exchange is configured on both author (producing instance) and public (consumer instance).

Client configuration

The connector module ships with a sample client configuration found here: /modules/rabbitmq-connector/rabbitmq-client/sampleClient

clientName

required

The value specified here must match the clientName property in the activate and deactivate commands for RabbitMQ.

Example: /modules/activation/commands/withContentSyncingVersioned/activate/rabbitmq-activate@clientName

enabled

required

Enables and disables the client. Toggling this property restarts the client.

hostName

required

Address of the broker.

password

required

Password of the user account to connect with.

portNumber

required

Port number to connect the AMQP service.

username

required

Username of the user account to connect with.

virtualHost

required

Virtual host to connect to.

Exchange configuration

The exchange is the kind of router that the queues connect to. The producer instance pushes messages to the exchange and the exchange then decides what to do with them. You must configure the exchange configuration on the author or producing instance. An example exchange configuration is found with the sample client found here: /modules/rabbitmq-connector/rabbitmq-client/sampleClient/exchangeConfig

name

required

The name specified here must match the exchangeName property in the activate and deactivate commands for RabbitMQ.

Example: /modules/activation/commands/withContentSyncingVersioned/activate/rabbitmq-activate@exchangeName

type

required

Exchange type. See AMQP 0-9-1 Model Explained for more.

Options: fanout, direct, topic, headers

Queue configuration

A list of queue definitions to bind with the exchange should be configured under the node queueConfigList. A sample queue configuration can be found here: /modules/rabbitmq-connector/rabbitmq-client/sampleClient/exchangeConfig/queueConfigList/queue

autodelete

optional

Automatically deletes the queue when last consumer unsubscribes.

exclusive

optional, default is false

Used for only one connection. Queue is deleted when that connection closes.

name

required

Name of the queue.

routingKey

optional

Key used to route messages. The queue binds to the exchange with the routing key. Applicable when direct mode is enabled in the exchange.

Consumer configuration

Consuming messages on a public instance is called activation or publication. A sample consumer definition can be found here: /modules/rabbitmq-connector/rabbitmq-client/sampleClient/consumerDefinition/activationConsumer

ackExchangeName

optional

Used for monitoring. An ACK exchange is bound to a queue on which activation confirmation messages are sent.

clientName

required

Name of the client to use.

consumerClass

required

Consumer class which implements AbstractMQConsumerJob.

Example: info.magnolia.rabbitmq.activation.jobs.ActivationConsumer

enabled

required

Enables and disables the consumer.

name

required

Name of the consumer.


Examples

Here are example to set up a producer and consumer using CloudAMPQ.

Producer (author)

The producer (author) configuration is located here: /modules/rabbitmq-connector/rabbitmq-client/

# Configuration shown in YAML but must exist as a JCR Node.
wolverine: 
  clientName: wolverine
  enabled: true
  hostName: wolverine.rmq.cloudamqp.com
  password: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  portNumber: 5672
  userName: thonoqso
  virtualHost: thonoqso
  exchangeConfig: 
    name: amq.fanout
    type: fanout
    queueConfigList: 
      queue1: 
        autodelete: false
        exclusive: false
        name: queue1
        routingKey: queue1
  consumerDefinitions: 
    activationConsumer: 
      enabled: false

Consumer (public)

The consumer (public) configuration is located here: /modules/rabbitmq-connector/rabbitmq-client/

# Configuration shown in YAML but must exist as a JCR Node.
wolverine: 
  clientName: wolverine
  enabled: true
  hostName: wolverine.rmq.cloudamqp.com
  password: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  portNumber: 5672
  userName: thonoqso
  virtualHost: thonoqso
  exchangeConfig: 
    name: amq.fanout
    type: fanout
    queueConfigList: 
      queue1: 
        autodelete: false
        exclusive: false
        name: queue1
        routingKey: queue1
  consumerDefinitions: 
    activationConsumer: 
      ackExchangeName: 
      clientName: wolverine
      consumerClass: info.magnolia.rabbitmq.activation.jobs.ActivationConsumer
      enabled: true
      name: activationConsumer
      queueName: queue1

Console

You can confirm the connection(s) from the RabbitMQ console. When the client(s) is enabled, the exchanges and queues should be visible. You can test this at  http://localhost:5672 when running RabbitMQ locally. Also, from the CloudAMQP console you can click the green RabbitMQ Manager button.

It's also possible to view the Queues and Exchanges from the console.

Commands

The RabbitMQ Activation module bootstraps activation commands into the standard Activation module. You can select commands/catalogs to match your purpose for using RabbitMQ.

Catalogs

The catalogs can be found here: /modules/rabbitmq-activation/commands

withContentSyncingAllows you to hook into transmission over RabbitMQ once you have done a standard activation.
withContentSyncingVersionedAdds versioning to withContentSyncing.
rbmqVersionedReplaces standard activation with RabbitMQ versioned activation.

Configuration

The activate and deactivate commands defined in the above catalogs depend on your client and exchange configurations.

clientName

required

Configured client name.

The clientName property must be the name of a configured RabbitMQ client.

exchangeName

required

Configured exchange name.

The exchangeName property must be the exchange name of a configured Rabbit MQ client.

delegateAck

required , default is true

When set to true message acknowledgements are delegated to RabbitMQ. Set to false if you want to use the rabbitmq-monitoring module instead.

stackSize

required , default is 1

Stack size allows you to send more nodes per activation message, making messages bigger but reducing the number of messages in the queue.

blocktime

optional , default is 100

Blocktime in seconds. Works only if delegateAck is false and if the ack exchange is configured correctly.

blockActivation

optional

Blockactivation. Works only if delegateAck is false and if the ack exchange is configured correctly.

minNbrOfPublics

optional

Minimum number of public instances to wait ack for. Works only if delegateAck is false and if the ack exchange i s configured correctly.

Adding to apps

Using the RabbitMQ commands in your apps is simple. Reference the command to use by the catalog name.

Here is an example using the Pages app:

Testing activation

To test the new commands activate content and then open the RabbitMQ console. You should see messages in the queue (fan1) . 

 

Activation mechanism 

Each activation is prefixed with a BOA (Beginning Of Activation) message and postfixed with a EOA (End Of Activation) message. When the consumer receives a message on a public instance, the message is stamped with the identity of the receiver and send back to the ack queue. When the author receives the message, the author knows exactly if the message started consuming the activation. In addition, identifying the activation in queue with BOA and EOA allow identification of TX inside the the queue.

Testing message consumption

When the activation consumer is configured, the receiving instance should start consuming messages.

Open the RabbitMQ console to verify that the specific queue (fan1) is being emptied and that the nodes appear on the receiving instance. 

 

Content Syncing

While transactions are possible with RabbitMQ, they are complicated and slow down publishing. Ideally, the same activation message would be received by all instances at the same time, but this is technically very difficult to achieve and does not add much advantage in any event.

What is important however is that a Magnolia instance knows when a node is out of sync with another node, and that the load balancer knows which node is late. This is achieved by recording the time stamp and latest message state

Whenever a message has been consumed and the content activated on the instance, the instance stores the message id (which is a long that is incremented) and the date the message was activated.

Example:

InstanceMessage IdTimestamp
Pubic 141441631639
Public 241441631640
Public 331441631600


This information is either sent to the load balancer or made available over a servlet. The goal is that the load balancer knows which instance is behind and by how much. In the example above the load balancer controlling tool knows that Public 1 and Public 2 are synced. It can decide to wait for Public 3 to sync, and if this does not happen, to alert the systems administrator or trigger the creation of a new instance.

SyncState and SyncStore mechanism

The Magnolia RabbitMQ implementation introduces the concepts of SyncState and SyncStore to achieve content synchronicity between public instances.

The mechanism is basically a counter that is incremented when each activation message is sent. The counter is sent in the message header. When the receiving instance gets the message, the instance increments its own activation counter. This means that two public instances which have different SyncStates are out of sync until the counters once again have the same number.

The load balancer calls the REST service of the public nodes and in this way knows which public node is most up to date and which one is out of sync.

Examples

StateReasonAction
REST service is not responding.Public node down.Remove from the pool.
Rest service's sync state is 0.Public Node is up but completely out of sync.

average(syncstates)-syncstate(pubnode)>=x

Public node is working but under heavy load.Remove public node temporarily from the pool until <x.

The loadbalancer also knows when the public instances are getting activated. SyncStates are persisted to the JCR.

Guaranteeing synchronicity 

The Dynamically Weighted Least Connections Algorithm on your load balancer guarantees synchronicity and ensures that public instances do not get too far out of sync. See Least-Connection Algorithm based on variable weight for multimedia transmission for more.

The load balancer plays a big role in this. During a very big and long activation process, the load balancer can decide to redirect the traffic to the instances which have the highest and most equal ids. This increases traffic on these instances, which in turn slows down the activation process. Instances with initial low and unequal ids now have a higher chance of reaching the same state as the instances under load. 

The algorithm creates a kind of "auto-damping" system that slows down instances by giving them more traffic, while speeding up the activation process on the others . Having the load balancer actively take the activation mechanism into consideration stabilizes the whole public node group.

Warnings

  • This module is at INCUBATOR level.

Changelog

  • Version 1.7 - Initial release of the extensions version of the module.
    • QAARQ-22 - Getting issue details... STATUS
    • QAARQ-21 - Getting issue details... STATUS