Is granular replica forest configuration possible?

[mflatscher]

Forests-per-host with replica forests configured in ml-config works great for me. However, I've had no luck with the following; was wondering whether that's possible from ml-config or whether it requires an app_specific step:

• Assume I have a 3-node cluster
• 2 content forests per host with replication via ml-config (This works great for me.)
• Now I'd like to configure one forest each on one of my three hosts for my Modules and Triggers databases respectively, and one replica forest for each on another host. (I haven't gotten this to work yet)

[grtjn]

There is limited control available in the current dev branch. The procedure used by Roxy is as follows:

• take group from database
• n content forests are created on each host participating in that group (you can control this with hosts/host/group)
• for each forest it looks for replicas (assignment/replica-names/replica-name), these are created and attached to next hosts (with rotation) within same group

Note: let replica-name point to forest-name itself, Roxy will append -replica to its name.

Does this work good enough for you? Feedback welcome!

[mflatscher]

Thanks, Geert. This sounds good, but I'm still struggling with it—I'm doing something wrong :-/

Note that both forest creation and assignments work great with the content database; I'm only encountering the below when trying to add replication for modules and triggers:

With Roxy 1.7.1,

(1) If I point replica-name to forest-name, I get the following on bootstrap: ADMIN-DUPLICATENAME: (err:FOER0000) Forest '[forest-name]' already exists.

Adding a "-replica" suffix creates the forests, but something goes wrong for me during assignments:

(2) With this in my ml-config.xml

<assignment>
  <forest-name>@ml.modules-db</forest-name>
  <replica-names>
    <replica-name>@ml.modules-db-replica</replica-name>
  </replica-names>
  <data-directory>@ml.data-directory</data-directory>
</assignment>
...
<assignment>
  <forest-name>@ml.modules-db-replica</forest-name>
  @ml.forest-data-dir-xml
</assignment>

and this

<database group="@ml.default-group">
  <database-name>@ml.modules-db</database-name>
  <forests-per-host>1</forests-per-host>
  <forests>
    <forest-id name="@ml.modules-db"/>
    <data-directory>@ml.data-directory</data-directory>
  </forests>
  ...
</database>

I get the following on bootstrap: "Cannot delete forest xxxx unless replica forest xxxx is detached or also deleted" (from setup:do-wipe()), and wipe is broken afterwards.

(3) Finally, could you elaborate on your second bullet item re: hosts/host/group? I can't find that path in any of the sample ml-configs.

Thanks a lot!

Markus

[grtjn]

Host/group support is very new, and replica support has been touched recently as well. I think you are running with the last official release, the master branch. You can use the upgrade command to pull in the 'nightly-build' from the dev branch:

./ml upgrade --branch=dev

Try again with dev version, that will probably make difference..

[mflatscher]

Thanks a lot, Geert. I tried both the dev branch and the 1.7.1 release, and they did behave the same for my purposes.

I just realized that the cause of the ADMIN-DELETEREPLICATOO that is thrown during bootstrap for my project is (1) I have replication configured for my Modules database, and (2) I have ml.schemas-db point to my ml.modules-db. (I believe (1) and (2) should be sufficient to reproduce the error.)

By extension, I'd expect that the same would happen if you had your triggers live in your modules database.

Would you consider this a bug?

[grtjn]

Re-reading your earlier comment after running a few tests myself as well..

(1) sorry, my bad. You need to append some suffix to replica-names, as mentioned in #147

(2) the config you show seems to work just fine for me, but maybe that is because I am not yet using Modules as Schemas. Creating multiple forests for the moduled-db, including replicas works nicely for me. I'll try modules as schemas next..

(3) if you are just using one group (Default), you don't need this bit. forests-per-host causes bootstrap to create forests on every host, and there is special code in there to make sure replica forests are on different hosts. If you take out the forests-per-host from modules-db, you would have to assign the replica forest to a specific other host using an extra <host-name>...</host-name> inside the assignment element. I like the forests-per-host for modules better though. If you intend to use multiple groups though, you could target specific hosts to specific groups using the hosts configuration as shown in the sample config:

https://github.com/marklogic/roxy/blob/dev/deploy/sample/ml-config.sample.xml

<hosts xsi:schemaLocation="http://marklogic.com/xdmp/hosts hosts.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://marklogic.com/xdmp/hosts">
  <host>
    <host-name>myhostname</host-name>
    <group name="@ml.group"/>
  </host>
</hosts>

[grtjn]

About using modules-db as schemas-db, that should work as well. I added this to my properties:

schemas-db=${modules-db}

Also you need to make sure to disable the following lines in ml-config.xml:

@ml.schemas-assignment

and

@ml.schemas-db-xml

With these changes using modules-db as schemas-db should work. bootstrap and wipe run smoothly for me. With latest from dev branch that is..

[grtjn]

About using modules-db as triggers-db, I'd expect that should work equally well if you follow the same pattern as for schemas-db. Haven't tried though..

[mflatscher]

But of course—@ml.schemas-assignment and @ml.schemas-db-xml are not expected if schemas live inside a modules database. Makes perfect sense.

Works great with the 1.7.1 release as well.

Thank you very much for your patience and your help!

[grtjn]

Closing ticket..

[grtjn]

@mflatscher You might be interested I have opened a few new tickets regarding forest replication, and have pieces of code to improve deployment of replication: #366, #378, #379, #380, #383. Hoping to make PRs soon..