Skip to content

Latest commit

 

History

History
598 lines (438 loc) · 21.8 KB

README.md

File metadata and controls

598 lines (438 loc) · 21.8 KB
Raw

ISLE: Islandora Enterprise 8 Prototype

Nightly CI

LICENSE

Introduction

Docker Compose project facilitating creation and management of Islandora 8 Infrastructure under Docker using Docker Compose.

This is a prototype of the docker-compose file, Docker service and image configuration structure for the ISLE Phase III - ISLE / Islandora 8 Prototype (isle-dc) project.

The workflow for this repository centers around using the provided Makefile to generate an appropriate docker-compose.yml file.

There are three ENVIRONMENTs or ways of development that this repository supports:

  • demo (Example site for testing the images)
  • local (Local development using composer/drush in the codebase folder)
  • custom (Use a custom built image or generate one from the codebase folder)

To quickly get started, we recommend running the demo environment first after you have completed the Installation.

Requirements

  • Composer 1.10+
  • Desktop / laptop / VM (Must be able to support running GNU Make)
  • Docker-CE 19.x+ (If using Docker Desktop for Windows, any stable release after 2.2.0.4, or use a 2.2.0.4 with a patch due to a bug)
  • Docker-compose version 1.25.x+
  • Drush 9.0+
  • Git 2.0+
  • GNU Make 4.0+
  • PHP 7.2+ (Also requires the same ext packages you intend to use in your site.)

Installation

Configuring the Environment

To run the containers you must first generate a docker-compose.yml file. It is the only orchestration mechanism provided to launch all the containers, and have them work as a whole.

To get started generate the defaults with the following command:

make

This will create the following files which you can customize:

File Purpose
.env Responsible for setting variables used in docker-compose.*.yml files
Determines which docker-compose.*.yml are included in docker-compose.yml
docker-compose.env.yml Allows the user to set environment settings inside of containers and override any services configuration

At a minimum, you'll want to consider setting ENVIRONMENT in the .env file to either demo, local, or custom. The default is demo.

Changing the host name

By default, the domain traefik.me is used, which resolves to localhost, but allows us to treat things as if there were a fully qualified domain name. Namely, we can have https in development and staging scenarios, even when all you have is an IP address.

However, if you are deploying somewhere other than localhost and you own a domain, you can change it by setting DRUPAL_SITE_HOST in the .env file. That is, for example.org:

DRUPAL_SITE_HOST=example.org

Using an IP address

If you have an IP address but no domain, you can set the value to X-X-X-X.traefik.me, where X-X-X-X is your IP address, but with hyphens instead of dots. For example, if your IP address is 123.45.67.89:

DRUPAL_SITE_HOST=123-45-67-89.traefik.me

There are also a handful of variables in docker-compose.env.yml you'll want to adjust if using an IP address with traefik.me. For each of these, change the dot between COMPOSE_PROJECT_NAME and DRUPAL_SITE_HOST to a hyphen (i.e. ${COMPOSE_PROJECT_NAME-isle-dc}.${DRUPAL_SITE_HOST-traefik.me} becomes ${COMPOSE_PROJECT_NAME-isle-dc}-${DRUPAL_SITE_HOST-traefik.me}). If you have any doubts about what you're doing, just copy/paste these values directly into place in your docker-compose.env.yml file.

Variable Value
DRUPAL_DEFAULT_CANTALOUPE_URL https://islandora-${COMPOSE_PROJECT_NAME-isle-dc}-${DRUPAL_SITE_HOST-traefik.me}/cantaloupe/iiif/2
DRUPAL_DEFAULT_DB_HOST ${DRUPAL_DATABASE_SERVICE}-${COMPOSE_PROJECT_NAME-isle-dc}-${DRUPAL_SITE_HOST-traefik.me}
DRUPAL_DEFAULT_FCREPO_HOST fcrepo-${COMPOSE_PROJECT_NAME-isle-dc}-${DRUPAL_SITE_HOST-traefik.me}
DRUPAL_DEFAULT_MATOMO_URL https://islandora-${COMPOSE_PROJECT_NAME-isle-dc}-${DRUPAL_SITE_HOST-traefik.me}/matomo/
DRUPAL_DEFAULT_SITE_URL https://islandora-${COMPOSE_PROJECT_NAME-isle-dc}.${DRUPAL_SITE_HOST-traefik.me}
MATOMO_SITE_HOST islandora-${COMPOSE_PROJECT_NAME-isle-dc}-${DRUPAL_SITE_HOST-traefik.me}

When using an IP address, your site will be available at https://islandora-isle-dc-X-X-X-X.traefik.me/, where X-X-X-X is your IP address. For example, https://islandora-isle-dc-123-45-67-89.traefik.me/

Applying changes

Once you are happy with your changes to the above files you can regenerate the docker-compose.yml, and pull the required images using the Makefile like so:

make

After this point you can just interact with the docker-compose.yml file like you would normally.

docker-compose up -d

With Docker Compose there are many features such as displaying logs among other things for which you can find detailed descriptions in the Docker Composer CLI Documentation

Demo Environment

To quickly get started, we recommend running the demo environment first.

This is the default environment if you do a clean checkout of this repository and run the following:

make
docker-compose up -d

This environment is just meant as a show case of the basic islandora site.

You should be able to reach it in your browser at islandora-isle-dc.traefik.me if you followed the instructions under the Installation section.

Local Environment

This environment is intended for local development. Users will create a Drupal site in the folder codebase, which gets bind mounted into the drupal service container. Allowing developers to use composer / drush locally to work out of the codebase folder.

There are a three ways in which you can create a local environment.

  • From the islandora/demo image.
  • From an existing site.
  • From scratch.

N.B: Before attempting any of these methods make sure you have set ENVIRONMENT to local in your .env file.

After you have setup your local site you can then work directly out of the codebase folder, as if you had installed Drupal locally.

Create Local Environment from islandora/demo Image

The following Makefile command is provided as method to quickly get a site up using the islandora/demo image as base.

make create-codebase-from-demo

It will take a few minutes to spin up the demo instance, export its configuration and copy the site into the codebase folder.

Once this is done you can bring up your local site using docker-compose.

docker-compose up -d

At this point the site will not be installed. There are two was to do an installation, from an existing configuration or in a stepwise fashion.

After either of your chosen methods you will still need to update services like solr and blazegraph, etc, these commands are combined into a single target for convenience.

make hydrate

From existing Configuration

To be able to install from an existing configuration you must change the following environment variables in docker-compose.env.yml:

Environment Variable Value
DRUPAL_DEFAULT_CONFIGDIR /var/www/drupal/config/sync
DRUPAL_DEFAULT_INSTALL_EXISTING_CONFIG "true"
DRUPAL_DEFAULT_PROFILE minimal

Regenerate your docker-compose.yml file, and restart the container.

make docker-compose.yml
docker-compose up -d

You also need to change the site configuration to use the minimal profile.

make remove_standard_profile_references_from_config

At this point you should be able to perform the installation

make install

Finally configure the rest of the site which depends on environment variables.

make hydrate

N.B.: There is a bug which affects islandora_fits. For now you must manually set a value. Visit http://islandora-isle-dc.traefik.me/taxonomy/term/1/edit and set the value URL to https://projects.iq.harvard.edu/fits.

Manually

you can install it by running the following Makefile command:

make install

That will create the required database and install a bare bones site.

Now that the drupal service is running you can update the settings.php with the appropriate settings from the environment variables defined in docker-compose.env.yml, by running the following Makefile command:

make update-settings-php

At this point the site is still bare-bones as we have not imported the site configuration. Or you can manually setup the site as you see fit. To import an existing configuration you can use the following command.

make config-import

N.B: You'll have to run this twice, due to bugs in the dependencies of configurations!

Finally configure the rest of the site which depends on environment variables.

make hydrate

N.B.: There is a bug which affects islandora_fits. For now you must manually set a value. Visit http://islandora-isle-dc.traefik.me/taxonomy/term/1/edit and set the value URL to https://projects.iq.harvard.edu/fits.

Create Local Environment from Existing Site

Copy or clone your existing site into the codebase folder. Start the system with docker-compose up -d, then run composer install via docker-compose exec drupal with-contenv bash -lc 'COMPOSER_MEMORY_LIMIT=-1 composer install' and make sure your database and user are created with make databases, and that your settings.php file is correct with make update-settings-php.

Then you have a number of options you can:

Or you can import an existing database if you have it.

make database-import SRC=/tmp/dump.sql

Finally configure the rest of the site which depends on environment variables.

make hydrate

Create Local Environment from Scratch

You can create a composer project for your drupal site from scratch.

Some popular examples:

  • drupal/recommended-project
  • drupal-composer/drupal-project:8.x-dev
  • islandora/drupal-project:8.8.1
  • born-digital/drupal-project:dev-isle8-dev
mkdir ./codebase
cd ./codebase
composer create-project --ignore-platform-reqs --no-interaction --no-install drupal/recommended-project .
composer require -- drush/drush
composer install
make
docker-compose up -d
make install

At this point you should have a functioning Drupal site that you can customize as you see fit using composer / drush commands in the codebase folder.

Custom Environment

This environment is used to run your custom drupal image which can be produced outside of this repository. You can specify the image in your .env file using the settings PROJECT_DRUPAL_DOCKERFILE if you want to build it in the context of this repository.

For convenience a sample.Dockerfile is provided from which you can generate a custom image from the codebase folder. For example if you followed the guide above to create the codebase folder from the islandora/demo image.

And then run it by changing ENVIRONMENT to be custom and regenerating the docker-compose.yml file and building the image.

make docker-compose.yml
make build

At this point you could run it using docker-compose:

docker-compose up -d

To specify an image created outside of this repository, you can add the following to docker-compose.env.yml:

drupal:
  image: YOUR_CUSTOM_IMAGE

Services

Islandora is composed of many different services, this project has split these services up such that they have their own Container.

For in-depth documentation of the various islandora images see the isle-buildkit repository.

Other services will be documented below:

Watchtower

The watchtower container monitors the running Docker containers and watches for changes to the images that those containers were originally started from. If watchtower detects that an image has changed, it will automatically restart the container using the new image. This allows for automatic deployment, and overall faster development time.

Note however Watchtower will not restart stopped container or containers that exited due to error. To ensure a container is always running, unless explicitly stopped, add restart: unless-stopped property to the container in the [docker-compose.yml] file. For example:

mariadb:
    image: islandora/mariadb:latest
    restart: unless-stopped

The watchtower can be disabled/enabled via the INCLUDE_WATCHTOWER_SERVICE variable in your .env file.

# Includes `watchtower` as a service.
INCLUDE_WATCHTOWER_SERVICE=true

Traefik

The traefik container acts as a reverse proxy, and exposes some containers through port 80/443/3306. This allows access to the following urls by default.

Since Drupal passes links to itself in the messages it passes to the microservices, and occasionally other urls need to be resolved on containers that do not have external access, we define aliases for most services on the internal network.

Aliases like so are defined on most services to mimic their routing rules in Traefik:

drupal:
    image: islandora/demo:latest
    # ...
    networks:
      default:
        aliases:
          - islandora-isle-dc.traefik.me

The traefik service can be disabled/enabled via the INCLUDE_TRAEFIK_SERVICE variable in your .env file.

# Includes `traefik` as a service.
INCLUDE_TRAEFIK_SERVICE=true

ETCD

The etcd container is a distributed reliable key-value store, which this project can use for configuration settings and secrets.

It is not enabled by default.

# Includes `etcd` as a service.
INCLUDE_ETCD_SERVICE=false

Troubleshooting/Issues

Post your questions here and subscribe for updates, meeting announcements, and technical support

FAQ

Question: When doing an drush config:import -y I get one of the following errors

Error:

Site UUID in source storage does not match the target Storage.

This occurs when your configuration has come from a site other than the one that is currently installed. You can run the following command to override the site uuid so that you can import your configuration:

make set-site-uuid

Error:

Entities exist of type <em class="placeholder">Shortcut link</em> and <em class="placeholder">Shortcut set</em>
<em class="placeholder">Default</em>. These entities need to be deleted before importing

These are entities created by the standard installation profile, you can delete them with the following command in the codebase folder:

make delete-shortcut-entities

Error:

Error: Call to a member function getConfigDependencyName() on null in ... Entity/EntityDisplayBase.php on line 325 #0 ... /codebase/web/core/lib/Drupal/Core/Config/Entity/ConfigEntityBase.php(318): Drupal\Core\Entity\EntityDisplayBase->calculateDependencies()

There is some bug in the dependencies of the various modules. Until those dependencies issues are resolved just rerun the command until they go away.

drush config:import -y

Question: When doing an drush sql:dump I get one of the following errors

Error:

This arises from our use of MariaDB for the database in Docker, not matching the same client on your host system. Which probably uses the MySQL clients mysqldump executable. You can specify the following command using drush to get around it:

drush sql:dump --extra-dump="--column-statistics=0" > /tmp/dump.sql

Or you can use the provided method in the Makefile which should be more portable.

make database-dump DEST=/tmp/dump.sql

Development

If you would like to contribute to this project, please check out CONTRIBUTING.md. In addition, we have helpful Documentation for Developers info, as well as our Developers section on the Islandora.ca site.

drush --debug sql:dump --extra-dump="--column-statistics=0" > /tmp/dump.sql^C

Maintainers/Sponsors

Architecture Team

Sponsors

This project has been sponsored by:

  • Grinnell College
  • Tri-College (Bryn Mawr College, Haverford College, Swarthmore College)
  • Wesleyan University
  • Williams College
  • Colgate University
  • Hamilton College
  • Amherst College
  • Mount Holyoke College
  • Franklin and Marshall College
  • Whitman College
  • Smith College
  • Arizona State University
  • Canadian Writing Research Collaboratory (CWRC)
  • Johns Hopkins University
  • Tulane University
  • LYRASIS
  • Born-Digital

License

MIT