Memcached process not started once container running

[johangant]

Tell us about your setup

Lando version: v3.0.0-rc.22
Operating system: macOS 10.14.6

Tell us about your .lando.yml

name: my-lando-app
recipe: drupal8
config:
  webroot: .
services:
  memcached:
    type: memcached
    portforward: true

Tell us about the command you were running

lando init
# ? From where should we get your app's codebase? current working directory
# ? What recipe do you want to use? drupal8
# ? Where is your webroot relative to the init destination? ./
# ? What do you want to call this app? my-lando-app
lando start
lando ssh -s appserver --user=root
apt update
apt install telnet
telnet memcached 11211

Tell us about the error you got

Running the above steps should give you a refused connection from the memcached container which, although reported as healthy, has no running memcached service inside it so refuses connections from applications or other tools.

What I've noticed is that if you run these commands, you can get it to connect:

lando ssh -s memcached
/opt/bitnami/memcached/bin/memcached &
# exit ssh
lando ssh -s appserver --user=root
telnet memcached 11211
# Should now connect

Tell us generally about your bug

If I run the docker image with docker exec -it memcached /bin/bash it has a working memcached process running.

I'm a little unclear whether it's the lando helper scripts that might need adjusting or if it's an upstream change that might have caused a fault.

Some output from lando logs -s memcached which may be relevant:

Attaching to mylandoapp_memcached_1
memcached_1     | mkdir: cannot create directory '/etc/ssh': Permission denied
memcached_1     | Running command /bin/sh -c chmod +x /launch.sh && /launch.sh
memcached_1     | /bin/sh: 1: /launch.sh: Permission denied
memcached_1     | mkdir: cannot create directory '/etc/ssh': Permission denied
memcached_1     | Running command /bin/sh -c chmod +x /launch.sh && /launch.sh
memcached_1     | /bin/sh: 1: /launch.sh: Permission denied
memcached_1     | Scanning /lando/keys for keys...
memcached_1     | Found keys
memcached_1     | Scanning /user/.ssh for keys...
memcached_1     | Found keys /user/.ssh/id_rsa
memcached_1     | Ensuring permissions and ownership of /user/.ssh/id_rsa...
memcached_1     | Checking whether /user/.ssh/id_rsa is a private key...
memcached_1     | Scanning /var/www/.ssh for keys...
memcached_1     | Found keys
memcached_1     | Using the following keys: /user/.ssh/id_rsa
memcached_1     | chmod: changing permissions of '/etc/ssh/ssh_config': Operation not permitted
memcached_1     | Running command /bin/sh -c chmod +x /launch.sh && /launch.sh
memcached_1     | /bin/sh: 1: /launch.sh: Permission denied

[craigwillis85]

I can confirm that I am also seeing this in rc22 (also rc21)

The hack that @johangant mentioned does seem to allow my app to connect to memcached

[brunodbo]

Also seeing this in rc22. Starting memcached manually as mentioned by @johangant works for me too.

This is the output of lando logs -s cache, right after rebuilding:

Attaching to textillia_cache_1
cache_1       | chmod: changing permissions of '/helpers/add-cert.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/auth.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/binding.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/generate-key.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/get-remote-url.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/lando-entrypoint.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/load-keys.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/mounter.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/pantheon.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/pull.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/push.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/refresh-certs.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/setup-ca.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/sql-export.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/sql-import.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/switch.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/user-perm-helpers.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/user-perms.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/user-perms.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/scripts/load-keys.sh': Operation not permitted
cache_1       | mkdir: cannot create directory '/etc/ssh': Permission denied
cache_1       | Running command /bin/sh -c chmod +x /launch.sh && /launch.sh
cache_1       | chmod: changing permissions of '/launch.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/add-cert.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/auth.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/binding.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/generate-key.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/get-remote-url.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/lando-entrypoint.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/load-keys.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/mounter.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/pantheon.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/pull.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/push.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/refresh-certs.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/setup-ca.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/sql-export.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/sql-import.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/switch.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/user-perm-helpers.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/user-perms.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/user-perms.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/scripts/load-keys.sh': Operation not permitted
cache_1       | mkdir: cannot create directory '/etc/ssh': Permission denied
cache_1       | Running command /bin/sh -c chmod +x /launch.sh && /launch.sh
cache_1       | chmod: changing permissions of '/launch.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/add-cert.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/auth.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/binding.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/generate-key.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/get-remote-url.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/lando-entrypoint.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/load-keys.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/mounter.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/pantheon.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/pull.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/push.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/refresh-certs.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/setup-ca.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/sql-export.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/sql-import.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/switch.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/user-perm-helpers.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/user-perms.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/helpers/user-perms.sh': Operation not permitted
cache_1       | chmod: changing permissions of '/scripts/load-keys.sh': Operation not permitted
cache_1       | mkdir: cannot create directory '/etc/ssh': Permission denied
cache_1       | Running command /bin/sh -c chmod +x /launch.sh && /launch.sh
cache_1       | chmod: changing permissions of '/launch.sh': Operation not permitted

I tried starting memcached in the build/build_as_root step of the cache service, but that didn't help:

cache:
    type: memcached:1.5.12
    portforward: true
    mem: 64
    build_as_root:
      - /opt/bitnami/memcached/bin/memcached &

[andrebonon]

Starting memcache manually works for me too.
I got the same error as @brunodbo .

[edurenye]

Looks like there are a lot of issues like this, that something happens with Bitnami and the container does not work anymore with Lando.
We should look for a way to prevent that from happening.

[tormi]

This is a workable memcached service setup based on the @johangant's solution:

services:
  memcached:
    type: memcached
    portforward: true
    run:
      - /bin/sh -c '/opt/bitnami/memcached/bin/memcached start -d'

[hkirsman]

Why I didn't come here earlier. Was pulling my hair out with this. Well good thing is I have a bit alternative solution:

services:
  memcached:
    type: memcached
    run:
      - memcached -d

I'll add also my Drupal configuration needed to get this work:

<?php

$lando_info = json_decode(getenv('LANDO_INFO'), TRUE);

$settings['memcache']['servers'] = [$lando_info['memcached']['internal_connection']['host'] . ':' . $lando_info['memcached']['internal_connection']['port'] => 'default'];
$settings['memcache']['bins'] = ['default' => 'default'];
$settings['memcache']['key_prefix'] = '';
$settings['cache']['default'] = 'cache.backend.memcache';

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this if you are wondering why we auto close issues.

[hanoii]

I'd think this is still worth fixing?

[pirog]

So it looks like the issue here is that the new launch.sh script was not being mounted correctly because the memcached configuration directory was set to null instead of its actual location.

When docker tries to mount a non-existent file it usually ends up creating an empty directory instead of the actual file. If you ssh into a memcache container and check out /launch.sh you can see that it is a directory.

This should be easy to fix by itself. we should also get better tests for memcached so regressions like ths are easier to catch.

[pirog]

Alright, this is merged into master and will be in the next release