added support for installing plugins such as apoc at runtime

[eastlondoner]

This uses the same mechanism as neo4j desktop to fetch plugins from github.

It allows you to have apoc / graph algorithms / graph ql / etc. plugins by setting a json array:

e.g.

docker run --env NEO4JLABS_PLUGINS='["graph-algorithms", "apoc"]' neo4j

Plugins are always downloaded on container startup. A future improvement would be to use a hash (e.g. sha256 or md5) of the plugin to allow for caching of the dowloaded jar if the plugins dir is persisted.

[jexp]

Looks great!

[eastlondoner]

I see what’s wrong with the tests - I need to rub docker with —user set (and add the apoc loading logic to all 3.x.

I’m taking the fam to the beach today, so prob won’t work on it till tomorrow at least

[kfreytag]

@eastlondoner Is there a reason why you're specifying a path for the plugins as opposed to an abstract name? As a user, I'd prefer to not have any sort of implementation details leak through to the command line here and just say NEO4JPLUGINS='["apoc","algos"]' or something like that. That also allows us to change the implementation under the sheets in the future without affecting the user surface.

[eastlondoner]

@kfreytag it's not a path - it's the GitHub user / organisation where the plugin is coming from.
To avoid maintaining an extra dictionary of plugin name -> GitHub location and to hopefully be more future proof.

But you're totally right that this is an implementation detail - we could internally manage the mapping of plugin name -> where plugin exists and then it would provide a consistent interface to the user that doesn't depend on the implementation

[kfreytag]

By "path" I did mean GitHub path / repo, but yeah - the point remains. Let's abstract that away from the user. It also allows us to constrain which plugins can be "activated" in our official plugin as a NEO4JPLUGIN automatically.

[eastlondoner]

I changed the syntax to NEO4JLABS_PLUGINS='["apoc", "graph-algorithms"]'

I limited this change to versions >= 3.3

[jennyowen]

This looks fine.
It seems like the secure mode and this new setting might have some weird interaction. We probably need to unit test the combinations (that doesn't really affect this PR though).

[eastlondoner]

@jennyowen it shouldn’t be too weird - mainly it adds /plugins to /data and /logs as places we need to be able to write to.

[jennyowen]

@eastlondoner yeah, I realise the requirements aren't complicated, but following the logic through the docker-entrypoint.sh made for confusion.

[CarlColglazier]

I am trying to get the graph algorithms working on my Docker deploy. It seems like this was reverted in ec4cbd2, but I can't figure out why. Does anyone know?

[jennyowen]

@CarlColglazier it was reverted because it caused test failures and had issues with out CI infrastructure. We fixed it since then though, and the feature is available in 3.5.10 onwards.

[CarlColglazier]

@jennyowen Thank you!

I was having issues getting the plugins to download using docker-compose, but it seemed to work once I used my own Dockerfile:

FROM neo4j:3.5.11

ENV NEO4JLABS_PLUGINS '["apoc", "graph-algorithms"]'

CMD ["neo4j"]

[eastlondoner]

@CarlColglazier I don't think that will work. Setting the CMD as neo4j executes the neo4j binary directly, bypassing the extra logic that we have in the docker entry point.

Can you share the docker-compose file that you were having problems with?

[joelduerksen]

@eastlondoner this is a really nice feature, and I immediately switched over to using it. Very clean, I was able to clean the wgets out of my scripts.

Now I'm working to move to neo4j 4.0, graph-algorithms plugin not loading, and notice that it assumes referenced libraries will always have the same version as the neo4j db. I took a look at the code in docker-entrypoint.sh...

It might be nice to be able to specify a version (if there is a compatible jar). In the past some jars were compatible (across minor update versions)..

** graph-algo 3.5.14.0 does not appear to work with neo4j 4.0. Adding the jar causes neo4j to shutdown with a seemly unrelated error (but if I remove the jar, it starts OK). It was a big update, so this is not a surprise.

[eastlondoner]

@joelduerksen the docker container looks up the correct plugin version to match the neo4j version from the plugin's repo. In this case: https://github.com/neo4j-contrib/neo4j-graph-algorithms/raw/master/versions.json

There is currently no published graph-algos version that is compatible with Neo4j 4.0
@AliciaFrame can provide more information about graph-algorithms compatibility...

[AliciaFrame]

@joelduerksen - the graph algorithms library (3.5.14) isn't compatible with Neo4j 4.0, and won't be.

We'll be releasing an updated Graph Data Science, or GDS, library in the next month or so (new surface, officially supported, improved algorithms and infrastructure) and formally deprecating the older algos library. We're currently planning to provide a Neo4j 4.0 compatible release by April of this year.

Sorry for the confusion!

[davidburkitt]

This feature doesn't seem to be valid for docker-compose. Can you provide a working example if this should be the case please?

[eastlondoner]

@davidburkitt could you possibly provide the docker compose file that you are using which isn't working ? I guess you already have one. It will take me a little while to make one from scratch.

[davidburkitt]

version: '3.7'
services:
web:
container_name: neo4j_web
image: neo4j:latest
restart: unless-stopped
ports:
- "7474:7474"
- "7687:7687"
volumes:
- ${HOME}/neo4j/data:/data
- ${HOME}/neo4j/logs:/logs
- ${HOME}/neo4j/import:/import
environment:
- NEO4JLABS_PLUGINS='["apoc"]'
- NEO4J_AUTH=${CREDS}
- NEO4J_dbms_security_procedures_unrestricted='apoc.\\\*'
- NEO4J_apoc_export_file_enabled=true
- NEO4J_apoc_import_file_enabled=true
- NEO4J_apoc_import_file_use__neo4j__config=true

[eastlondoner]

@davidburkitt The problem here is to do with string quoting in yaml. You do not need to quote the values of the env vars being set in yaml.
So this is fine:

- NEO4JLABS_PLUGINS=["apoc"]

If you do want to quote them then you have to quote the entire string:

- 'NEO4JLABS_PLUGINS=["apoc"]'

n.b. you should not need to add NEO4J_dbms_security_procedures_unrestricted if you are using NEO4JLABS_PLUGINS

So this should work fine:

version: '3.7'
services:
  web:
    container_name: neo4j_web
    image: neo4j:latest
    restart: unless-stopped
    ports:
      - "7474:7474"
      - "7687:7687"
    volumes:
      - ${HOME}/neo4j/data:/data
      - ${HOME}/neo4j/logs:/logs
      - ${HOME}/neo4j/import:/import
    environment:
      - NEO4JLABS_PLUGINS=["apoc"]
      - NEO4J_AUTH=${CREDS}
      - NEO4J_apoc_export_file_enabled=true
      - NEO4J_apoc_import_file_enabled=true
      - NEO4J_apoc_import_file_use__neo4j__config=true

[davidburkitt]

I think this might be an issue with string quotes in Docker on macOS as I'm testing applications locally on a MacBook Pro. I've tried every conceivable combination of quotes/escapes/no quotes without success.

[eastlondoner]

@davidburkitt here's a screenshot from my Mac. I am using Docker Desktop for Mac. You can see it log the message installing plugin 'apoc' before starting neo4j

Are you seeing any particular errors?

[davidburkitt]

@eastlondoner apologies - I should have stated; I'm trying to run locally but detached 'docker-compose up -d'. It's entirely possible I'm missing the point and conceptually that's incorrect?

[eastlondoner]

@davidburkitt also fine:

Could you provide some more information on what error message you're getting or the way that it's not working?

[AliciaFrame]

If you're looking to run Graph Algorithms, that library has been deprecated in favor of the GDS library - https://github.com/neo4j/graph-data-science -- which has support for Neo4j 4.0 and 4.1. You should be able to add it with --env NEO4JLABS_PLUGINS=["graph-data-science"].

…

On Tue, Aug 11, 2020 at 9:45 AM davidburkitt ***@***.***> wrote: @eastlondoner <https://github.com/eastlondoner> apologies - I should have stated; I'm trying to run locally but detached 'docker-compose up -d'. It's entirely possible I'm missing the point and conceptually that's incorrect? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#176 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALQYDTNS7FPUU6CM33FOUIDSAFDPFANCNFSM4HZ5X2XQ> .

-- *Alicia Frame* +1.415.636.6771 alicia.frame@neo4j.com [image: The Graph Platform Company] <https://neo4j.com/>

[davidburkitt]

@eastlondoner Your example works perfectly - ignore my previous - must have been a local versioning issue maybe. Removing quoting/escapes from the env vars was the fix. Thanks