Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Datasette Plugins #14

Closed
simonw opened this issue Oct 23, 2017 · 22 comments
Closed

Datasette Plugins #14

simonw opened this issue Oct 23, 2017 · 22 comments
Labels

Comments

@simonw
Copy link
Owner

simonw commented Oct 23, 2017

It would be neat if additional functionality could be opted-in to the system in the form of easy-to-add plugins, hosted as separate packages. First example: a Google Analytics plugin, which adds GA tracking code with your tracking ID to the web interface for your dataset.

This may be an opportunity to experiment with entry points: http://amir.rachum.com/blog/2017/07/28/python-entry-points/

@simonw simonw added this to the V2: visualization edition milestone Oct 23, 2017
@simonw
Copy link
Owner Author

simonw commented Nov 11, 2017

The plugin system can also allow alternative providers for the publish command - e.g. maybe hook up hyper.sh as an option for publishing containers.

@simonw simonw changed the title Some kind of plugin system Plugin system Nov 14, 2017
@simonw
Copy link
Owner Author

simonw commented Nov 14, 2017

Plugins should be able to interact with the build step. This would give plugins an opportunity to modify the SQL databases and help prepare them for serving - for example, a full-text search plugin might create additional FTS tables, or a mapping plugin might pre-calculate a bunch of geohashes for tables that have latitude/longitude values. Plugins could really take advantage of the immutable nature of the dataset here.

@simonw
Copy link
Owner Author

simonw commented Nov 16, 2017

For visualizations, Google Maps should be made available as a plugin. The default visualizations can use Leaflet and Open Street Map, but there's no reason to not make Google Maps available as a plugin, especially if the plugin can provide a mechanism for configuring the necessary API key.

I'm particularly excited in the Google Maps heatmap visualization https://developers.google.com/maps/documentation/javascript/heatmaplayer as seen on http://mochimachine.org/wasteland/

@simonw
Copy link
Owner Author

simonw commented Nov 21, 2017

@jacobian
Copy link
Contributor

I'd also suggest taking a look at stevedore, which has a ton of tools for doing plugin stuff. I've had good luck with it in the past.

@simonw
Copy link
Owner Author

simonw commented Nov 22, 2017

Oh thanks, that definitely looks like an interesting option.

@simonw simonw changed the title Plugin system Datasette Plugins Apr 15, 2018
@simonw
Copy link
Owner Author

simonw commented Apr 15, 2018

I started a thread on Twitter asking people for good examples of Python projects with a strong plugin ecosystem: https://twitter.com/simonw/status/985377670388105216

The most impressive example that came back was pytest - which now has nearly 400 plugins: https://plugincompat.herokuapp.com/

The pytest plugin infrastructure is available as an independent package called pluggy - which appears to offer everything I need for Datasette. I'm going to give that a go and see how well it works: https://pluggy.readthedocs.io/en/latest/

@simonw
Copy link
Owner Author

simonw commented Apr 15, 2018

Datasette 1.0 will be the release of Datasette that attempts to provide a stable plugin API: https://github.com/simonw/datasette/milestone/7

There's a lot of work to be done before then, but as a starting point I'm going to support two very simple extension mechanisms:

  • Template system plugins - where the hook gets passed the Jinja environment and can freely register new template tags and filters
  • SQLite connection plugins - where the hook gets passed a new SQLite connection and can register custom SQLite functions

The template system hook will go near here:

datasette/datasette/app.py

Lines 1225 to 1228 in efbb4e8

self.jinja_env.filters['escape_css_string'] = escape_css_string
self.jinja_env.filters['quote_plus'] = lambda u: urllib.parse.quote_plus(u)
self.jinja_env.filters['escape_sqlite'] = escape_sqlite
self.jinja_env.filters['to_css_class'] = to_css_class

The SQLite connection hook will go near here:

datasette/datasette/app.py

Lines 1094 to 1098 in efbb4e8

def prepare_connection(self, conn):
conn.row_factory = sqlite3.Row
conn.text_factory = lambda x: str(x, 'utf-8', 'replace')
for name, num_args, func in self.sqlite_functions:
conn.create_function(name, num_args, func)

These two feel simple enough that I'm not worried that I might design an API that I later regret.

@simonw
Copy link
Owner Author

simonw commented Apr 15, 2018

Tox is a good example of a project that uses pluggy in the way I want to use it (function hooks rather than classes): https://github.com/tox-dev/tox/blob/master/tox/hookspecs.py

simonw added a commit that referenced this issue Apr 15, 2018
Uses pluggy: https://pluggy.readthedocs.io/

Two example plugins - an uppercase template filter and a convert_units() SQL function.
@simonw
Copy link
Owner Author

simonw commented Apr 15, 2018

OK, from that prototype in f2720b0 it looks like pluggy provides a solid path forward.

Next steps:

  • Build a demo plugin that uses setuptools entrypoints to register with the datasette plugin manager via pluggy
  • Figure out a mechanism for registering plugins without first needing to publish them to PyPI. Can I load plugins from a special plugins/ directory similar to the --template-dir=templates/ option already supported by Datasette? Load plugins from a --plugins-dir=plugins/ directory #211

@simonw
Copy link
Owner Author

simonw commented Apr 15, 2018

Here's a demo of the convert_units() SQL function I prototyped in f2720b0

2018-04-15 at 4 23 pm

@simonw
Copy link
Owner Author

simonw commented Apr 15, 2018

Once I've got the plugins mechanism stable and people start releasing plugins it would be useful to have a dedicated Trove classifier on PyPI for Datasette plugins - Framework :: Datasette for example.

This would help me build a Datasette equivalent of the http://plugincompat.herokuapp.com/ site, which works by scanning PyPI for items with the Framework :: Pytest classifier:

https://github.com/pytest-dev/plugincompat/blob/8bdf1a6fb82807091ece0c68c196103ee8270194/update_index.py#L52-L53

It looks like the mechanism for requesting new PyPI classifiers is to file a ticket against warehouse, like these ones: pypi/warehouse#3570 and pypi/warehouse#2881

@simonw
Copy link
Owner Author

simonw commented Apr 16, 2018

I created https://github.com/simonw/datasette-plugin-demos which is now published to PyPI and can be installed with pip install datasette-plugin-demos - I've confirmed that if you DO install it my Datasette plugins branch picks up the plugins, and select random_integer(1, 4) works as it should.

@simonw
Copy link
Owner Author

simonw commented Apr 16, 2018

Slight code design problem... when I tried installing my branch in a fresh virtual environment I got this error, because setup.py now depends on pluggy (from importing __version__):

      File "/private/var/folders/jj/fngnv0810tn2lt_kd3911pdc0000gp/T/pip-req-build-dftqdezt/setup.py", line 2, in <module>
        from datasette import __version__
      File "/private/var/folders/jj/fngnv0810tn2lt_kd3911pdc0000gp/T/pip-req-build-dftqdezt/datasette/__init__.py", line 2, in <module>
        from .hookspecs import hookimpl # noqa
      File "/private/var/folders/jj/fngnv0810tn2lt_kd3911pdc0000gp/T/pip-req-build-dftqdezt/datasette/hookspecs.py", line 1, in <module>
        from pluggy import HookimplMarker
    ModuleNotFoundError: No module named 'pluggy'

Looks like I've run into point 6 on https://packaging.python.org/guides/single-sourcing-package-version/ :

2018-04-15 at 5 34 pm

simonw added a commit that referenced this issue Apr 16, 2018
Running `from datasette import __version__` in `setup.py` was throwing
an error `ModuleNotFoundError: No module named 'pluggy'`

See https://packaging.python.org/guides/single-sourcing-package-version/

Refs #14
simonw pushed a commit that referenced this issue Apr 16, 2018
Uses https://pluggy.readthedocs.io/ originally created for the py.test project

We're starting with two plugin hooks:

prepare_connection(conn)

This is called when a new SQLite connection is created. It can be used to register custom SQL functions.

prepare_jinja2_environment(env)

This is called with the Jinja2 environment. It can be used to register custom template tags and filters.

An example plugin which uses these two hooks can be found at https://github.com/simonw/datasette-plugin-demos or installed using `pip install datasette-plugin-demos`

Refs #14
@simonw
Copy link
Owner Author

simonw commented Apr 16, 2018

I should check if it's possible to have two template registration function plugins in a single plugin module. If it isn't maybe I should use class plugins instead of module plugins.

@simonw
Copy link
Owner Author

simonw commented Apr 16, 2018

Annoyingly, the following only results in the last of the two prepare_connection hooks being registered:

from datasette import hookimpl
import pint
import random

ureg = pint.UnitRegistry()


@hookimpl
def prepare_connection(conn):
    def convert_units(amount, from_, to_):
        "select convert_units(100, 'm', 'ft');"
        return (amount * ureg(from_)).to(to_).to_tuple()[0]
    conn.create_function('convert_units', 3, convert_units)


@hookimpl
def prepare_connection(conn):
    conn.create_function('random_integer', 2, random.randint)

@simonw
Copy link
Owner Author

simonw commented Apr 16, 2018

I think that's OK. The two plugins I've implemented so far (prepare_connection and prepare_jinja2_environment) both make sense if they can only be defined once-per-plugin. For the moment I'll assume I can define future hooks to work well with the same limitation.

The syntactic sugar idea in #220 can help here too.

@simonw simonw changed the title Datasette Plugins First working version of Datasette Plugins Apr 17, 2018
@simonw
Copy link
Owner Author

simonw commented Apr 17, 2018

I just shipped Datasette 0.19 with where I'm at so far: https://github.com/simonw/datasette/releases/tag/0.19

@simonw
Copy link
Owner Author

simonw commented Apr 18, 2018

I added a mechanism for plugins to serve static files and define custom CSS and JS URLs in #214 - see new documentation on http://datasette.readthedocs.io/en/latest/plugins.html#static-assets and http://datasette.readthedocs.io/en/latest/plugins.html#extra-css-urls

simonw added a commit that referenced this issue Apr 19, 2018
@simonw simonw changed the title First working version of Datasette Plugins Datasette Plugins Apr 20, 2018
@simonw
Copy link
Owner Author

simonw commented Apr 20, 2018

I released everything we have so far in Datasette 0.20 and built and released an example plugin, datasette-cluster-map. Here's my blog entry about it: https://simonwillison.net/2018/Apr/20/datasette-plugins/

@simonw
Copy link
Owner Author

simonw commented Apr 20, 2018

@simonw simonw modified the milestones: Visualization edition, Datasette 1.0 Jun 20, 2018
simonw added a commit that referenced this issue Jul 26, 2018
This change introduces a new plugin hook, publish_subcommand, which can be
used to implement new subcommands for the "datasette publish" command family.

I've used this new hook to refactor out the "publish now" and "publish heroku"
implementations into separate modules. I've also added unit tests for these
two publishers, mocking the subprocess.call and subprocess.check_output
functions.

As part of this, I introduced a mechanism for loading default plugins. These
are defined in the new "default_plugins" list inside datasette/app.py

Closes #217 (Plugin support for datasette publish)
Closes #348 (Unit tests for "datasette publish")
Refs #14, #59, #102, #103, #146, #236, #347
simonw pushed a commit that referenced this issue Jul 26, 2018
… heroku/now (#349)

This change introduces a new plugin hook, publish_subcommand, which can be
used to implement new subcommands for the "datasette publish" command family.

I've used this new hook to refactor out the "publish now" and "publish heroku"
implementations into separate modules. I've also added unit tests for these
two publishers, mocking the subprocess.call and subprocess.check_output
functions.

As part of this, I introduced a mechanism for loading default plugins. These
are defined in the new "default_plugins" list inside datasette/app.py

Closes #217 (Plugin support for datasette publish)
Closes #348 (Unit tests for "datasette publish")
Refs #14, #59, #102, #103, #146, #236, #347
russss added a commit to russss/datasette that referenced this issue Apr 28, 2019
This adds two new plugin hooks:

The `inspect` hook allows plugins to add data to the inspect
dictionary.

The `prepare_sanic` hook allows plugins to hook into the web
router. I've attached a warning to this hook in the docs in light
of simonw#272 but I want this hook now...

On quick inspection, I don't think it's worthwhile to try and make
this hook independent of the web framework (but it looks like Starlette
would make the hook implementation a bit nicer).

Ref simonw#14
russss added a commit to russss/datasette that referenced this issue Apr 29, 2019
This adds two new plugin hooks:

The `inspect` hook allows plugins to add data to the inspect
dictionary.

The `prepare_sanic` hook allows plugins to hook into the web
router. I've attached a warning to this hook in the docs in light
of simonw#272 but I want this hook now...

On quick inspection, I don't think it's worthwhile to try and make
this hook independent of the web framework (but it looks like Starlette
would make the hook implementation a bit nicer).

Ref simonw#14
@simonw simonw removed this from the Datasette 1.0 milestone May 13, 2019
@simonw
Copy link
Owner Author

simonw commented May 13, 2019

We've grown a bunch of plugin hooks over the past two years: https://datasette.readthedocs.io/en/latest/plugins.html#plugin-hooks

Since the plugin system will never be 100% "finished", I'm closing this in favor of the label: plugins

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants