Doesn't work on authenticated Datasette instances

[simonw]

Tried this plugin on Datasette Cloud and got this error:

The problem is here:

datasette-enrichments/datasette_enrichments/views.py

Lines 38 to 42 in 96cbf51

	stuff = (
	await datasette.client.get(
	datasette.urls.table(database, table, "json") + "?" + query_string
	)
	).json()

datasette-enrichments/datasette_enrichments/views.py

Lines 87 to 91 in 96cbf51

	stuff = (
	await datasette.client.get(
	datasette.urls.table(database, table, "json") + "?" + query_string
	)
	).json()

https://github.com/datasette/datasette-enrichments/blob/96cbf510115b3ab738360f88a6e7e8070575a171/datasette_enrichments/__init__.py#L79C1-L79C1

Those calls to datasette.client.get() don't attempt to pass through authentication information, so a Datasette instance with extra authentication plugins installed refuses them.

[simonw]

Options:

1. Pass through cookies and potentially other authentication headers too. I'll try this first.
2. Modify Datasette itself to make it easier to call datasette.client.get() etc while passing through authentication credentials.

[simonw]

I wrote this function:

async def get_with_auth(datasette, request, *args, **kwargs):
    cookies = kwargs.pop("cookies") or {}
    headers = kwargs.pop("headers") or {}
    # Copy across cookies from request
    for key, value in request.cookies.items():
        if key not in cookies:
            cookies[key] = value
    # Also the authorization header, if set
    if "authorization" in request.headers:
        headers["authorization"] = request.headers["authorization"]
    kwargs["cookies"] = cookies
    kwargs["headers"] = headers
    return await datasette.client.get(*args, **kwargs)

But it's not going to work! Because of this code:

datasette-enrichments/datasette_enrichments/__init__.py

Lines 70 to 133 in 96cbf51

	async def enqueue(
	self, datasette, db, table, filter_querystring, config, actor_id=None
	):
	# Enqueue a job
	qs = filter_querystring
	if qs:
	qs += "&"
	qs += "_size=0&_extra=count"
	table_path = datasette.urls.table(db.name, table)
	response = await datasette.client.get(table_path + ".json" + "?" + qs)
	row_count = response.json()["count"]
	await db.execute_write(CREATE_JOB_TABLE_SQL)
	def _insert(conn):
	with conn:
	cursor = conn.execute(
	"""
	insert into _enrichment_jobs (
	enrichment, status, database_name, table_name, filter_querystring,
	config, started_at, row_count, error_count, done_count, cost_100ths_cent, actor_id
	) values (
	:enrichment, 'p', :database_name, :table_name, :filter_querystring, :config,
	datetime('now'), :row_count, 0, 0, 0{}
	)
	""".format(
	", :actor_id" if actor_id else ", null"
	),
	{
	"enrichment": self.slug,
	"database_name": db.name,
	"table_name": table,
	"filter_querystring": filter_querystring,
	"config": json.dumps(config or {}),
	"row_count": row_count,
	"actor_id": actor_id,
	},
	)
	return cursor.lastrowid
	job_id = await db.execute_write_fn(_insert)
	if self.runs_in_process:
	await self.start_enrichment_in_process(datasette, db, job_id)
	async def start_enrichment_in_process(self, datasette, db, job_id):
	loop = asyncio.get_event_loop()
	job_row = (
	await db.execute("select * from _enrichment_jobs where id = ?", (job_id,))
	).first()
	if not job_row:
	return
	job = dict(job_row)
	async def run_enrichment():
	next_cursor = job["next_cursor"]
	while True:
	# Get next batch
	table_path = datasette.urls.table(
	job["database_name"], job["table_name"], format="json"
	)
	qs = job["filter_querystring"]
	if next_cursor:
	qs += "&_next={}".format(next_cursor)
	qs += "&_size={}".format(self.batch_size)
	response = await datasette.client.get(table_path + "?" + qs)

Neither of those methods have access to the request object - because enrichments are designed to run in-process but outside of the user's request.

Authentication is considered when a user first queues up an enrichment, but after that point the code runs independently and needs to be able to fetch new rows without adding extra authentication headers.

Options for solving this:

• Stop using datasette.client.get("/db/table.json?...") to fetch new rows - go straight to the database instead. This is frustrating because I'll have to re-implement pagination etc.
• Modify Datasette core to give datasette.client a way to bypass authentication. This is harder.
• Maybe figure out a way to call the existing Datasette TableView mechanism in a way that bypasses authentication.

Or... there may be one more option. I could define my own permission_allowed() hook in this plugin which uses some kind of internal criteria unique to this plugin, such that calls to datasette.client.get() can complete here without having to intefere with other aspects of Datasette's existing auth system.

I'm going to experiment with that path first.

[simonw]

That seems to work!

[simonw]

Got this working on Datasette Cloud now.