GPKG: add a NOLOCK=YES option to open a file without any lock (for read-only access

[rouault]

helps fixing qgis/QGIS#23991, but requires QGIS changes as well

[nyalldawson]

This is exciting!

The code looks good to me, and given that it's opt in it's safe to merge.

That said, boy it's hard to find any discussion/documentation about this option anywhere, and whether there's any hidden nasties it exposes. 😮 ! But I did see that grass was investigating it here too: OSGeo/grass#1667 . Was this PR a result of the GRASS change?

[rouault]

Was this PR a result of the GRASS change?

no, this is an independant finding.

I experimented with the following python script to explore locking issues:

import os
import sqlite3

if os.path.exists('example.db'):
    os.unlink('example.db')

con = sqlite3.connect('example.db')
con.execute("CREATE TABLE foo(bar TEXT)")
con.execute("INSERT INTO foo VALUES ('1')")
con.commit()
con.close()

con = sqlite3.connect('example.db')
#con = sqlite3.connect('file:example.db?nolock=1', uri=True)
cur = con.cursor()
#cur.execute("PRAGMA journal_mode=WAL")
cur.execute("SELECT * FROM foo")

con2 = sqlite3.connect('example.db', timeout=0.5, isolation_level=None)
# The following will timeout unless WAL is enabled before the SELECT
# or con is opened in nolock=1 mode
con2.execute("INSERT INTO foo VALUES ('3')")

print(cur.fetchall())

The potential downside I'm aware of of the combination of this PR + the corresponding QGIS one is that read-only connections (opened in nolock mode) could potentially see corrupted content (or have sqlite returns an error) if reading while a file is modified. But that is restricted to connections before the file is edited, since when a file is edited in QGIS, WAL is enabled, and future read-only connections will cooperate with WAL and don't use nolock mode (since sqlite actually refuses to open a nolock connection on a WAL file as demonstrated by below snippet:)

import os
import sqlite3

if os.path.exists('example.db'):
    os.unlink('example.db')

con = sqlite3.connect('example.db')
con.execute("PRAGMA journal_mode=WAL")
con.execute("CREATE TABLE foo(bar TEXT)")
con.execute("INSERT INTO foo VALUES ('1')")
con.commit()
con.close()

con2 = sqlite3.connect('file:example.db?nolock=1', uri=True)
cur2 = con2.cursor()
cur2.execute("SELECT * FROM foo") # exception: sqlite3.OperationalError: unable to open database file

[nyalldawson]

The potential downside I'm aware of of the combination of this PR + the corresponding QGIS one is that read-only connections (opened in nolock mode) could potentially see corrupted content (or have sqlite returns an error) if reading while a file is modified.

That sounds like a perfectly reasonable trade off to me.

[github-actions]

The backport to release/3.4 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-release/3.4 release/3.4
# Navigate to the new working tree
cd .worktrees/backport-release/3.4
# Create a new branch
git switch --create backport-5207-to-release/3.4
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick --mainline 1 82408860c8b74a05a4fbc42effcc5376c0520343
# Push it to GitHub
git push --set-upstream origin backport-5207-to-release/3.4
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-release/3.4

Then, create a pull request where the base branch is release/3.4 and the compare/head branch is backport-5207-to-release/3.4.