Support for SQLite Encryption Extension (SEE) #509

barnettben · 2019-04-03T08:12:11Z

This pull request adds code to support the SQLite Encryption Extension.
(see #506)

The changes are almost all additive and mostly involve adding functions to set/change a key.

All code for this is behind the conditional compilation flag GRDB_SQLITE_SEE.

Notes/questions:

The SEE code needs a custom build of SQLite to be used. I intend to submit patches to SQLiteLib to make it easier to support this.
It can't be used at the same time as GRDBCipher and should probably throw a compiler error if both GRDB_SQLITE_SEE and GRDBCIPHER are defined. I haven't implemented this. Where would be the best place to put this?
You need a paid license from Hwaci to use the SEE code. This means that many people won't be able to run the code or tests. I have run them and it builds/tests ok here.
Documentation hasn't been updated yet. I will submit changes to the custom builds guide before asking for a merge.
This is my first contribution to a well-used open source project and would be grateful for any suggestions for improvement

Only expose SQLCipher validation code if we are using SQLCipher and not another encryption extension.

It’s all hidden behind a new flag called GRDB_SQLITE_SEE. This must be defined along with SQLITE_HAS_CODEC.

Also renamed existing SQLCipher tests to make distinction clear.

groue · 2019-04-03T11:48:36Z

Thank you very much @BenRB!

All code for this is behind the conditional compilation flag GRDB_SQLITE_SEE.

Notes/questions:

The SEE code needs a custom build of SQLite to be used. I intend to submit patches to SQLiteLib to make it easier to support this.

All right. This can be done independently from this pull request And I'll be happy merging a future PR with an updated SQLiteLib.

It can't be used at the same time as GRDBCipher and should probably throw a compiler error if both GRDB_SQLITE_SEE and GRDBCIPHER are defined. I haven't implemented this. Where would be the best place to put this?

Do you think it's a mistake we should expect people to do, and thus plan for with an explicit error? If not, I don't think this is really necessary.

You need a paid license from Hwaci to use the SEE code. This means that many people won't be able to run the code or tests. I have run them and it builds/tests ok here.

Yes. I hope you will tell us when a GRDB change inadvertently breaks SEE builds 😅. I'll take care of avoiding the most obvious mistakes. Consider SEE as an officially supported platform now 👍

Documentation hasn't been updated yet. I will submit changes to the custom builds guide before asking for a merge.

Thank you! A documentation update will be very much appreciated.

This is my first contribution to a well-used open source project and would be grateful for any suggestions for improvement

I'll have a detailed look very shortly. What I see looks like a very fine first contribution 🤝

Missing from last commit

groue

This is a very good pull request. And thank you for fitting in the existing code base, this is really appreciated. I suggest a few changes that I hope you will agree on.

GRDB/Core/DatabasePool.swift

GRDB/Core/Database.swift

GRDBCustom.xcodeproj/project.pbxproj

GRDB/Core/Configuration.swift

Tests/GRDBTests/SQLiteEncryptionExtensionTests.swift

barnettben · 2019-04-03T17:55:22Z

Thank you for taking the time to provide feedback, it's much appreciated. I will implement the changes and push updates to this branch.

EncryptionType -> Algorithm passphrase -> key Tidy enum values

groue · 2019-04-04T05:58:07Z

Thanks for the changes!

What should we do now? Keep this pull request open until the SQLiteLib update, so that we can update the documentation with accurate information?

barnettben · 2019-04-04T20:01:23Z

I've updated the CustomSQLiteBuilds documentation and noted that Swift 5 is required to use SEE.

I've also submitted a pull request to SQLiteLib: swiftlyfalling/SQLiteLib#40

Hopefully that will be accepted and all the pieces will be in place to allow this to be merged and closed.

groue · 2019-04-05T07:12:01Z

OK, let's wait for @swiftlyfalling!

groue · 2019-04-10T11:50:57Z

@BenRB, I've just merged #508, which unfortunately brings conflicts with your pull request. Tell me if you have any trouble merging or rebasing GRDB-4.0 in your branch: I'll easily overcome the difficulty since I'm familiar with both pull requests.

I hope @swiftlyfalling will soon look at swiftlyfalling/SQLiteLib#40. I know how to contact him by email, but I think we should wait a couple more days before I do.

Only expose SQLCipher validation code if we are using SQLCipher and not another encryption extension.

It’s all hidden behind a new flag called GRDB_SQLITE_SEE. This must be defined along with SQLITE_HAS_CODEC.

Also renamed existing SQLCipher tests to make distinction clear.

Missing from last commit

EncryptionType -> Algorithm passphrase -> key Tidy enum values

Don’t bother testing for error descriptions, these might change. Just check error codes and messages from SQLite as these remain more constant.

… sqlite-see-support # Conflicts: # GRDB/Core/Configuration.swift # GRDB/Core/Database.swift # Tests/GRDBTests/SQLiteEncryptionExtensionTests.swift

barnettben · 2019-04-12T07:37:34Z

I've rebased my changes onto the new GRDB-4.0 branch and checked that the tests pass

groue

Thanks for rebasing your changes, @BenRB!

Something went off, though, because your pull request does not just add commits to GRDB-4.0. It looks like it was forked from some previous commit. This is not a big problem, but this has hidden some issues in the rebase:

Your pull requests adds five methods that are no longer used and should be removed:

static func activateExtendedCodes
static func validateSQLCipher
static func set(passphrase: forConnection:)
static func set(cipherPageSize: forConnection:)
static func set(kdfIterations: forConnection:)

The GitHub diff does not show this very clearly, because of the weird rebasing. But I was able to spot it very clearly by checking out your branch, and diffing from the head of GRDB-4.0.

Meanwhile, I contact @swiftlyfalling.

GRDB/Core/Database.swift

groue · 2019-04-17T07:46:59Z

I still have no news from @swiftlyfalling.

I don't see myself fork SQLiteLib and commit to its maintenance. I'm not comfortable enough with low-level Xcode project configuration. GRDB support for custom SQLite builds would become a pain point, and this is the highest danger for a free open source library with a very very little core team (read: me).

So we may have to ship GRDB 4 without SEE support, depending on my own schedule.

I'm not very worried about your particular setup, @BenRB, because you show a great autonomy. But I would be pleased to add SEE to the "officially" supported platforms.

barnettben · 2019-04-21T07:29:26Z

I have made the requested changes and removed the extra code left by the bad rebase.

If it is OK with you, I am happy to wait for SQLiteLib to be updated (assuming @swiftlyfalling is OK with my change!). Until then, I'm fine to use my own fork of GRDB for my projects and keep updating this PR as needed.

groue · 2019-05-27T08:42:02Z

@BenRB, do you have any update on this PR?

swiftlyfalling · 2019-05-28T20:13:24Z

@groue, @BenRB: Apologies for not posting an update here as well - swiftlyfalling/SQLiteLib#40 was merged a little while ago, so should be all set there.

groue · 2019-05-29T07:03:23Z

Hello @swiftlyfalling, it's good to hear from you :-) Let's wait for @BenRB, there is no rush. And thanks for SQLite 3.28 in swiftlyfalling/SQLiteLib#41 👍

groue · 2019-06-13T19:36:27Z

Oops, I did not intend to close this PR, but I deleted the target GRDB-4.0 branch. @BenRB, if you intend to complete the support for SEE, please reopen and target the development branch.

barnettben added 3 commits March 31, 2019 08:50

Update conditions for SQLCipher validation

535c74a

Only expose SQLCipher validation code if we are using SQLCipher and not another encryption extension.

Add support for SQLite Encryption Extension

c174af8

It’s all hidden behind a new flag called GRDB_SQLITE_SEE. This must be defined along with SQLITE_HAS_CODEC.

Add tests to check new encryption code

22bbc1f

Also renamed existing SQLCipher tests to make distinction clear.

groue mentioned this pull request Apr 3, 2019

Suggested change to support SQLite Encryption Extension #506

Closed

Add renamed tests to other targets

02385ea

Missing from last commit

groue requested changes Apr 3, 2019

View reviewed changes

barnettben added 2 commits April 3, 2019 20:58

Renaming things for consistency

22fde35

EncryptionType -> Algorithm passphrase -> key Tidy enum values

Rename test suites for encryption

e4d32b2

groue approved these changes Apr 4, 2019

View reviewed changes

groue added the enhancement label Apr 4, 2019

groue changed the title ~~Sqlite see support~~ Support for SQLite Encryption Extension (SEE) Apr 4, 2019

barnettben mentioned this pull request Apr 4, 2019

Include SQLite-SEE if present in project swiftlyfalling/SQLiteLib#40

Merged

Document including SEE in GRDBCustomSQLite builds

e6d7acb

groue added this to the GRDB 4.0.0 milestone Apr 5, 2019

groue mentioned this pull request Apr 5, 2019

GRDB 4.0 #479

Merged

26 tasks

barnettben added 9 commits April 12, 2019 07:56

Update conditions for SQLCipher validation

fbbfc0e

Only expose SQLCipher validation code if we are using SQLCipher and not another encryption extension.

Add support for SQLite Encryption Extension

f68dabe

It’s all hidden behind a new flag called GRDB_SQLITE_SEE. This must be defined along with SQLITE_HAS_CODEC.

Add tests to check new encryption code

01885a1

Also renamed existing SQLCipher tests to make distinction clear.

Add renamed tests to other targets

6eaf5c8

Missing from last commit

Renaming things for consistency

8f33112

EncryptionType -> Algorithm passphrase -> key Tidy enum values

Rename test suites for encryption

59f5316

Document including SEE in GRDBCustomSQLite builds

63f9ccc

Fix merge/rebase artifacts

48efabd

Fix incorrectly failing tests

60f3db5

Don’t bother testing for error descriptions, these might change. Just check error codes and messages from SQLite as these remain more constant.

Merge branch 'sqlite-see-support' of github.com:benrb/GRDB.swift into…

0ec9101

… sqlite-see-support # Conflicts: # GRDB/Core/Configuration.swift # GRDB/Core/Database.swift # Tests/GRDBTests/SQLiteEncryptionExtensionTests.swift

groue requested changes Apr 12, 2019

View reviewed changes

GRDB/Core/Database.swift Outdated Show resolved Hide resolved

Make SEE code match style of SQLCipher code

4f695f1

groue removed this from the GRDB 4.0.0 milestone May 27, 2019

groue closed this Jun 13, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for SQLite Encryption Extension (SEE) #509

Support for SQLite Encryption Extension (SEE) #509

barnettben commented Apr 3, 2019 •

edited by groue

Loading

groue commented Apr 3, 2019

groue left a comment

barnettben commented Apr 3, 2019

groue commented Apr 4, 2019

barnettben commented Apr 4, 2019

groue commented Apr 5, 2019

groue commented Apr 10, 2019 •

edited

Loading

barnettben commented Apr 12, 2019

groue left a comment •

edited

Loading

groue commented Apr 17, 2019 •

edited

Loading

barnettben commented Apr 21, 2019

groue commented May 27, 2019

swiftlyfalling commented May 28, 2019

groue commented May 29, 2019

groue commented Jun 13, 2019

Support for SQLite Encryption Extension (SEE) #509

Support for SQLite Encryption Extension (SEE) #509

Conversation

barnettben commented Apr 3, 2019 • edited by groue Loading

groue commented Apr 3, 2019

groue left a comment

Choose a reason for hiding this comment

barnettben commented Apr 3, 2019

groue commented Apr 4, 2019

barnettben commented Apr 4, 2019

groue commented Apr 5, 2019

groue commented Apr 10, 2019 • edited Loading

barnettben commented Apr 12, 2019

groue left a comment • edited Loading

Choose a reason for hiding this comment

groue commented Apr 17, 2019 • edited Loading

barnettben commented Apr 21, 2019

groue commented May 27, 2019

swiftlyfalling commented May 28, 2019

groue commented May 29, 2019

groue commented Jun 13, 2019

barnettben commented Apr 3, 2019 •

edited by groue

Loading

groue commented Apr 10, 2019 •

edited

Loading

groue left a comment •

edited

Loading

groue commented Apr 17, 2019 •

edited

Loading