Master

.github/workflows/CI.yml

@@ -0,0 +1,60 @@
+name: SQLite FDW test
+
+on:
+  workflow_dispatch:
+  pull_request:  
+  push:
+    branches:
+      - master
+      - main
+jobs:
+  detect-pgversion:
+    runs-on: ubuntu-22.04
+    outputs:
+      pgversion: ${{ steps.detect-pgversion.outputs.targets }}
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: detect-pgversion
+        id: detect-pgversion
+        run: |
+          targets=`bash GitHubActions/detect_targets.sh`
+          echo "targets=$targets" >> $GITHUB_OUTPUT
+
+  test:
+    needs: detect-pgversion
+    env:
+      SQLITE_VERSION : "3420000"
+      SQLITE_YEAR: "2023"
+      HTTP_PROXY: ""
+      HTTPS_PROXY: ""
+    strategy:
+      fail-fast: false
+      matrix:
+        pg: ${{ fromJSON(needs.detect-pgversion.outputs.pgversion) }}
+
+    name: Test on PostgreSQL ${{ matrix.pg }}
+    runs-on: ubuntu-22.04
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: tar
+        run: tar zcvf sqlite_fdw.tar.gz ./*
+
+      - name: set_proxy
+        run: bash GitHubActions/env.sh
+
+      - name: install locales
+        run: bash GitHubActions/install_locales.sh
+
+      - name: build PostgreSQL ${{ matrix.pg }}
+        run: bash GitHubActions/build_postgres.sh ${{ matrix.pg }}
+
+      - name: install SQLite
+        run: bash GitHubActions/install_sqlite.sh ${{ env.SQLITE_VERSION }} ${{ env.SQLITE_YEAR }}
+
+      - name: build sqlite_fdw
+        run: bash GitHubActions/build_sqlite_fdw.sh ${{ matrix.pg }}
+
+      - name: execute sqlite_fdw test
+        run: bash GitHubActions/execute_test.sh ${{ matrix.pg }} 

GitHubActions/README.md

@@ -0,0 +1,10 @@
+# CI environment of sqlite_fdw.
+
+Tests will be executed automatically when commited to main/master branch and when a pull request was opened/updated.
+It is realized by using GitHub actions.
+
+The CI process is defined in .github/workflows/CI.yml file.
+Scripts in this directory (GitHubActions/*.sh) are referred by CI.yml. 
+
+The regression test will be executed for multi-versions of PostgreSQL.
+Target versions are determined automatically based on directory names in "sql" directory.

GitHubActions/build_postgres.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+
+################################################################################
+#
+# This script downloads PostgreSQL from the official web site into ./workdir
+# then builds it.
+#
+# Usage: ./build_postgres.sh pg_version
+#     pg_version is a PostgreSQL version to be installed like 16.0.
+#
+# Requirements
+# - be able to connect to the PostgreSQL official web site by curl.
+#
+################################################################################
+
+VERSION=$1
+mkdir -p ./workdir
+cd ./workdir
+curl -O https://ftp.postgresql.org/pub/source/v${VERSION}/postgresql-${VERSION}.tar.bz2
+tar xjf postgresql-${VERSION}.tar.bz2
+cd postgresql-${VERSION}
+./configure
+make

GitHubActions/build_sqlite_fdw.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+################################################################################
+#
+# This script builds sqlite_fdw in PostgreSQL source tree.
+#
+# Usage: ./build_sqlite_fdw.sh pg_version
+#     pg_version is a PostgreSQL version like 16.0 to be built in.
+#
+# Requirements
+# - the source code of sqlite_fdw is available by git clone.
+# - the source code of PostgreSQL is located in ~/workdir/postgresql-{pg_version}.
+# - SQLite development package is installed in a system.
+################################################################################
+
+VERSION=$1
+mkdir -p ./workdir/postgresql-${VERSION}/contrib/sqlite_fdw
+tar zxvf ./sqlite_fdw.tar.gz -C ./workdir/postgresql-${VERSION}/contrib/sqlite_fdw/
+cd ./workdir/postgresql-${VERSION}/contrib/sqlite_fdw
+export LD_LIBRARY_PATH=/usr/local/lib/:$LD_LIBRARY_PATH
+make

GitHubActions/detect_targets.sh

@@ -0,0 +1,30 @@
+#!/bin/bash
+
+################################################################################
+#
+# This script detects target PostgreSQL versions for sqlite_fdw testing from 
+# directory names in ./sql directory. Detected versions will be outputed to
+# the standard output as an array of string like ["15.4","16.0"].
+#
+# Usage: ./detect_targets.sh
+#
+# Requirements
+# - there is a directory named "sql" in a curent directory.
+#
+################################################################################
+
+dirs="./sql/*"
+pattern="[0-9]+\.[0-9]+"
+targets="["
+for pathname in $dirs; do
+	if [[ "$pathname" =~ $pattern ]]; then
+		target=`basename $pathname`
+		if [ "$targets" != "[" ]; then
+			targets+=","
+		fi
+		targets+="\"$target\""
+	fi
+done
+targets+="]"
+
+echo "$targets"

GitHubActions/env.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+
+################################################################################
+#
+# This script configures apt.conf to set a proxy if an environment variable
+# HTTP_PROXY or HTTPS_PROXY is set.
+#
+# Usage: ./env.sh
+#
+# Requirements
+# - having superuser privileges
+#
+################################################################################
+
+if [ -z $HTTP_PROXY ] && [ "$HTTP_PROXY" != "" ]; then
+	echo 'Acquire::http::proxy "$HTTP_PROXY";' | sudo tee /etc/apt/apt.conf
+fi
+if [ -z $HTTPS_PROXY ] && [ "$HTTPS_PROXY" != "" ]; then
+	echo 'Acquire::https::proxy "$HTTPS_PROXY";' | sudo tee -a /etc/apt/apt.conf
+fi

GitHubActions/execute_test.sh

@@ -0,0 +1,42 @@
+#!/bin/bash
+
+################################################################################
+#
+# This script executes a regression test pf sqlite_fdw by calling test.sh in
+# sqlite_fdw. If all tests are passed, this script will exit successfully.
+# Otherwise, it will exit with failure.
+
+# Usage: ./execute_test.sh pg_version
+#     pg_version is a PostgreSQL version to be tested like 16.0.
+#
+# Requiremets
+# - the source code of PostgreSQL is located in ./workdir/postgresql-{pg_version}.
+# - the source code of sqlite_fdw is loacted in ./workdir/postgresql-{pg_version}/contrib/sqlite_fdw.
+# - PostgreSQL and sqlite_fdw were built.
+# - this script assumes that tests are passed if this file (created by executing
+#   the test) contains " ALL {number} tests passed" at the last or the 3rd line
+#   from the end.
+#
+################################################################################
+
+VERSION=$1
+cd ./workdir/postgresql-${VERSION}/contrib/sqlite_fdw
+chmod +x ./test.sh
+./test.sh
+
+last_line=$(tail -n 1 make_check.out)
+third_line_from_the_last=$(tail -n 3 make_check.out | head -n 1)
+
+pattern=" All [0-9]+ tests passed.+"
+
+if [[ "$last_line" =~ $pattern ]]; then
+	echo "last_line"
+
+elif [[ "$third_line_from_the_last" =~ $pattern ]]; then
+	 echo "$third_line_from_the_last"
+else
+	echo "Error : not All the tests passed"
+	echo "last line : '$last_line'"
+	echo "thierd_line_from_the_last : '$third_line_from_the_last'"
+	exit 1
+fi

GitHubActions/install_locales.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+################################################################################
+#
+# This script installs some locales and language packs used by sqlite_fdw
+# tests in Ubuntu.
+#
+# Usage: ./install_locales.sh
+# 
+# Requirements:
+# - having superuser privileges
+#
+################################################################################
+
+sudo apt-get update
+sudo apt-get install locales language-pack-ja
+sudo locale-gen ja_JP.EUC-JP
+sudo apt-get install language-pack-ko-base language-pack-ko
+sudo locale-gen ko_KR.EUC-KR
+sudo apt -get install language-pack-bg-base language-pack-bg
+sudo locale-gen bg_BG

GitHubActions/install_sqlite.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+
+################################################################################
+#
+# This sript downloads SQLite source code from the official web site into
+# ./workdir then builds and installs it.
+#
+# Usage: ./install_sqlite.sh version year
+#     version: SQLite version to be installed
+#     year: A year of SQLite released. It is used for determining a download URL.
+#
+#     Ex) ./install_sqlite.sh 3420000 2023
+#
+# Requirements
+# - be able to connect to the SQLite official web site by curl.
+# - having superuser privileges
+#
+################################################################################
+
+VERSION=$1
+YEAR=$2
+mkdir -p ./workdir
+cd ./workdir
+curl -O https://www.sqlite.org/${YEAR}/sqlite-src-${VERSION}.zip
+unzip sqlite-src-${VERSION}.zip
+cd sqlite-src-${VERSION}
+./configure --enable-fts5
+make
+sudo make install

Makefile

@@ -15,7 +15,7 @@ OBJS = connection.o option.o deparse.o sqlite_query.o sqlite_fdw.o sqlite_data_n
 EXTENSION = sqlite_fdw
 DATA = sqlite_fdw--1.0.sql sqlite_fdw--1.0--1.1.sql
 
-REGRESS = extra/sqlite_fdw_post extra/float4 extra/float8 extra/int4 extra/int8 extra/numeric extra/bitstring extra/bool extra/timestamp extra/uuid extra/join extra/limit extra/aggregates extra/prepare extra/select_having extra/select extra/insert extra/update extra/encodings sqlite_fdw type aggregate selectfunc 
+REGRESS = extra/sqlite_fdw_post extra/bitstring extra/bool extra/float4 extra/float8 extra/int4 extra/int8 extra/numeric extra/macaddr6 extra/macaddr8 extra/out_of_range extra/timestamp extra/uuid extra/join extra/limit extra/aggregates extra/prepare extra/select_having extra/select extra/insert extra/update extra/encodings sqlite_fdw type aggregate selectfunc 
 REGRESS_OPTS = --encoding=utf8
 
 SQLITE_LIB = sqlite3

README.md

@@ -36,11 +36,11 @@ Features
 - Support Bulk `INSERT` by using `batch_size` option
 - Support `INSERT`/`UPDATE` with generated column
 - Support `ON CONFLICT DO NOTHING`
-- Support mixed SQLite [data affinity](https://www.sqlite.org/datatype3.html) input and filtering (`SELECT`/`WHERE` usage) for such dataypes as
+- Support mixed SQLite [data affinity](https://www.sqlite.org/datatype3.html) input and filtering (`SELECT`/`WHERE` usage) for such data types as
 	- `timestamp`: `text` and `int`,
 	- `uuid`: `text`(32..39) and `blob`(16),
  	- `bool`: `text`(1..5) and `int`.
-- Support mixed SQLite [data affinity](https://www.sqlite.org/datatype3.html) output (`INSERT`/`UPDATE`) for such dataypes as
+- Support mixed SQLite [data affinity](https://www.sqlite.org/datatype3.html) output (`INSERT`/`UPDATE`) for such data types as
 	- `timestamp`: `text`(default) or `int`,
  	- `uuid`: `text`(36) or `blob`(16)(default).
 
@@ -55,7 +55,7 @@ Features
 - `mod()` is pushdowned. In PostgreSQL gives [argument-dependend data type](https://www.postgresql.org/docs/current/functions-math.html), but result from SQLite always [have `real` affinity](https://www.sqlite.org/lang_mathfunc.html#mod).
 - `upper`, `lower` and other character case functions are **not** pushed down because they does not work with UNICODE character in SQLite.
 - `WITH TIES` option is **not** pushed down.
-- Bit string `#` (XOR) operator is **not** pushed down becasuse there is no equal SQLite operator.
+- Bit string `#` (XOR) operator is **not** pushed down because there is no equal SQLite operator.
 
 ### Notes about pushdowning
 
@@ -69,7 +69,7 @@ Features
 - For `numeric` data type, `sqlite_fdw` use `sqlite3_column_double` to get value, while SQLite shell uses `sqlite3_column_text` to get value. Those 2 APIs may return different numeric value. Therefore, for `numeric` data type, the value returned from `sqlite_fdw` may different from the value returned from SQLite shell.
 - `sqlite_fdw` can return implementation-dependent order for column if the column is not specified in `ORDER BY` clause.
 - When the column type is `varchar array`, if the string is shorter than the declared length, values of type character will be space-padded; values of type `character varying` will simply store the shorter string.
-- [String literals for `boolean`](https://www.postgresql.org/docs/current/datatype-boolean.html) (`t`, `f`, `y`, `n`, `yes`, `no`, `on`, `off` etc. case insensetive) can be readed and filtred but cannot writed, because SQLite documentation recommends only `int` affinity values (`0` or `1`)  for boolean data and usually text boolean data belongs to legacy datasets.
+- [String literals for `boolean`](https://www.postgresql.org/docs/current/datatype-boolean.html) (`t`, `f`, `y`, `n`, `yes`, `no`, `on`, `off` etc. case insensitive) can be readed and filtred but cannot writed, because SQLite documentation recommends only `int` affinity values (`0` or `1`)  for boolean data and usually text boolean data belongs to legacy datasets.
 
 Also see [Limitations](#limitations)
 
@@ -300,7 +300,7 @@ sqlite_fdw_version
 Identifier case handling
 ------------------------
 
-PostgreSQL folds identifiers to lower case by default, SQLite is case insensetive by default
+PostgreSQL folds identifiers to lower case by default, SQLite is case insensitive by default
 and doesn't differ uppercase and lowercase ASCII base latin letters. It's important
 to be aware of potential issues with table and column names.
 
@@ -533,7 +533,7 @@ Limitations
 - For `sum` function of SQLite, output of `sum(bigint)` is `integer` value. If input values are big, the overflow error may occurs on SQLite because it overflow within the range of signed 64bit. For PostgreSQL, it can calculate as over the precision of `bigint`, so overflow does not occur.
 - SQLite promises to preserve the 15 most significant digits of a floating point value. The big value which exceed 15 most significant digits may become different value after inserted.
 - SQLite does not support `numeric` type as PostgreSQL. Therefore, it does not allow to store numbers with too high precision and scale. Error out of range occurs.
-- SQLite does not support special values for IEEE 754-2008 numbers such as `NaN`, `+Infinity` and `-Infinity` in SQL expressions with numeric context. Also SQLite can not store this values with `real` [affinity](https://www.sqlite.org/datatype3.html). In opposite to SQLite, PostgreSQL can store special values in columns belongs to `real` datatype family such as `float` or `double precision` and use arithmetic comparation for this values. In oppose to PostgreSQL, SQLite stores `NaN`, `+Infinity` and `-Infinity` as a text values. Also conditions with special literals (such as ` n < '+Infinity'` or ` m > '-Infinity'` ) isn't numeric conditions in SQLite and gives unexpected result after pushdowning in oppose to internal PostgreSQL calculations. During `INSERT INTO ... SELECT` or in `WHERE` conditions `sqlite_fdw` uses given by PostgreSQL standard case sensetive literals **only** in follow forms: `NaN`, `-Infinity`, `Infinity`, not original strings from `WHERE` condition. *This can caused selecting issues*.
+- SQLite does not support special values for IEEE 754-2008 numbers such as `NaN`, `+Infinity` and `-Infinity` in SQL expressions with numeric context. Also SQLite can not store this values with `real` [affinity](https://www.sqlite.org/datatype3.html). In opposite to SQLite, PostgreSQL can store special values in columns belongs to `real` datatype family such as `float` or `double precision` and use arithmetic comparation for this values. In oppose to PostgreSQL, SQLite stores `NaN`, `+Infinity` and `-Infinity` as a text values. Also conditions with special literals (such as ` n < '+Infinity'` or ` m > '-Infinity'` ) isn't numeric conditions in SQLite and gives unexpected result after pushdowning in oppose to internal PostgreSQL calculations. During `INSERT INTO ... SELECT` or in `WHERE` conditions `sqlite_fdw` uses given by PostgreSQL standard case sensitive literals **only** in follow forms: `NaN`, `-Infinity`, `Infinity`, not original strings from `WHERE` condition. *This can caused selecting issues*.
 
 ### Boolean values
 - `sqlite_fdw` boolean values support exists only for `bool` columns in foreign table. SQLite documentation recommends to store boolean as value with `integer` [affinity](https://www.sqlite.org/datatype3.html). `NULL` isn't converted, 1 converted to `true`, all other `NOT NULL` values converted to `false`. During `SELECT ... WHERE condition_column` condition converted only to `condition_column`.
@@ -546,7 +546,7 @@ for `INSERT` and `UPDATE` commands. PostgreSQL supports both `blob` and `text` [
 
 ### bit and varbit support
 - `sqlite_fdw` PostgreSQL `bit`/`varbit` values support based on `int` SQLite data affinity, because there is no per bit operations for SQLite `blob` affinity data. Maximum SQLite `int` affinity value is 8 bytes length, hence maximum `bit`/`varbit` values length is 64 bits.
-- `sqlite_fdw` doesn't pushdown `#` (XOR) operator becasuse there is no equal SQLite operator.
+- `sqlite_fdw` doesn't pushdown `#` (XOR) operator because there is no equal SQLite operator.
 
 Tests
 -----