Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: allow dbconfig options #968

Merged
merged 1 commit into from
Feb 27, 2024
Merged

feat: allow dbconfig options #968

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
143 changes: 85 additions & 58 deletions provider/postgis/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,33 @@
# PostGIS
The PostGIS provider manages querying for tile requests against a Postgres database with the [PostGIS](http://postgis.net/) extension installed. The connection between tegola and Postgis is configured in a `tegola.toml` file. An example minimum connection config:

The PostGIS provider manages querying for tile requests against a Postgres
database with the [PostGIS](http://postgis.net/) extension installed.
The connection between tegola and Postgis is configured in a `tegola.toml` file.
An example minimum connection config:

```toml
[[providers]]
name = "test_postgis" # provider name is referenced from map layers (required)
type = "postgis" # the type of data provider must be "postgis" for this data provider (required)
# provider name is referenced from map layers (required)
name = "test_postgis"

uri = "postgres://tegola:supersecret@localhost:5432/tegola?sslmode=prefer" # PostGIS connection string (required)
# the type of data provider must be "postgis" for this data provider (required)
type = "postgis"

# PostGIS connection string (required)
uri = "postgres://tegola:supersecret@localhost:5432/tegola?sslmode=prefer" #

# PostGIS connection config run time parameter to label
# the origin of a connection
# The default is "tegola"
# (optional)
application_name = "tegola"

# PostGIS connection config run time parameter (optional)
# A read-only SQL transaction cannot alter non-temporary tables.
# This parameter controls the default read-only status of each new transaction.
# The default is OFF (read/write).
# (optional)
default_transaction_read_only = "off"

host = "localhost" # PostGIS database host (deprecated)
port = 5432 # PostGIS database port (deprecated)
Expand All @@ -19,83 +39,86 @@ max_connection_idle_time = "30m" # PostGIS max connection idle time (deprecated
max_connection_lifetime = "1h" # PostGIS max connection life time (deprecated)
```

### Connection Properties
## Connection Properties

Establishing a connection via connection string (`uri`) will become the default connection method as of v0.16.0.
Connecting via host/port/database is flagged for deprecation as of v0.15.0 but will be possible until v0.16.0 still.
Establishing a connection via connection string (`uri`) will become the default
connection method as of v0.16.0. Connecting via host/port/database is flagged
for deprecation as of v0.15.0 but will be possible until v0.16.0 still.

- `uri` (string): [Required] PostGIS connection string
- `name` (string): [Required] provider name is referenced from map layers
- `type` (string): [Required] the type of data provider. must be "postgis" to use this data provider
- `srid` (int): [Optional] The default SRID for the provider. Defaults to WebMercator (3857) but also supports WGS84 (4326)
- `uri` (string): [Required] PostGIS connection string
- `name` (string): [Required] provider name is referenced from map layers
- `type` (string): [Required] the type of data provider. must be "postgis" to use this data provider
- `srid` (int): [Optional] The default SRID for the provider. Defaults to WebMercator (3857) but also supports WGS84 (4326)

#### Connection string properties
### Connection string properties

**Example**
#### Example

```
# {protocol}://{user}:{password}@{host}:{port}/{database}?{options}=
postgres://tegola:supersecret@localhost:5432/tegola?sslmode=prefer&pool_max_conns=10
```

**Options**
#### Options

- `sslmode`: [Optional] PostGIS SSL mode. Default: "prefer"
- `pool_max_conns`: [Optional] The max connections to maintain in the connection pool. Defaults to 100. 0 means no max.
- `pool_max_conn_idle_time`: [Optional] The maximum time an idle connection is kept alive. Defaults to "30m".
- `max_connection_lifetime` [Optional] The maximum time a connection lives before it is terminated and recreated. Defaults to "1h".
- `sslmode`: [Optional] PostGIS SSL mode. Default: "prefer"
- `pool_max_conns`: [Optional] The max connections to maintain in the connection pool. Defaults to 100. 0 means no max.
- `pool_max_conn_idle_time`: [Optional] The maximum time an idle connection is kept alive. Defaults to "30m".
- `max_connection_lifetime` [Optional] The maximum time a connection lives before it is terminated and recreated. Defaults to "1h".

### [DEPRECATED] Connection Properties

- `uri` (string): [Required] PostGIS connection string
- `name` (string): [Required] provider name is referenced from map layers
- `type` (string): [Required] the type of data provider. must be "postgis" to use this data provider
- `host` (string): [deprecated] PostGIS database host
- `port` (int): [deprecated] PostGIS database port (required)
- `database` (string): [deprecated] PostGIS database name
- `user` (string): [deprecated] PostGIS database user
- `password` (string): [deprecated] PostGIS database password
- `srid` (int): [Optional] The default SRID for the provider. Defaults to WebMercator (3857) but also supports WGS84 (4326)
- `ssl_mode`: (string): [Optional]. PostGIS SSL mode. Default is "prefer".
- `max_connections` (int): [deprecated] The max connections to maintain in the connection pool. Defaults to 100. 0 means no max.
- `max_connection_idle_time` (duration string): [deprecated] The maximum time an idle connection is kept alive.
- `max_connection_lifetime` (duration string): [deprecated] The maximum time a connection lives before it is terminated and recreated.
- `uri` (string): [Required] PostGIS connection string
- `name` (string): [Required] provider name is referenced from map layers
- `type` (string): [Required] the type of data provider. must be "postgis" to use this data provider
- `host` (string): [deprecated] PostGIS database host
- `port` (int): [deprecated] PostGIS database port (required)
- `database` (string): [deprecated] PostGIS database name
- `user` (string): [deprecated] PostGIS database user
- `password` (string): [deprecated] PostGIS database password
- `srid` (int): [Optional] The default SRID for the provider. Defaults to WebMercator (3857) but also supports WGS84 (4326)
- `ssl_mode`: (string): [Optional]. PostGIS SSL mode. Default is "prefer".
- `max_connections` (int): [deprecated] The max connections to maintain in the connection pool. Defaults to 100. 0 means no max.
- `max_connection_idle_time` (duration string): [deprecated] The maximum time an idle connection is kept alive.
- `max_connection_lifetime` (duration string): [deprecated] The maximum time a connection lives before it is terminated and recreated.

## Provider Layers

In addition to the connection configuration above, Provider Layers need to be configured. A Provider Layer tells tegola how to query PostGIS for a certain layer. An example minimum config:

```toml
[[providers.layers]]
name = "landuse"
# this table uses "geom" for the geometry_fieldname and "gid" for the id_fieldname so they don't need to be configured
# this table uses "geom" for the geometry_fieldname and "gid" for the
# id_fieldname so they don't need to be configured
tablename = "gis.zoning_base_3857"
```

### Provider Layers Properties

- `name` (string): [Required] the name of the layer. This is used to reference this layer from map layers.
- `tablename` (string): [*Required] the name of the database table to query against. Required if `sql` is not defined.
- `geometry_fieldname` (string): [Optional] the name of the filed which contains the geometry for the feature. defaults to `geom`.
- `id_fieldname` (string): [Optional] the name of the feature id field. defaults to `gid`.
- `fields` ([]string): [Optional] a list of fields to include alongside the feature. Can be used if `sql` is not defined.
- `srid` (int): [Optional] the SRID of the layer. Supports `3857` (WebMercator) or `4326` (WGS84).
- `geometry_type` (string): [Optional] the layer geometry type. If not set, the table will be inspected at startup to try and infer the gemetry type. Valid values are: `Point`, `LineString`, `Polygon`, `MultiPoint`, `MultiLineString`, `MultiPolygon`, `GeometryCollection`.
- `sql` (string): [*Required] custom SQL to use use. Required if `tablename` is not defined. Supports the following tokens:
- `!BBOX!` - [Required] will be replaced with the bounding box of the tile before the query is sent to the database. `!bbox!` and`!BOX!` are supported as well for compatibilitiy with queries from Mapnik and MapServer styles.
- `!ZOOM!` - [Optional] will be replaced with the "Z" (zoom) value of the requested tile.
- `!X!` - [Optional] will be replaced with the "X" value of the requested tile.
- `!Y!` - [Optional] will be replaced with the "Y" value of the requested tile.
- `!Z!` - [Optional] will be replaced with the "Z" value of the requested tile.
- `!SCALE_DENOMINATOR!` - [Optional] scale denominator, assuming 90.7 DPI (i.e. 0.28mm pixel size)
- `!PIXEL_WIDTH!` - [Optional] the pixel width in meters, assuming 256x256 tiles
- `!PIXEL_HEIGHT!` - [Optional] the pixel height in meters, assuming 256x256 tiles
- `!ID_FIELD!` - [Optional] the id field name
- `!GEOM_FIELD!` - [Optional] the geom field name
- `!GEOM_TYPE!` - [Optional] the geom type field name
- `name` (string): [Required] the name of the layer. This is used to reference this layer from map layers.
- `tablename` (string): [*Required] the name of the database table to query against. Required if `sql` is not defined.
- `geometry_fieldname` (string): [Optional] the name of the filed which contains the geometry for the feature. defaults to `geom`.
- `id_fieldname` (string): [Optional] the name of the feature id field. defaults to `gid`.
- `fields` ([]string): [Optional] a list of fields to include alongside the feature. Can be used if `sql` is not defined.
- `srid` (int): [Optional] the SRID of the layer. Supports `3857` (WebMercator) or `4326` (WGS84).
- `geometry_type` (string): [Optional] the layer geometry type. If not set, the table will be inspected at startup to try and infer the gemetry type. Valid values are: `Point`, `LineString`, `Polygon`, `MultiPoint`, `MultiLineString`, `MultiPolygon`, `GeometryCollection`.
- `sql` (string): [*Required] custom SQL to use use. Required if `tablename` is not defined. Supports the following tokens:
- `!BBOX!` - [Required] will be replaced with the bounding box of the tile before the query is sent to the database. `!bbox!` and`!BOX!` are supported as well for compatibilitiy with queries from Mapnik and MapServer styles.
- `!ZOOM!` - [Optional] will be replaced with the "Z" (zoom) value of the requested tile.
- `!X!` - [Optional] will be replaced with the "X" value of the requested tile.
- `!Y!` - [Optional] will be replaced with the "Y" value of the requested tile.
- `!Z!` - [Optional] will be replaced with the "Z" value of the requested tile.
- `!SCALE_DENOMINATOR!` - [Optional] scale denominator, assuming 90.7 DPI (i.e. 0.28mm pixel size)
- `!PIXEL_WIDTH!` - [Optional] the pixel width in meters, assuming 256x256 tiles
- `!PIXEL_HEIGHT!` - [Optional] the pixel height in meters, assuming 256x256 tiles
- `!ID_FIELD!` - [Optional] the id field name
- `!GEOM_FIELD!` - [Optional] the geom field name
- `!GEOM_TYPE!` - [Optional] the geom type field name

`*Required`: either the `tablename` or `sql` must be defined, but not both.

**Example minimum custom SQL config**
#### Example minimum custom SQL config

```toml
[[providers.layers]]
Expand All @@ -106,21 +129,25 @@ sql = "SELECT gid, ST_AsBinary(geom) AS geom FROM gis.rivers WHERE geom && !BBOX
```

## Environment Variable support

Helpful debugging environment variables:

- `TEGOLA_SQL_DEBUG`: specify the type of SQL debug information to output. Supports the following values:
- `LAYER_SQL`: print layer SQL as they’re parsed from the config file.
- `EXECUTE_SQL`: print SQL that is executed for each tile request and the number of items it returns or an error.
- `LAYER_SQL:EXECUTE_SQL`: print `LAYER_SQL` and `EXECUTE_SQL`.
- `TEGOLA_SQL_DEBUG`: specify the type of SQL debug information to output. Supports the following values:
- `LAYER_SQL`: print layer SQL as they’re parsed from the config file.
- `EXECUTE_SQL`: print SQL that is executed for each tile request and the number of items it returns or an error.
- `LAYER_SQL:EXECUTE_SQL`: print `LAYER_SQL` and `EXECUTE_SQL`.

Example:

```
```bash
$ TEGOLA_SQL_DEBUG=LAYER_SQL tegola serve --config=/path/to/conf.toml
```

## Testing
Testing is designed to work against a live PostGIS database. To see how to set up a database check this [github actions script](https://github.com/go-spatial/tegola/blob/master/.github/worksflows/on_pr_push.yml). To run the PostGIS tests, the following environment variables need to be set:

Testing is designed to work against a live PostGIS database. To see how to set
up a database check this [github actions script](https://github.com/go-spatial/tegola/blob/master/.github/worksflows/on_pr_push.yml).
To run the PostGIS tests, the following environment variables need to be set:

```bash
$ export RUN_POSTGIS_TESTS=yes
Expand Down
118 changes: 78 additions & 40 deletions provider/postgis/postgis.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,41 +175,45 @@ const (
)

const (
DefaultURI = ""
DefaultPort = 5432
DefaultSRID = tegola.WebMercator
DefaultMaxConn = 100
DefaultMaxConnIdleTime = "30m"
DefaultMaxConnLifetime = "1h"
DefaultSSLMode = "prefer"
DefaultSSLKey = ""
DefaultSSLCert = ""
DefaultURI = ""
DefaultPort = 5432
DefaultSRID = tegola.WebMercator
DefaultMaxConn = 100
DefaultMaxConnIdleTime = "30m"
DefaultMaxConnLifetime = "1h"
DefaultSSLMode = "prefer"
DefaultSSLKey = ""
DefaultSSLCert = ""
DefaultApplicationName = "tegola"
DefaultDefaultTransactionReadOnly = "TRUE"
)

const (
ConfigKeyName = "name"
ConfigKeyURI = "uri"
ConfigKeyHost = "host"
ConfigKeyPort = "port"
ConfigKeyDB = "database"
ConfigKeyUser = "user"
ConfigKeyPassword = "password"
ConfigKeySSLMode = "ssl_mode"
ConfigKeySSLKey = "ssl_key"
ConfigKeySSLCert = "ssl_cert"
ConfigKeySSLRootCert = "ssl_root_cert"
ConfigKeyMaxConn = "max_connections"
ConfigKeyMaxConnIdleTime = "max_connection_idle_time"
ConfigKeyMaxConnLifetime = "max_connection_lifetime"
ConfigKeySRID = "srid"
ConfigKeyLayers = "layers"
ConfigKeyLayerName = "name"
ConfigKeyTablename = "tablename"
ConfigKeySQL = "sql"
ConfigKeyFields = "fields"
ConfigKeyGeomField = "geometry_fieldname"
ConfigKeyGeomIDField = "id_fieldname"
ConfigKeyGeomType = "geometry_type"
ConfigKeyName = "name"
ConfigKeyURI = "uri"
ConfigKeyHost = "host"
ConfigKeyPort = "port"
ConfigKeyDB = "database"
ConfigKeyUser = "user"
ConfigKeyPassword = "password"
ConfigKeySSLMode = "ssl_mode"
ConfigKeySSLKey = "ssl_key"
ConfigKeySSLCert = "ssl_cert"
ConfigKeySSLRootCert = "ssl_root_cert"
ConfigKeyMaxConn = "max_connections"
ConfigKeyMaxConnIdleTime = "max_connection_idle_time"
ConfigKeyMaxConnLifetime = "max_connection_lifetime"
ConfigKeySRID = "srid"
ConfigKeyLayers = "layers"
ConfigKeyLayerName = "name"
ConfigKeyTablename = "tablename"
ConfigKeySQL = "sql"
ConfigKeyFields = "fields"
ConfigKeyGeomField = "geometry_fieldname"
ConfigKeyGeomIDField = "id_fieldname"
ConfigKeyGeomType = "geometry_type"
ConfigKeyApplicationName = "application_name"
ConfigKeyDefaultTransactionReadOnly = "default_transaction_read_only"
iwpnd marked this conversation as resolved.
Show resolved Hide resolved
)

// isSelectQuery is a regexp to check if a query starts with `SELECT`,
Expand Down Expand Up @@ -361,19 +365,36 @@ func BuildURI(config dict.Dicter) (*url.URL, *url.Values, error) {
return u, params, nil
}

type DBConfigOptions struct {
Uri string
DefaultTransactionReadOnly string
ApplicationName string
}

func (opts *DBConfigOptions) GetRuntimeParams() map[string]string {
pr := map[string]string{
ConfigKeyApplicationName: opts.ApplicationName,
}

// as per https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-DEFAULT-TRANSACTION-READ-ONLY
// default_transaction_read_only accepts boolean, and is not set by default
// hence if OFF, we do not add it to RuntimeParams
if opts.DefaultTransactionReadOnly != "" && strings.ToUpper(opts.DefaultTransactionReadOnly) != "OFF" {
pr[ConfigKeyDefaultTransactionReadOnly] = strings.ToUpper(opts.DefaultTransactionReadOnly)
}

return pr
}

// BuildDBConfig build db config with defaults
func BuildDBConfig(uri string) (*pgxpool.Config, error) {
dbconfig, err := pgxpool.ParseConfig(uri)
func BuildDBConfig(opts *DBConfigOptions) (*pgxpool.Config, error) {
dbconfig, err := pgxpool.ParseConfig(opts.Uri)
if err != nil {
return nil, err
}

dbconfig.ConnConfig.LogLevel = pgx.LogLevelWarn
dbconfig.ConnConfig.RuntimeParams = map[string]string{
"default_transaction_read_only": "TRUE",
"application_name": "tegola",
}

dbconfig.ConnConfig.RuntimeParams = opts.GetRuntimeParams()
var hstore hstoreOID

dbconfig.AfterConnect = func(ctx context.Context, conn *pgx.Conn) error {
Expand Down Expand Up @@ -466,7 +487,24 @@ func CreateProvider(config dict.Dicter, maps []provider.Map, providerType string
return nil, err
}

dbconfig, err := BuildDBConfig(uri.String())
default_transaction_read_only := DefaultDefaultTransactionReadOnly
default_transaction_read_only, err = config.String(ConfigKeyDefaultTransactionReadOnly, &default_transaction_read_only)
if err != nil {
return nil, err
}

application_name := DefaultApplicationName
application_name, err = config.String(ConfigKeyApplicationName, &application_name)
if err != nil {
return nil, err
}

dbconfig, err := BuildDBConfig(
&DBConfigOptions{
Uri: uri.String(),
DefaultTransactionReadOnly: default_transaction_read_only,
ApplicationName: application_name,
})
if err != nil {
return nil, fmt.Errorf("failed while building db config: %w", err)
}
Expand Down
Loading
Loading