[Feature][Connector-V2][Persistiq]Add Persistiq source connector

docs/en/connector-v2/source/Persistiq.md

@@ -0,0 +1,296 @@
+# Persistiq
+
+> Persistiq source connector
+
+## Description
+
+Used to read data from Persistiq.
+
+## Key features
+
+- [x] [batch](../../concept/connector-v2-features.md)
+- [ ] [stream](../../concept/connector-v2-features.md)
+- [ ] [exactly-once](../../concept/connector-v2-features.md)
+- [x] [schema projection](../../concept/connector-v2-features.md)
+- [ ] [parallelism](../../concept/connector-v2-features.md)
+- [ ] [support user-defined split](../../concept/connector-v2-features.md)
+
+##  Options
+
+| name                        | type   | required | default value |
+| --------------------------- | ------ | -------- | ------------- |
+| url                         | String | Yes      | -             |
+| password                    | String | Yes      | -             |
+| method                      | String | No       | get           |
+| schema                      | Config | No       | -             |
+| schema.fields               | Config | No       | -             |
+| format                      | String | No       | json          |
+| params                      | Map    | No       | -             |
+| body                        | String | No       | -             |
+| json_field                  | Config | No       | -             |
+| content_json                | String | No       | -             |
+| poll_interval_ms            | int    | No       | -             |
+| retry                       | int    | No       | -             |
+| retry_backoff_multiplier_ms | int    | No       | 100           |
+| retry_backoff_max_ms        | int    | No       | 10000         |
+| common-options              | config | No       | -             |
+
+### url [String]
+
+http request url
+
+### password [String]
+
+API key for login, you can get it at Persistiq website
+
+### method [String]
+
+http request method, only supports GET, POST method
+
+### params [Map]
+
+http params
+
+### body [String]
+
+http body
+
+### poll_interval_ms [int]
+
+request http api interval(millis) in stream mode
+
+### retry [int]
+
+The max retry times if request http return to `IOException`
+
+### retry_backoff_multiplier_ms [int]
+
+The retry-backoff times(millis) multiplier if request http failed
+
+### retry_backoff_max_ms [int]
+
+The maximum retry-backoff times(millis) if request http failed
+
+### format [String]
+
+the format of upstream data, now only support `json` `text`, default `json`.
+
+when you assign format is `json`, you should also assign schema option, for example:
+
+upstream data is the following:
+
+```json
+{
+  "code": 200,
+  "data": "get success",
+  "success": true
+}
+```
+
+you should assign schema as the following:
+
+```hocon
+
+schema {
+    fields {
+        code = int
+        data = string
+        success = boolean
+    }
+}
+
+```
+
+connector will generate data as the following:
+
+| code | data        | success |
+|------|-------------|---------|
+| 200  | get success | true    |
+
+when you assign format is `text`, connector will do nothing for upstream data, for example:
+
+upstream data is the following:
+
+```json
+{
+  "code": 200,
+  "data": "get success",
+  "success": true
+}
+```
+
+connector will generate data as the following:
+
+| content |
+|---------|
+| {"code":  200, "data":  "get success", "success":  true}        |
+
+### schema [Config]
+
+#### fields [Config]
+
+the schema fields of upstream data
+
+### content_json [String]
+
+This parameter can get some json data.If you only need the data in the 'book' section, configure `content_field = "$.store.book.*"`.
+
+If your return data looks something like this.
+
+```json
+{
+  "store": {
+    "book": [
+      {
+        "category": "reference",
+        "author": "Nigel Rees",
+        "title": "Sayings of the Century",
+        "price": 8.95
+      },
+      {
+        "category": "fiction",
+        "author": "Evelyn Waugh",
+        "title": "Sword of Honour",
+        "price": 12.99
+      }
+    ],
+    "bicycle": {
+      "color": "red",
+      "price": 19.95
+    }
+  },
+  "expensive": 10
+}
+```
+You can configure `content_field = "$.store.book.*"` and the result returned looks like this:
+
+```json
+[
+  {
+    "category": "reference",
+    "author": "Nigel Rees",
+    "title": "Sayings of the Century",
+    "price": 8.95
+  },
+  {
+    "category": "fiction",
+    "author": "Evelyn Waugh",
+    "title": "Sword of Honour",
+    "price": 12.99
+  }
+]
+```
+Then you can get the desired result with a simpler schema,like
+
+```hocon
+Http {
+  url = "http://mockserver:1080/contentjson/mock"
+  method = "GET"
+  format = "json"
+  content_field = "$.store.book.*"
+  schema = {
+    fields {
+      category = string
+      author = string
+      title = string
+      price = string
+    }
+  }
+}
+```
+
+Here is an example:
+
+- Test data can be found at this link [mockserver-contentjson-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-contentjson-config.json)
+- See this link for task configuration [http_contentjson_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_contentjson_to_assert.conf).
+
+### json_field [Config]
+
+This parameter helps you configure the schema,so this parameter must be used with schema.
+
+If your data looks something like this:
+
+```json
+{
+  "store": {
+    "book": [
+      {
+        "category": "reference",
+        "author": "Nigel Rees",
+        "title": "Sayings of the Century",
+        "price": 8.95
+      },
+      {
+        "category": "fiction",
+        "author": "Evelyn Waugh",
+        "title": "Sword of Honour",
+        "price": 12.99
+      }
+    ],
+    "bicycle": {
+      "color": "red",
+      "price": 19.95
+    }
+  },
+  "expensive": 10
+}
+```
+
+You can get the contents of 'book' by configuring the task as follows:
+
+```hocon
+source {
+  Http {
+    url = "http://mockserver:1080/jsonpath/mock"
+    method = "GET"
+    format = "json"
+    json_field = {
+      category = "$.store.book[*].category"
+      author = "$.store.book[*].author"
+      title = "$.store.book[*].title"
+      price = "$.store.book[*].price"
+    }
+    schema = {
+      fields {
+        category = string
+        author = string
+        title = string
+        price = string
+      }
+    }
+  }
+}
+```
+
+- Test data can be found at this link [mockserver-jsonpath-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-jsonpath-config.json)
+- See this link for task configuration [http_jsonpath_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_jsonpath_to_assert.conf).
+
+### common options 
+
+Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details
+
+## Example
+
+```hocon
+  Persistiq{
+    url = "https://api.persistiq.com/v1/users"
+    password = "Seatunnel-test"
+    content_field = "$.users.*"
+    schema = {
+        fields {
+          id = string
+          name = string
+          email = string
+          activated = boolean
+          default_mailbox_id = string
+          salesforce_id = string
+        }
+    }
+  }
+```
+
+## Changelog
+
+### next version
+
+- Add Persistiq Source Connector

plugin-mapping.properties

@@ -158,4 +158,5 @@ seatunnel.source.Notion = connector-http-notion
 seatunnel.sink.RabbitMQ = connector-rabbitmq
 seatunnel.source.RabbitMQ = connector-rabbitmq
 seatunnel.source.OpenMldb = connector-openmldb
-seatunnel.source.MySQL-CDC = connector-cdc-mysql
+seatunnel.source.MySQL-CDC = connector-cdc-mysql
+seatunnel.source.Persistiq = connector-http-persistiq

seatunnel-connectors-v2/connector-http/connector-http-persistiq/pom.xml

@@ -0,0 +1,40 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+    Licensed to the Apache Software Foundation (ASF) under one or more
+    contributor license agreements.  See the NOTICE file distributed with
+    this work for additional information regarding copyright ownership.
+    The ASF licenses this file to You under the Apache License, Version 2.0
+    (the "License"); you may not use this file except in compliance with
+    the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <parent>
+        <artifactId>connector-http</artifactId>
+        <groupId>org.apache.seatunnel</groupId>
+        <version>${revision}</version>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+
+    <artifactId>connector-http-persistiq</artifactId>
+
+    <dependencies>
+        <dependency>
+            <groupId>org.apache.seatunnel</groupId>
+            <artifactId>connector-http-base</artifactId>
+            <version>${project.version}</version>
+        </dependency>
+    </dependencies>
+
+</project>