Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update exchange parameter #2337

Merged
merged 1 commit into from
Nov 7, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs-2.0-en/import-export/nebula-exchange/ex-ug-FAQ.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ Solution:

- batch: The number of data contained in each nGQL statement sent to the NebulaGraph service.

- partition: The number of Spark data partitions, indicating the number of concurrent data imports.
- partition: The number of partitions to be created when the data is written to {{nebula.name}}, indicating the number of concurrent data imports.

- nebula.rate: Get a token from the token bucket before sending a request to NebulaGraph.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ For different data sources, the vertex configurations are different. There are m
|`tags.vertex.prefix`|string|-|No|Add the specified prefix to the VID. For example, if the VID is `12345`, adding the prefix `tag1` will result in `tag1_12345`. The underscore cannot be modified.|
|`tags.vertex.policy`|string|-|No|Supports only the value `hash`. Performs hashing operations on VIDs of type string.|
|`tags.batch`|int|`256`|Yes|The maximum number of vertices written into NebulaGraph in a single batch.|
|`tags.partition`|int|`32`|Yes|The number of Spark partitions.|
|`tags.partition`|int|`32`|Yes|The number of partitions to be created when the data is written to {{nebula.name}}. If `tags.partition ≤ 1`, the number of partitions to be created in {{nebula.name}} is the same as that in the data source.|

### Specific parameters of Parquet/JSON/ORC data sources

Expand Down Expand Up @@ -246,7 +246,7 @@ For the specific parameters of different data sources for edge configurations, p
|`edges.target.policy`|string|-|No|Supports only the value `hash`. Performs hashing operations on VIDs of type string.|
|`edges.ranking`|int|-|No|The column of rank values. If not specified, all rank values are `0` by default.|
|`edges.batch`|int|`256`|Yes|The maximum number of edges written into NebulaGraph in a single batch.|
|`edges.partition`|int|`32`|Yes|The number of Spark partitions.|
|`edges.partition`|int|`32`|Yes|The number of partitions to be created when the data is written to {{nebula.name}}. If `edges.partition ≤ 1`, the number of partitions to be created in {{nebula.name}} is the same as that in the data source.|

### Specific parameters for generating SST files

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}

Expand Down Expand Up @@ -277,7 +277,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}

Expand Down Expand Up @@ -310,7 +310,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# Number of pieces of data written to NebulaGraph in a single batch.
batch: 256

# Number of Spark partitions
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
# Set Tag Team information.
Expand Down Expand Up @@ -306,7 +306,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -258,7 +258,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
# Set the information about the Tag Team.
Expand Down Expand Up @@ -342,7 +342,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -247,7 +247,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
# Set the information about the Tag Team.
Expand Down Expand Up @@ -361,7 +361,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}

Expand Down Expand Up @@ -324,7 +324,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 10

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 10
# The interval for message reading. Unit: second.
interval.seconds: 10
Expand Down Expand Up @@ -277,7 +277,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# # The number of data written to NebulaGraph in a single batch.
# batch: 10

# # The number of Spark partitions.
# # The number of partitions to be created when the data is written to {{nebula.name}}.
# partition: 10

# # The interval for message reading. Unit: second.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}

Expand Down Expand Up @@ -298,7 +298,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# Batch operation types, including INSERT, UPDATE, and DELETE. defaults to INSERT.
#writeMode: INSERT

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition:10

# The number of data written to NebulaGraph in a single batch.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -236,7 +236,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
# Set the information about the Tag Team.
Expand Down Expand Up @@ -341,7 +341,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -231,7 +231,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
# Set the information about the Tag Team.
Expand Down Expand Up @@ -334,7 +334,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}

Expand Down Expand Up @@ -289,7 +289,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}

Expand Down Expand Up @@ -289,7 +289,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -185,7 +185,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 10

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 10
# The interval for message reading. Unit: second.
interval.seconds: 10
Expand Down Expand Up @@ -282,7 +282,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 10

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 10

# The interval for message reading. Unit: second.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -285,7 +285,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32

# Whether to repartition data based on the number of partitions of graph spaces in NebulaGraph when generating the SST file.
Expand Down Expand Up @@ -374,7 +374,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32

# Whether to repartition data based on the number of partitions of graph spaces in NebulaGraph when generating the SST file.
Expand Down
2 changes: 1 addition & 1 deletion docs-2.0-zh/import-export/nebula-exchange/ex-ug-FAQ.md
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,7 @@ Spark 日志提示 `Hive Schema version 1.2.0 does not match metastore's schema

- batch:每次发送给{{nebula.name}}服务的 nGQL 语句中包含的数据条数。

- partition:Spark 数据的分区数,表示数据导入的并发数。
- partition:数据写入{{nebula.name}}时需要创建的分区数,表示数据导入的并发数。

- nebula.rate:向{{nebula.name}}发送请求前先去令牌桶获取令牌。

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@
|`tags.vertex.prefix`|string|-|否|为 VID 增加指定的前缀。例如 VID 为`12345`,增加前缀`tag1`后为`tag1_12345`。下划线无法修改。|
|`tags.vertex.policy`|string|-|否|仅支持取值`hash`。对 string 类型的 VID 进行哈希化操作。|
|`tags.batch`|int|`256`|是|单批次写入{{nebula.name}}的最大点数量。|
|`tags.partition`|int|`32`|是|Spark 分片数量。|
|`tags.partition`|int|`32`|是|数据写入{{nebula.name}}时需要创建的分区数。如果`tags.partition ≤ 1`,在{{nebula.name}}中创建的分区数和数据源的分区数相同。|

### Parquet/JSON/ORC 源特有参数

Expand Down Expand Up @@ -249,7 +249,7 @@
|`edges.target.policy`|string|-|否|仅支持取值`hash`。对 string 类型的 VID 进行哈希化操作。|
|`edges.ranking`|int|-|否|rank 值的列。没有指定时,默认所有 rank 值为`0`。|
|`edges.batch`|int|`256`|是|单批次写入{{nebula.name}}的最大边数量。|
|`edges.partition`|int|`32`|是|Spark 分片数量。|
|`edges.partition`|int|`32`|是|数据写入{{nebula.name}}时需要创建的分区数。如果`edges.partition ≤ 1`,在{{nebula.name}}中创建的分区数和数据源的分区数相同。|

### 生成 SST 时的特有参数

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@
# 单批次写入{{nebula.name}}的数据条数。
batch: 256

# Spark 分区数量
# 数据写入{{nebula.name}}时需要创建的分区数。
partition: 32
}

Expand Down Expand Up @@ -283,7 +283,7 @@
# 单批次写入{{nebula.name}}的数据条数。
batch: 256

# Spark 分区数量
# 数据写入{{nebula.name}}时需要创建的分区数。
partition: 32
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@
# 指定单批次写入 {{nebula.name}} 的最大点数量。
batch: 256

# 指定 Spark 分片数量
# 数据写入{{nebula.name}}时需要创建的分区数
partition: 32
}

Expand Down Expand Up @@ -308,7 +308,7 @@
# 指定单批次写入 {{nebula.name}} 的最大边数量。
batch: 256

# 指定 Spark 分片数量
# 数据写入{{nebula.name}}时需要创建的分区数
partition: 32
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -215,7 +215,7 @@ ROW COLUMN+CELL
# 单批次写入 {{nebula.name}} 的数据条数。
batch: 256

# Spark 分区数量
# 数据写入{{nebula.name}}时需要创建的分区数。
partition: 32
}
# 设置 Tag team 相关信息。
Expand Down Expand Up @@ -304,7 +304,7 @@ ROW COLUMN+CELL
# 单批次写入 {{nebula.name}} 的数据条数。
batch: 256

# Spark 分区数量
# 数据写入{{nebula.name}}时需要创建的分区数。
partition: 32
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -254,7 +254,7 @@ scala> sql("select playerid, teamid, start_year, end_year from basketball.serve"
# 单批次写入 {{nebula.name}} 的最大数据条数。
batch: 256

# Spark 分区数量
# 数据写入{{nebula.name}}时需要创建的分区数。
partition: 32
}
# 设置 Tag team 相关信息。
Expand Down Expand Up @@ -338,7 +338,7 @@ scala> sql("select playerid, teamid, start_year, end_year from basketball.serve"
# 单批次写入 {{nebula.name}} 的最大数据条数。
batch: 256

# Spark 分区数量
# 数据写入{{nebula.name}}时需要创建的分区数。
partition: 32
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,7 @@ nebula-exchange_spark_2.2 仅支持单表查询,不支持多表查询。
# 单批次写入 {{nebula.name}} 的数据条数。
batch: 256

# Spark 分区数量
# 数据写入{{nebula.name}}时需要创建的分区数。
partition: 32
}
# 设置 Tag team 相关信息。
Expand Down Expand Up @@ -368,7 +368,7 @@ nebula-exchange_spark_2.2 仅支持单表查询,不支持多表查询。
# 单批次写入 {{nebula.name}} 的数据条数。
batch: 256

# Spark 分区数量
# 数据写入{{nebula.name}}时需要创建的分区数。
partition: 32
}

Expand Down
Loading