Skip to content

Commit

Permalink
chart/navi-async-matrix syncup
Browse files Browse the repository at this point in the history
  • Loading branch information
@uk-navi-ci @i-bogomazov
uk-navi-ci authored and i-bogomazov committed Oct 30, 2024
1 parent 661d270 commit fe726ad
Show file tree
Hide file tree
Showing 6 changed files with 150 additions and 25 deletions.
56 changes: 42 additions & 14 deletions charts/navi-async-matrix/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ See the [documentation](https://docs.2gis.com/en/on-premise/navigation/distance-
| Name | Description | Value |
| ------------------ | ----------- | ----------------------------------- |
| `image.repository` | Repository | `2gis-on-premise/navi-async-matrix` |
| `image.tag` | Tag | `1.11.2` |
| `image.tag` | Tag | `1.12.1` |
| `image.pullPolicy` | Pull Policy | `IfNotPresent` |

### Service account settings
Expand Down Expand Up @@ -139,14 +139,19 @@ See the [documentation](https://docs.2gis.com/en/on-premise/navigation/distance-

### Distance Matrix Async API settings

| Name | Description | Value |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ------ |
| `dm.port` | Distance Matrix Async API HTTP port. | `8000` |
| `dm.configType` | Configuration type. Must always be `env`. | `env` |
| `dm.logLevel` | Logging level, one of: DEBUG, INFO, WARNING, ERROR, CRITICAL. | `INFO` |
| `dm.workerCount` | Number of Distance Matrix Async workers. | `4` |
| `dm.citiesUrl` | URL of the information about cities provided by the Navi-Castle service, ex: http://navi-castle.svc/cities.conf. **Required** | `""` |
| `dm.citiesUpdatePeriod` | Period (in seconds) between requesting data from `citiesUrl`. | `3600` |
| Name | Description | Value |
| ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------- |
| `dm.port` | Distance Matrix Async API HTTP port. | `8000` |
| `dm.configType` | Configuration type. Must always be `env`. | `env` |
| `dm.logLevel` | Logging level, one of: DEBUG, INFO, WARNING, ERROR, CRITICAL. | `INFO` |
| `dm.workerCount` | Number of Distance Matrix Async workers. | `4` |
| `dm.citiesUrl` | URL of the information about cities provided by the Navi-Castle service, ex: http://navi-castle.svc/cities.conf. **Required** | `""` |
| `dm.citiesUpdatePeriod` | Period (in seconds) between requesting data from `citiesUrl`. | `3600` |
| `dm.taskSplitSize` | Minimum size of matrix to get split in archiver job. | `5000` |
| `dm.compositeTaskTimeoutSec` | Timeout for executing split tasks. | `3600` |
| `dm.archiver.enabled` | Enables archiver post-processor job. Required for `truck` routing. | `false` |
| `dm.archiver.image.repository` | Image repository for archiver. | `2gis-on-premise/navi-archiver-async-matrix` |
| `dm.archiver.image.tag` | Image tag for archiver. | `1.4.1` |

### Database settings

Expand All @@ -164,13 +169,28 @@ See the [documentation](https://docs.2gis.com/en/on-premise/navigation/distance-
| `db.tls.key` | Key of postgresql server. | `""` |
| `db.tls.mode` | Level of protection. | `verify-full` |

### Multi-DC settings

| Name | Description | Value |
| -------------------------------------- | ----------------------------------------------------------------------------------- | --------------- |
| `multiDc.enabled` | If multi-DC functionality enabled | `false` |
| `multiDc.location` | Primary DC identifier. | `default` |
| `multiDc.redirectHeader` | HTTP header to tell requests original from redirected. Set empty to skip the check. | `""` |
| `multiDc.secondaryTopics.statusTopic` | Name of `statusTopic` in secondary DC. | `status_topic` |
| `multiDc.secondaryTopics.cancelTopic` | Name of `cancelTopic` in secondary DC. | `cancel_topic` |
| `multiDc.secondaryTopics.archiveTopic` | Name of `archiveTopic` in secondary DC. | `archive_topic` |

### Kafka settings

| Name | Description | Value |
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------- |
| `kafka.groupId` | Distance Matrix Async API group identifier. | `navi_async_matrix` |
| `kafka.statusTopic` | Name of the topic for sending new tasks to. | `status_topic` |
| `kafka.cancelTopic` | Name of the topic for canceling or receiving information about finished tasks. | `cancel_topic` |
| `kafka.archiveTopic` | Name of the topic for archiving tasks. | `archive_topic` |
| `kafka.attractTopic` | Name of the topic for for attract tasks results | `""` |
| `kafka.oneToManyTopic` | Name of the topic for oneToMany tasks results | `""` |
| `kafka.vrpStatusTopic` | Name of the topic for VRP service integration | `""` |
| `kafka.properties` | Properties as supported by kafka-python. Refer to inline comments for details. | |
| `kafka.sensitiveProperties` | As kafka.properties, but kept in Secrets. Refer to inlines comments for details. | `{}` |
| `kafka.fileProperties` | As kafka.properties, but kept in a file, which passed to application as a filename. Refer to inline comments for details. | `{}` |
Expand All @@ -183,24 +203,32 @@ See the [documentation](https://docs.2gis.com/en/on-premise/navigation/distance-
| `kafka.taskTopicRules` | **Information about the topics that Distance Matrix Async API will use to send the requests.** | |
| `kafka.taskTopicRules[].topic` | Name of the topic. | |
| `kafka.taskTopicRules[].default` | If this topic is used for projects by default. | |
| `kafka.taskTopicRules[].type` | Routing type for tasks in the topic (`car`, `truck`), defaults to `car` | |
| `kafka.taskTopicRules[].projects` | List of projects to use this topic for, e.g., `['moscow']`. | |
| `kafka.attractTopicRules` | ** Rules to map request type to topic for attract tasks ** | `[]` |
| `kafka.attractTopicRules[0].topic` | Name of the topic. | |
| `kafka.attractTopicRules[0].default` | If this topic is used for projects by default. | |
| `kafka.attractTopicRules[0].type` | Routing type for tasks in the topic (`car`, `truck`), defaults to `car` | |
| `kafka.attractTopicRules[0].projects` | List of projects to use this topic for, e.g., `['moscow']`. | |

### S3-compatible storage settings

| Name | Description | Value |
| ----------------- | ---------------------------------------------------------------------------------------------- | ----- |
| `s3.host` | S3 endpoint URL, ex: http://async-matrix-s3.host. **Required** | `""` |
| `s3.bucket` | S3 bucket name. **Required** | `""` |
| `s3.region` | S3 region. | `""` |
| `s3.accessKey` | S3 access key for accessing the bucket. **Required** | `""` |
| `s3.secretKey` | S3 secret key for accessing the bucket. **Required** | `""` |
| `s3.publicNetloc` | Announce proxy URL for S3 results instead of s3.url if not empty. Must start with `http(s)://` | `""` |
| `s3.publicNetloc` | Announce proxy URL for S3 results instead of s3.url if not empty. Must start with `http(s)://` | `nil` |

### API keys service

| Name | Description | Value |
| ------------ | --------------------------------------------------------------------------- | ----- |
| `keys.url` | API keys service URL, ex: http://keys-api.svc/service/v1/keys. **Required** | `""` |
| `keys.token` | API token to authorize at the service. **Required** | `""` |
| Name | Description | Value |
| ----------------- | ------------------------------------------------------------------------------ | ----- |
| `keys.url` | API keys service URL, ex: http://keys-api.svc/service/v1/keys. **Required** | `""` |
| `keys.token` | API token to authorize at the service. Required if truck car routing in use. | `""` |
| `keys.truckToken` | Truck API token to authorize at the service. Required if truck routing in use. | `""` |

### customCAs **Custom Certificate Authority**

Expand Down
1 change: 0 additions & 1 deletion charts/navi-async-matrix/templates/configmap-ca.yaml
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@

{{- if .Values.customCAs.bundle }}
apiVersion: v1
kind: ConfigMap
Expand Down
4 changes: 4 additions & 0 deletions charts/navi-async-matrix/templates/configmap.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,10 @@ data:
"name": "S3_BUCKET",
"value": {{ .Values.s3.bucket | mustToJson }}
},
{
"name": "S3_REGION_NAME",
"value": {{ .Values.s3.region | mustToJson }}
},
{
"name": "S3_ACCESS_KEY",
"value": {{ .Values.s3.accessKey | mustToJson }}
Expand Down
3 changes: 3 additions & 0 deletions charts/navi-async-matrix/templates/secret.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@ data:
| b64enc | quote }}
s3key: {{ required "A valid .Values.s3.secretKey entry required" .Values.s3.secretKey | b64enc | quote }}
dmApiKey: {{ .Values.keys.token | b64enc | quote }}
{{- if .Values.keys.truckToken }}
dmApiTruckKey: {{ .Values.keys.truckToken | b64enc | quote }}
{{- end }}
{{- if (
(or .Values.kafka.sensitiveProperties .Values.kafka.fileProperties) |
or (or .Values.kafka.consumerOverrides.sensitiveProperties .Values.kafka.consumerOverrides.fileProperties) |
Expand Down
39 changes: 39 additions & 0 deletions charts/navi-async-matrix/templates/statefulset.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,8 @@ spec:
value: {{ required "A valid .Values.s3.host entry required" .Values.s3.host | quote }}
- name: DM_ASYNC_SERVICE_STORAGE_SETTING__S3_STORAGE_CONNECTION__S3_BUCKET
value: {{ required "A valid .Values.s3.bucket entry required" .Values.s3.bucket | quote }}
- name: DM_ASYNC_SERVICE_STORAGE_SETTING__S3_STORAGE_CONNECTION__REGION_NAME
value: {{ .Values.s3.region | quote }}
- name: DM_ASYNC_SERVICE_LOGGER_SETTING__LEVEL
value: {{ .Values.dm.logLevel | quote }}
{{- if .Values.s3.publicNetloc }}
Expand All @@ -209,6 +211,22 @@ spec:
- name: DM_ASYNC_SERVICE_TOPIC_RULES
{{- /* topicRules is a structure, hence mustToJson */}}
value: {{ .Values.kafka.taskTopicRules | mustToJson | quote }}
{{- if .Values.kafka.attractTopic }}
- name: DM_ASYNC_SERVICE_KAFKA_CONSUMER_SETTING__KAFKA_CONSUMER_ATTRACT_TOPIC
value: {{ .Values.kafka.attractTopic | quote }}
{{- end }}
{{- if .Values.kafka.oneToManyTopic }}
- name: DM_ASYNC_SERVICE_KAFKA_CONSUMER_SETTING__KAFKA_CONSUMER_ONE_TO_MANY_TOPIC
value: {{ .Values.kafka.oneToManyTopic | quote }}
{{- end }}
{{- if .Values.kafka.attractTopicRules }}
- name: DM_ASYNC_SERVICE_ATTRACT_TOPIC_RULES
value: {{ .Values.kafka.attractTopicRules | mustToJson | quote }}
{{- end }}
{{- if .Values.kafka.vrpStatusTopic }}
- name: DM_ASYNC_SERVICE_SUBSCRIBER_TOPIC
value: {{ .Values.kafka.vrpStatusTopic | quote }}
{{- end }}
{{- if .Values.dm.archiver.enabled }}
- name: DM_ASYNC_SERVICE_KAFKA_CONSUMER_SETTING__KAFKA_CONSUMER_ARCHIVE_RESPONSE_TOPIC
value: {{ .Values.kafka.archiveTopic | quote }}
Expand All @@ -228,6 +246,20 @@ spec:
- name: DM_ASYNC_SERVICE_COMPOSITE_TASK_TIMEOUT_IN_SECONDS
value: {{ .Values.dm.compositeTaskTimeoutSec | quote }}
{{- end }}{{- /* .Values.dm.archiver.enabled */}}
{{- if .Values.multiDc.enabled }}
- name: DM_ASYNC_SERVICE_APP__LOCATION
value: {{ .Values.multiDc.location | quote }}
- name: DM_ASYNC_SERVICE_REDIRECT_HEADER
value: {{ .Values.multiDc.redirectHeader | default "" | quote }}
- name: DM_ASYNC_SERVICE_KAFKA_CONSUMER_SETTING__KAFKA_CONSUMER_SYNC_TASK_TOPIC
value: {{ .Values.multiDc.secondaryTopics.statusTopic | quote }}
- name: DM_ASYNC_SERVICE_KAFKA_CONSUMER_SETTING__KAFKA_CONSUMER_SYNC_CANCEL_TOPIC
value: {{ .Values.multiDc.secondaryTopics.cancelTopic | quote }}
{{- if .Values.dm.archiver.enabled }}
- name: DM_ASYNC_SERVICE_KAFKA_CONSUMER_SETTING__KAFKA_CONSUMER_SYNC_ARCHIVE_RESPONSE_TOPIC
value: {{ .Values.multiDc.secondaryTopics.archiveTopic | quote }}
{{- end }}{{- /* .Values.dm.archiver.enabled */}}
{{- end }}{{- /* .Values.kafka.secondaryTopics */}}
- name: DM_ASYNC_SERVICE_NODE_ID
valueFrom:
fieldRef:
Expand Down Expand Up @@ -276,6 +308,13 @@ spec:
secretKeyRef:
name: {{ include "navi-async-matrix.fullname" . | quote }}
key: dmApiKey
{{- if .Values.keys.truckToken }}
- name: DM_ASYNC_SERVICE_KMS__TRUCK_DM_ASYNC_TOKEN
valueFrom:
secretKeyRef:
name: {{ include "navi-async-matrix.fullname" . | quote }}
key: dmApiTruckKey
{{- end }}
{{- if .Values.customCAs.bundle }}
- name: SSL_CERT_FILE
value: {{ include "custom.ca.mountPath" . }}/custom-ca.crt
Expand Down
Loading

0 comments on commit fe726ad

Please sign in to comment.