Guide to migrate Centralized Logging with OpenSearch (CLO) v1.0.3 to v2.0.0

[wchaws]

Prerequisites

• Ensure that your current installation of CLO is version v1.0.3. If your current CLO version is lower than v1.0.3, please upgrade to v1.0.3 first by following the version upgrade sequence step by step. For example, if you currently have v1.0.0 installed, upgrade sequentially to v1.0.1, v1.0.2, and then v1.0.3. Do not upgrade directly from v1.0.0 to v1.0.3, as this could lead to unforeseen errors.
• This upgrade might result in temporary interruptions in the log data pipeline.
• Check if customers network env has transit gateway? If yes, we MUST manually import opensearch domain. Not automatically import mode.

The following is log hub version list, the last log hub version is v1.2.1.

# LogHub 
v0.1.0
v0.2.0-dev
v0.2.0-test
v0.2.0
v1.0.0
v1.0.1
v1.0.2
v1.0.3 (SO8025) - Log Hub Solution. Template version v1.0.3
v1.1.0-beta
v1.1.0 (SO8025) - Log Hub Solution. Template version v1.1.0
v1.1.1
v1.2.0-syslog-beta
v1.2.0
v1.2.1
[loghub last version] v1.2.2 (SO8025) - Log Hub Solution. Template version v1.2.2
# CLO
v1.0.0 (SO8025) - Centralized Logging with OpenSearch Solution. Template version v1.0.0
v1.0.1
v1.0.2
v1.0.3
v2.0.0
v2.0.1
v2.1.0
v2.1.1

• CentralizedLogging-v101-runtime-fixed.template.json
• CentralizedLogging-v102-runtime-fixed.template.json
• CentralizedLogging-v103-runtime-fixed.template.json
• CentralizedLogging-v104-runtime-fixed.template.json
• CentralizedLoggingFromExistingVPC-v104-runtime-fixed.template.json

https://s3.amazonaws.com/solutions-reference/centralized-logging-with-opensearch/latest/CentralizedLogging.template
https://s3.amazonaws.com/solutions-reference/centralized-logging-with-opensearch/latest/CentralizedLoggingFromExistingVPC.template
https://s3.amazonaws.com/solutions-reference/centralized-logging-with-opensearch/latest/CentralizedLoggingWithOIDC.template
https://s3.amazonaws.com/solutions-reference/centralized-logging-with-opensearch/latest/CentralizedLoggingFromExistingVPCWithOIDC.template

Overview of Steps

1. Update CLO v1.0.3 to v1.0.4 using CloudFormation (CFN) update stack.
2. Deploy CLO v2.0.0 in the same region as CLO v1.0.3.
3. Manually migrate the syslog pipeline (optional).
4. Deploy the migration tool CFN stack in the same region.
5. Deploy CrossAccount.template stack in each member account. And link member account in v2.0.0 console. (optional if you don't have member account)
6. Manually update the EC2 Instance Profile / ASG IAM role policy.
7. Verify the post-deployment functionality.
8. Delete the migration tool CFN stack.
9. Delete the CLO v1.0.3 stack.

Operational Steps

1. Update CLO v1.0.3 to v1.0.4 using CloudFormation update stack.

   In CFN, locate the stack for CLO v1.0.3, click "update," and update it to version v1.0.4. The main purpose of this step is to prevent essential resources (such as VPC and KMS Key) from being deleted upon stack removal.

   

   Depending on whether the CLO v1.0.3 template is for a new VPC or OIDC, select the corresponding v1.0.4 version.

   https://aws-gcr-solutions.s3.amazonaws.com/centralized-logging-with-opensearch/v1.0.4/CentralizedLogging.template
   https://aws-gcr-solutions.s3.amazonaws.com/centralized-logging-with-opensearch/v1.0.4/CentralizedLoggingFromExistingVPC.template
   https://aws-gcr-solutions.s3.amazonaws.com/centralized-logging-with-opensearch/v1.0.4/CentralizedLoggingFromExistingVPCWithOIDC.template
   https://aws-gcr-solutions.s3.amazonaws.com/centralized-logging-with-opensearch/v1.0.4/CentralizedLoggingWithOIDC.template
   

2. Deploy CLO v2.0.0 in the same region as CLO v1.0.3.

   Choose the version of CLO v2.0.0 based on the following scenarios:

   • If v1.0.3 was deployed using a new VPC version (template name does not contain "FromExistingVPC"), deploy the v2.0.0 version of CentralizedLoggingFromExistingVPC.template or CentralizedLoggingFromExistingVPCWithOIDC.template. In this case, use the VPC and Subnets parameters from the v1.0.3 deployment. Refer to the provided images for relevant information.

     
     

   • If v1.0.3 was deployed using the FromExistingVPC version (template name contains "FromExistingVPC"), deploy the v2.0.0 version of CentralizedLoggingFromExistingVPC.template or CentralizedLoggingFromExistingVPCWithOIDC.template.

   Warning

   When deploying v2.0.0 in the China region, follow the instructions at https://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/launch-with-openid-connect-oidc.html. Note the following additional steps:

   1. Prepare a new domain for deploying v2.0.0 (adjust OIDC Provider's callback URLs as needed, illustrated here with Cognito as an example).
      
   2. Alternatively, clear the CNAME record of the original domain (otherwise, CloudFront may fail to bind Alternate domain names due to DNS record conflicts).

3. Manually migrate the syslog pipeline (optional).

   Automatic migration of syslog logs is currently not supported. To migrate the syslog pipeline from v1.0.3, follow these steps:

   1. Create a new syslog log pipeline in CLO v2.0.
   2. Manually connect your applications to the new CLO v2.0 syslog log pipeline endpoint. (Note: This operation will result in a short interruption in log transmission.)
   3. Once you confirm normal log transmission in CLO v2.0, manually delete the syslog log pipeline in CLO v1.0.3.

4. Deploy the migration tool CFN stack in the same region.

   1. Deploy https://aws-gcr-solutions.s3.amazonaws.com/centralized-logging-with-opensearch/v2.0.1-migration-tool/Migration.template.
   2. Enter the parameters:
      1. SourceStackName: Name of the CLO v1.0.3 stack.
      2. DestinationStackName: Name of the CLO v2.0.0 stack.

   

5. Deploy CrossAccount.template stack in each member account. And link member account in v2.0.0 console. (optional if you don't have member account)

   ⚠️ DO NOT DELETE THE OLD CROSSACCOUNT STACK TEMPLATE
   After investigation, it was found that the v1.0.3 CrossAccount table cannot be directly migrated to v2.0.0 (due to added fields and renaming of s3 buckets). Therefore, the process of migrating requires the rebuilding of Subaccount links.

   1. Go to "Member Accounts" section in CLO v2.0.0 console.
   2. Click "Link a member account".
   3. Download the CrossAccount.template file into your local machine.
   4. Edit the file, replace ":role/CL-EKS-LogAgent-Role-*" into ":role/*-EKS-LogAgent-Role-*".
   5. Deploy the edited CrossAccount.template as usual.

6. Manually update the EC2 Instance Profile / ASG IAM role policy.

   Due to slight changes in the EC2 Instance profile in v2.0.0, navigate to the Instance group panel in the v2.0.0 CLO console, find the Instance Permission Policy, and manually add it to the IAM Role of each machine in the Group. It's recommended to create this policy using the "Create inline policy" option.

   

7. Verify the post-deployment functionality.

   Check the availability of features in the v2.0.0 CLO console and ensure that OpenSearch is continuously collecting data. Contact Support if any issues arise.

8. Delete the migration tool CFN stack.

9. Delete the CLO v1.0.3 stack.

It is recommended to observe the operation of v2.0.0 for approximately two weeks after migration. If everything is functioning properly, you can proceed to delete the CLO v1.0.3 CFN stack. The migration process is now complete.

Known Issues

1. After Service/App pipeline migration, the Lambda Processor log-related panel does not display data.

   Due to the pipeline originating from v1.0.3, metrics for log volume of v1.0.3 log pipelines are currently not observable. This is normal behavior. If you wish to see this data, delete the existing pipeline and create a new one.

   

2. Is cross-account migration supported?

   Yes, You need to manually Deploy CrossAccount.template stack in each member account. And then re-link member account in v2.0.0 console.

3. The admin password is changed after migration to CLO v2.0.

   This is by design. If you are deploying with a non-OIDC template. The CLO creates the Cognito UserPool for you to manage Admin usernames and passwords. The migration tool won't migrate Cognito User Pools data. So you must log in to the CLO console with a new username and password.

4. An error occurred (AccessDenied) when calling the ListRolePolicies operation: User: arn:aws:sts::*****:assumed-role/clo-amp-prod-APIAppPipelineAPIAppPipelineHandlerSer-bUXDfcDRJqhO/clo-amp-prod-APIAppPipelineAPIAppPipelineHandler49-otQV7Zs3oHKe is not authorized to perform: iam:ListRolePolicies on resource: role LogHub-AppPipe-f17c7-BufferAccessRoleDF53FD85-1OOE1J2AW7TQC because no identity-based policy allows the iam:ListRolePolicies action

   This is due to the fact that you are deleting the LogHub v1.x Pipeline in CLO v2.0, and the naming of the Role has changed significantly due to the version difference. The upgrade we have tested so far is from CLO v1.x to CLO v2.x, which requires all Pipeline resources to be created in CLO v1.x instead of LogHub version. To resolve this issue, it is recommended that you remove this Pipeline from the LogHub console and then remove this record from the AppPipeline DynamoDB Table.

[wchaws]

TODO

• If users can upgrade from log hub v1.0.3 to v1.2.1 directly?
• If users can upgrade from log hub v1.0.3 to v1.2.1 and then upgrade to CLO v1.0.3?
• If users can upgrade from log hub v1.0.3 to v1.2.1 and then upgrade to CLO v1.0.3, and then migrate to CLO v2.0.0?
• Any issues if users don't upgrade?

Log hub upgrade to guide #85

Log Hub version list

latest => v1.2.1
v0.1.0
v0.2.0-dev
v0.2.0-test
v0.2.0
v1.0.0
v1.0.1
v1.0.2
v1.0.3
v1.1.0-beta
v1.1.0
v1.1.1
v1.2.0-syslog-beta
v1.2.0
v1.2.1
v1.2.2

v1.0.0..v1.0.1

c909a454 (tag: v1.0.1) fix: update mkdocs variable to use latest version cloudformation template in deployment link
e7f8b5b8 fix: hotfix to enable http outbound traffic to fix ssm auto patch issue & add AmazonSSMManagedInstanceCore to enable ssm-agent

v1.0.1..v1.0.3

ebece6dc (tag: v1.0.3) fix: fix gp3 issue
fde7539f fix(Stack): python3.6 is no longer supported by lambda. Update to use python 3.9
37f80e80 doc: fix WorkshopDemo link url

v1.0.3..v1.1.0

7c8021d8 fix: fix the fluent-bit config filepath bug in ec2
123a903d fix: remove failed, creating ingestion from EKS yaml
590182a9 fix: fix change language to change title
86b87ddb fix: change cross account select description
c2e4db01 fix: run-all-tests.sh run twice error
ab74a5a1 fix: fix the template of nginx, Apache bug.
ce260ba0 chore: fix sonarqube Issues.
a97fdb01 fix: remove duplicated padding
f3dcd67a fix: fix bug "EKS Nginx and Apache Create Dashboard failed".
7cf618ef fix: suppress cn pipeline can't get s3://solutions-build-assets/changelog-spec.yml
4960ea1e fix: fix i18n file cache
77efbc2d fix: pipeline failed due to wrong cwd
902f19af fix: global pipeline doesn't support softlink 3
e318a95e fix: global pipeline doesn't support softlink 2
e8155a24 fix: global pipeline doesn't support softlink
29f057b3 Fix: fix workshop and implementation guide typos
d9ae6838 fix: freeze the version of python unit test package
c09d7adb fix: add tips when user import eks cluster
a789efe2 fix: add tips when user import eks cluster
482bfad4 chore: add error output prefix for waf
203445c0 fix: update alarm stack to stay code style consistency
ff22bf6b fix: fix eks notice issue
7ccc43dc fix: add ec2 role to check user if need upgrade pipeline
c14b1eb8 fix: fix create esk ingestion issue
96ebe21d fix: fix ui issues
64f7d486 fix: fix ui issues
ee325201 fix: add upgrade notice when user select exists v1.0 pipeline in eks
d80bd607 fix: fix infomation not show after enable logging bucket
9451d0b1 fix: fix the cross account S3 notification creation and deletion
260850ca fix: fix typo
ca0151cd docs: fix centos openssl implementation guide
fbb3413a Doc: Add instructions for upload iamcertificate in CN region
45935f74 fix: the ARN isn't valid when wafv2.get_logging_configuration due to get global waf resource use regional endpoint
44687b23 fix: the ARN isn't valid when wafv2.get_logging_configuration due to get global waf resource use regional endpoint
0bef5971 fix: iam permission and add source field for LoggingBucket graphql
d7d473b6 fix: fix the deletion of cloudwatch filter
49e46790 fix(Stack): python3.6 is no longer supported by lambda. Update to use python 3.6
3a65029e fix: fix the elb dashboard abnormal requests filter
ea118bc0 fix: remove space in single line and multiline regex
565b2bf6 Fix: allow oidcClientId to be longer than 24 letters && prohibit empty oidcCustomerDomain
0cc15d40 fix: fix "After the stack is upgraded to 1.1, choose not to upgrade the app log pipeline and create an ingestion".
5c59ce50 fix: fix app log ingestion UT.
50a17e54 fix: 1.fix S3 bucket based log ingestion in current account 2.fix list app ingestions bug
f2378152 fix: fix create eks ingestion from existed pipeline in cross account case.
73834aad fix: fix the cfn paramters duplication
0e5a01bf fix: add upgrade pipeline step in eks ingestion
7a9eca74 fix: fix the unit test
76eb8de4 fix: fix the download latest aws-sdk in customer resource
98266fb0 fix: remove slash in role arn
e7b7d01f Fix(stack):fix service log (s3 destination) cannot get cross-account's s3 object. The boto3 was not wrapped by STS and Lambda's role cannot assume cross-account role
77d0a7a9 fix: add regex to check link account input
bcd324fd fix: fix "when the same account EKS Cluster and the cross-account EKS Cluster have the same name, only one of the clusters can be imported" bug
1afdb918 Doc: Add link an account implementation guide, also fixed service-pipiline-cloudwatch's architecture diagram
4da9e356 fix: 1.chang Fluent-Bit docker image version to 2.25.1  2.fix Lambda, RDS log ingestion with cross account bug 3. fix WAF list resource in China-region bug
f0211769 refactor: 1.WAF sampling support for querying all CloudFront WebACLs in non "us-east-1" regions 2.fix WAF schedule bug
1be9e66d refactor: 1.WAF sampling support for querying all CloudFront WebACLs in non "us-east-1" regions 2.fix WAF schedule bug
10c65842 Fix(API):The rolename created during pipeline upgrade doesn't match with SubAccount's assume role pattern
4d691be1 fix: update kds role arn
1ff3e9eb Fix(API): TrustRelationship will not affect immediately while ingest Cross-Account logs. Try to assume the role to make sure the trustRelationship is added before deploy fluent-bit config
a6585444 Fix: fix nginx proxy log rotation bug
399b83c9 fix: fix the log ingestion of RDS, Lambda
ca2c4e60 fix: fix "enable ALB log"  bug & "time key" for app log ingestion.
7a45206c Fix: fix alarm stack parameter bug
176666c2 fix: fix eks permission bug for cross acct scenarios.
9c082994 fix: fix the "pipeline cannot be created successfully when multiple appPipelines are created at the same time" bug.
12b95a1e Fix(API): S3 as Source APP Log ingestion failed to send log to AOS
ba628e2c fix: support create lambda pipeline sample dashboard in ui and add clarify tips
78696d2e fix: fix ui blocker bugs
3cdb9757 fix: fix "log ingestion" bug for CN region.
441c8902 fix: ubuntu incorrect ibsasl2.so.3 installation
00674667 fix: fix "Enable Logging Bucket for Cross Account VPC Flow Log and WAF full log"
0e6b3fe1 fix: 1.fix ec2 ingestion bug 2.add "aws filter"
5ff32e13 fix: 1.fix dashboard bug for eks log ingestion 2. change fluent-bit parameters
259b3248 fix: add upgrade notice when user create ingestion in v1.0 pipeline
44249ca2 fix: fix "Check Agent status" bug
eb06ec79 fix: fix permission bug of eks log ingestion
c8194276 fix: fix suffix error in EKS log pipeline.
f338d85e fix: fix create s3 log source missing account id in current account
70ea330a fix: fix the ut of log source api
a34a5646 fix: add the get logging bucket for Lambda Log Pipeline
658fc865 fix: fix the log processor policy and sub account policy
22f3325d fix: avoid creating pipeline when overlap/duplicate with creating/deleting status
3c6aa8e7 feat: add cross account in eks log with ui and fix ui issues
0f009ca5 fix:1.fix bugs about ec2 log and eks log 2.add the switch flag for obtaining extra eks metadata
dcce6032 fix: force creating pipeline when overlap/duplicate with inactive index prefix
0227fac2 fix: fix "check Config existing" bug
a88d372e fix: fix the service pipeline depoly in current account
fdd19f54 fix: fix "DaemonSet Guide"  and "createDashboard" parameter bugs
c48fcb62 fix: fix "substring not found" error in eks log ingestion.
d4410c8a fix: fix "substring not found" error in eks log ingestion.
498ba8ec fix: fix "substring not found" error in eks log ingestion.
5d2f26d8 fix: fix the output of EKS, app log pipeline sequence bug
3669e184 fix:1.add "kdsRoleArn" for app log Pipeline in graphql 2.add "kdsRoleName" in creating app log pipeline flow
682ca794 fix:fix CloudFormation output sequence.
22afefd1 fix: enable WAFlog
fdcb4afa fix:1. fix unit testing 2.create the KDS role in creating app log pipeline flow 3. add wafsampling feature
b098aa6d fix: add cross account in service pipeline with ui
537ff9f7 fix: add cross account in service pipeline with ui
7b1d1be6 fix: change bucketDomainName to bucketName for logSourceS3NotificationFn env
be61cf6f fix: fix log-process bug for cross-account scenarios.
4ac0cc8b fix: update the UT for pipeline
6267de5b fix: add log path in create esk ingestion and fix duplicated index prefix issue
704f856f fix: fix the unit test for cross account scenarios' instance-meta, app log ingestion and eks-cluster api
3299b67e fix: missing force arguments for createEKSClusterPodLogIngestion API
6baebf58 doc: fix WorkshopDemo link url
24bcdcd0 fix: fix the unit test for instance group and cross account api
1bfbedee fix: fix some unit test
4c0cac4e chore: add the output of service pipeline and fix some bugs
13ec187c fix: fix app ingestion bugs with cross-account scenario
b6cd4d01 feat: add link cross account page and fix ui issues
cbee9085 fix: add the remove action before create a new policy version for cross account api
2b804fe6 fix: cfn_nag
11468dfb fix: Create EKS Ingestion did not check index prefix overlap
57f726a6 fix: app-pipline: validate index prefix failed
9043c990 fix: cfn_nag
0b23db73 fix: update schema.graphql
b4570a0b fix: change pagesize
906999c0 fix: remove global interval variable
6aa7c900 fix: add auto refresh data in async function page and optimize duplicated index prefix tips
f915c0b0 fix: add auto refresh data in async function page and optimize duplicated index prefix tips
81cc2fe9 fix:fix opensearch alarm bug
7aca67f7 fix: fix the gitignore and crossaccount sub stack output
96574e77 fix(API): make createDashboard a request property
2fd85e37 fix: add checkTimeFormatRes in graphql
46bbcea4 fix: Remove OpenSearch before removing EKS cluster
317e22fd fix: fix the error message of source name and config name in duplicate log ingestion
29fc324b fix: support multi date type in config and upgrade node-sass for M1 chip
80edd8af fix: fix the list resource in CloudTrail Pipeline and CloudFront Pipeline
9bebf7c4 fix: add validation for create alarm and fix ui issues
624dff5a fix: update the index_pattern of nginx type log
ee64df07 docs: fix the typo
b92a42aa fix: fix the miss of aws-sdk in package.json
7a54ab8c Fix: enable http outbound traffic to fix ssm auto patch issue

v1.1.0..v1.2.0

44fb922d fix: ci geo ip location test failure
41ada9df fix: fix ci build error
fc3dee85 fix: change ui words
d55cdf3b fix: add aos-agent role in permission
50327aed fix: add asg link and rename s3 to Amazon S3
4128b739 fix: fix ui bugs
75e00aec Fix: doc typo about creating instance groups from ASG
b8c37dba fix: fix the ssm document name changes in cross account scenario
47b5f8db fix: update fluent-bit version to 1.9.9 (docker image version 2.28.4)
99fb2742 fix: fix the cross account fluent-bit version from aws2.21.1 to aws2.25.1
1520b3ac fix: fix ci build failed and add filter samples
ec2f760a fix: fix bug "eks ingestion failed with cross account after upgrade v1.2 from v1.1"
e01ce84b doc: fix the error in ELB/CloudFront CFN parameters
e1c11860 fix: fix ui bugs
82be8e1d fix: fix ui bugs
90e3955d fix: fix the error exception in app ingestion
30cf848c fix: fix lua script bug with eks pattern
ebfa6937 fix: fix bug "time_key error"
014d6217 Fix: asg group cross account issue && proxy launch policy issue
2e36c3fc fix: fix bug "fluent-bit conf error with sidecar pattern"
407c2f1a fix: check buffer s3 input
ad2fa671 fix: fix bug "Action/metadata line [1] contains an unknown parameter [_type] with syslog"
d6823ce5 fix: fix bug "Action/metadata line [1] contains an unknown parameter [_type]"
4c23819c fix: fix ui bugs
39e133ab Fix: fix asg ingestion deletion bug and s3 prefix error
076c1639 fix: fix the syslog nlb timeout issue
2996d153 fix: fix lua script error with ec2 fluent-bit.
0e3fe8bd fix: fix ui bugs
db931a16 fix: fix bug "import aos domain error"
d66a064f fix: fix bug "fluent bit configuration error for app log ingestion with ec2"
e15e17f0 fix: fix bug "EKS Cluster: Multi-Line Parser position need to fixed in yaml file"
d742b339 fix: fix bug "error configmap for sidecar"
6e00996a fix: fix bug "EKS Cluster: ListImportedEKSClusters API return deploymentKind is wrong"
f6fe44c6 fix: api vtl bug for asg and instance group api
31d43f21 fix: fix the rds audit log parser to support CONNECT type logs
34759e65 fix: update UT to unblock pipeline
68373e0d fix: update vtl for asg instance group
e11be898 fix: fix the unit test
e2192f6b fix: add the lua to log-agent-helper v2 and fix the s3 output bug
1132a58a fix: add/delete instance vtl and error reporting message
126f65e7 fix:fix aos bug "如果proxy还在删除的过程中，就remove了AOS domain，会出现list domain 页面错误"
f0ad505e fix: fix the vpc flow logs bug
65f9a8e0 fix:fix bug "Automatic domain import does not work for subnet without explicitly specified route table."
d2c3f010 fix: fix bug "An error occurred when calling EKSAgentConfigGeneration"
012784a5 fix: fix rename bugs and disable kds autoscaling in cn-northwest-1 region
89ec3fed fix: fix rename bugs and disable kds autoscaling in cn-northwest-1 region
fa703089 fix: rename getEKSDeploymentConf
c3bc7b35 fix: portal can\'t not show old pipeline detail
c7f59ef3 fix: fix upgrade error "Only one resolver is allowed per field."
f39162ea fix: portal can\'t not show old pipeline detail
4649e5c4 fix: fix create s3 pipeline with default s3 bucket
1a52f11e fix: add the syslog config type to ingestion helper
2faca498 fix: App Pipeline S3 Buffer has a wrong S3 Notification prefix
80a8e88a fix: App Pipeline S3 Buffer has a wrong S3 Notification prefix
19a259f3 fix: fix fluent-bit ec2 template issues
715be9e2 fix: fix fluent-bit STS assume role
f245152e fix: App Pipeline S3 Buffer has a wrong S3 Notification prefix
e7ab7d37 fix: fix the UT of log conf
24a98f80 fix: app pipeline s3 log processor check the wrong index template
c5d66b5b fix: Added support for OS processors and EKS versions for containerd runtime
b8d7b5ac fix: app pipeline s3 log processor check the wrong index template
1f4ccb26 fix: fix the s3 and opensearch output config generator
7c89d8a1 fix: fix bug "App Log Pipeline Log Processor crash"
f36ac43e fix: 1.encode /decode the conf field "timeRegularExpression" 2. add unit into s3 conf for fluentbit
66c57dee fix: fix eks ingestion bugs
904b9121 fix: fix the check syslog port api
307735f7 fix: add role ARN into "fluent-bit-output-aos.template.yaml " and "fluent-bit-output-kds.template.yaml"
fe8846a3 fix: fix some bugs in schema - log source
a377a8c0 fix: fix ut in log ingestion
68d62409 chore: fix ui with new api
f938dba0 fix: Fix list eks domains missing AOS info
02e33999 fix: Fix list app pipeline missing info error
9a879abe chore: fix error code "6".
cd9cc69f fix: Fix app pipeline status not updated error
6356dfb1 fix: add the cfn nag on Syslog sub stack
5b046d18 1. Fixed Error when processorFilterRegex.enable is False. 2. Add Unit test for processorFilterRegex.enable is False.
eb1a800f fix: fix the missing of deleting-firehose policy
62c519ff Fix: fix elb access log bug
fcaac3e5 doc: fix the architecture description not match issue, update the cost section
7b2e3277 fix: enable time_offset to change timezone
9abdfb84 chore: fix UT error for app pipeline
f8c18433 update API for filter
259084a9 fix: error when time key is null
8fff95aa fix: error when time key is null
756da54c fix: Fix UT error for log conf api
6214d223 <doc> minor fix
a792091c <doc> add security section, translate cost section, translate parameters in cross-account ingestion section, fix errors found in v1.1.0 release
7a443914 fix: Fix missing version issue
417f712e fix: upgrade public.ecr.aws/aws-observability/aws-for-fluent-bit to 2.28.1
32d661e3 fix: upgrade public.ecr.aws/aws-observability/aws-for-fluent-bit to 2.28.1
6528c257 fix: remove format encode and decode in logConf
c4dd83a3 fix: fix logconf encode and decode issues
5fb39b75 fix: app log incorrect timezone
fa95c67d fix: Revised according to VTL uniform specification
da673b44 Fix: Fix proxy validation corner case when domain was not provided
fa766752 fix: Fix conf, proxy, alarm validation error
8d1ce1ef fix: fix security issues of graphQL interface(encode/decode, validation) in resource, service pipeline
c0e79340 fix: fixsecurity issues of graphQL interface(encode/decode, validation) in api, log conf and instance meta stack
632a045b fix: Fix import domain validation error
65068418 fix: encode/decode graphQL interface(EKS, Instance group, Cross Account).
1d24d2dd fix: fix security issues of graphQL interface(encode/decode, validation) in app pipeline, app ingestion and log source
fcceb741 chore: fix CDK_NAG compliance for KDSStack and enabled access log, CWL and request validator for APIGateway
3c57db27 chore: fix CDK_NAG error: S3toKDSStack && part of KDSStack/KDSStackNoAutoScaling
72c47b1c fix: fix CDK_NAG error: Main Stack
36176ea4 fix: fix CDK_NAG error: Service Log From CloudWatch, CrossAccount-stack, waf-sampled-stack
9f609125 chore: fix errors in regional deployment section.
83d8fa63 chore: fix errors in regional deployment section.
851a6797 fix: fix build error in node v14 version
cdc2835d fix: fix dependatbot issues

v1.2.0..v1.2.1

3438828e 1. Fixed creating ingestion error when processorFilterRegex is None.
ad47050e fix: The status of App Log Ingestion is always Creating #61

[wchaws]

升级后希望能把 CloudFront Service 日志从 S3 上恢复。

[wchaws]

如果用户版本太老不升级会碰到 Cloudformation 报 NodeJS 12 不支持问题

[wchaws]

前期调研

• 客户当前使用的版本，主堆栈 CloudFormation 的 Description 发一下。
• 目前使用了哪些功能：
  1. 多少个 Service Pipeline
  2. 多少个 App Pipeline
  3. 是否使用过 EKS 采集日志么？
  4. 是否用了跨账号功能？
  5. 有多少实例组？
  6. 多少日志配置？

如果版本差异太大，比如：Loghub v1.2.2 升级到 CLO v2，建议同时部署 CLO v2 和 Loghub v1.2.2 版本，然后逐步把配置从老版本抄到新版本里面！

拆除重建步骤

Warning
升级过程中会出现日志中断现象，请注意！

Warning
需要检查是否包含 CentOS7

1. 检查客户版本

2. 检查网络环境

3. 安装 VPC 子网 https://gist.github.com/wchaws/6db97e4239b3a17d92a6261b0a7a56d7

4. 安装 https://s3.amazonaws.com/solutions-reference/centralized-logging-with-opensearch/latest/CentralizedLoggingFromExistingVPC.template

5. 手动重建 CLO 资源

   • 对于 App log 重新创建 Log config 和 Instance group 重新创建 App log pipeline 即可。然后去老版本的 CLO 把 App log pipeline 删除即可。
   • 对于 Service log 必须先去老版本 CLO 把 Service log pipeline 删除，之后再去 CLO v2 创建 Service log pipeline，并且请指定一个新的 index name，否则可能会出现 index name 与老版本重名问题。

6. 创建新的 OpenSearch Proxy

[alexzon-tr]

I'm currently using version 1.0.1.
To update to version 1.0.3, do I only need to update the template? Do I need to update every cross-account template to 1.0.3?

After updating to 2.0.0, to update to 2.0.1 is only a matter of updating the template as well?

[wchaws]

@alexzon-tr

For CLO v1.0.1 you can directly update the main template and and the cross-account template to v1.0.3. Then follow the above guide #184 (comment) to upgrade. But I found the migration tool CFN stack (https://aws-gcr-solutions.s3.amazonaws.com/centralized-logging-with-opensearch/migration_tool/Migration.template) will raise exception in some cases which is still under fixing. So the easiest way to upgrade is to directly deploy the v2.0.0 in the same region. And you must create setup a new vpc with this template then deploy CentralizedLoggingFromExistingVPC.template if your v1.0.1's template is CentralizedLogging.template. Otherwise your CLO v2.0.0's vpc CIDR will conflicts with v1.0.1 during vpc peering in the process of importing OpenSearch domain.

After updating to 2.0.0, to update to 2.0.1 is only a matter of updating the template as well?

Yes. Since, v1.0.1 to v2.0.0 is a breaking change the process is a little bit complex.