2024.01 release

Signed-off-by: Junbo Liang <68558268+junbo75@users.noreply.github.com>

.github/ISSUE_TEMPLATE/bug_report.md

@@ -7,16 +7,14 @@ assignees: ''
 
 ---
 If you have an active AWS support contract, please open a case with AWS Premium Support team using the below documentation to report the issue:
-
 https://docs.aws.amazon.com/awssupport/latest/user/case-management.html
 
-Before submitting a new issue, please search through open GitHub Issues (https://github.com/aws/res/issues) and check out the troubleshooting documentation.
-
 Please make sure to add the following data in order to facilitate the root cause detection.
+
 **Describe the bug**
-A clear and concise description of what the bug is.
+A clear and concise description of what the bug is. Before submitting a new issue, please search through open [GitHub Issues](https://github.com/aws/res/issues) and check out the [troubleshooting documentation](https://github.com/aws/res/wiki/Troubleshooting).
 
-**To Reproduce**
+**Steps to reproduce**
 Steps to reproduce the behavior:
 1. Go to '...'
 2. Click on '....'
@@ -26,13 +24,18 @@ Steps to reproduce the behavior:
 **Expected behavior**
 A clear and concise description of what you expected to happen.
 
-**Screenshots**
-If applicable, add screenshots to help explain your problem.
+**Actual behavior**
+A clear and concise description of what actually happened.
 
-**Desktop (please complete the following information):**
- - OS: [e.g. iOS]
- - Browser [e.g. chrome, safari]
- - Version [e.g. 22]
+**Screenshots/Video**
+If applicable, add screenshots and/or a video to help explain your problem.
+
+**Environment (please complete the following information):**
+ - RES Version: [e.g. 2023.11]
+ - Software Stack AMI ID: [e.g. ami-0fceec18b58bfda68]
+ - Software Stack OS: [e.g. Windows, Amazon Linux 2, CentOS 7, RedHat Enterprise Linux 7]
 
 **Additional context**
-Add any other context about the problem here.
+Add any other context about the problem here.
+
+

CHANGELOG.md

@@ -1,5 +1,30 @@
 # Change Log
-All notable changes to this project will be documented in this file.
+This file is used to list changes made in each release of Research and Engineering Studio (RES).
 
-The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
-and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+2024.01
+------
+
+**ENHANCEMENTS**
+
+- Add support for snapshots that enable migration between versions of RES.
+  - The migration process involves taking a snapshot, deploying the new version (e.g. 2024.01) and applying the snapshot from the previous version (e.g. 2023.11) on the new version. Allows for admins to confirm the new version works before transferring users.
+- Add support for private subnets.
+  - Enable deployments of RES infrastructure hosts in private subnets with internet access.
+  - RES infrastructure hosts refer to the Cluster Manager, Bastion Host, VDC Gateway, VDC Controller, and VDC Broker.
+- Deprecation of the analytics stack.
+  - Removed required OpenSearch dependency.
+  - Reduces RES environment deployment and deletion time by approximately 30 minutes.
+- Add support for use of custom Amazon Linux 2 EC2 AMI for RES infrastructure hosts.
+  - Enable specifying an AL2 EC2 AMI use for RES infrastructure hosts for users that require specific software or updates installed on their hosts.
+  - RES infrastructure hosts refer to the Cluster Manager, Bastion Host, VDC Gateway, VDC Controller, and VDC Broker.
+- Add support for ldap_id_mapping “True” in SSSD.
+  - Previously the AD sync code required the groups and users to have POSIX attributes uidNumber and gidNumber in order to sync with RES. This conflicted with the IDs generated by SSSD, potentially causing RES users to not be able to access filesystems if they were using SSSD with other systems (e.g. ParallelCluster) .
+- Add support for four new regions Asia Pacific (Tokyo), Asia Pacific (Seoul), Canada (Central), Europe (Milan).
+- Add ability to add users to projects. Previously only groups could be added to project permissions.
+
+**BUG FIXES**
+
+- Added validation for FSx ONTAP filesystem creation
+- Narrowed installation IAM permissions
+- Skipped deletion of batteries included related resources
+- VDI no longer tries to mount filesystems after removing filesystem from project

README.md

@@ -31,7 +31,7 @@ If you have a [support plan](https://aws.amazon.com/premiumsupport/) with AWS Su
 
 You can also [open an issue](https://github.com/aws/res/issues/new/choose) and choose from one of our templates for guidance, bug reports, or feature requests. Please check for open [similar issues](https://github.com/aws/res/issues) before opening another one.
 
-##More Resources
+## More Resources
 
 * [Changelog](https://github.com/aws/res/blob/mainline/CHANGELOG.md)
 * [Amazon Web Services Discussion Forums](https://repost.aws/) 

RES_VERSION.txt

@@ -1 +1 @@
-2023.11
+2024.01

codeCoverage.yml

@@ -5,7 +5,6 @@ source/idea/idea-data-model/src/ideadatamodel/aws:                    0.8
 source/idea/idea-data-model/src/ideadatamodel/cluster_resources:      0.8
 source/idea/idea-data-model/src/ideadatamodel/common:                 0.8
 source/idea/idea-data-model/src/ideadatamodel:                        0.8
-source/idea/idea-sdk/src/ideasdk/analytics:                           0.8
 source/idea/idea-sdk/src/ideasdk/auth:                                0.8
 source/idea/idea-sdk/src/ideasdk/aws:                                 0.8
 source/idea/idea-sdk/src/ideasdk/utils:                               0.8

pyproject.toml

@@ -39,12 +39,13 @@ cdk-app = "idea.app:main"
 
 [project.optional-dependencies]
 dev = [
-    "black",
+    "black~=24.1.0",
     "tox",
     "pytest",
     "pytest-cov",
     "boto3-stubs-lite[essential]",
     "boto3-stubs-lite[cloudformation]",
+    "types-requests",
 ]
 
 [tool.black]

requirements/dev.txt

@@ -1,4 +1,4 @@
-aiofiles==0.8.0
+aiofiles==23.2.1
 alembic==1.7.7
 arrow==1.2.1
 astroid==2.12.11
@@ -33,10 +33,10 @@ dill==0.3.5.1
 exceptiongroup==1.0.0rc8
 fastcounter==1.1.0
 ghp-import==2.1.0
-greenlet==1.1.2
+greenlet>=1.1.2
 httptools==0.4.0
 idna==3.3
-importlib-metadata==4.11.3
+importlib-metadata>=4.11.3
 iniconfig==1.1.1
 invoke==1.7.1
 ipaddress==1.0.23
@@ -59,7 +59,6 @@ multidict==6.0.2
 mypy==0.950
 mypy-extensions==0.4.3
 openapi-schema-pydantic==1.2.4
-opensearch-py==2.3.1
 orjson==3.6.5
 packaging==21.3
 pep517==0.12.0
@@ -102,6 +101,7 @@ rich==12.4.1
 s3transfer==0.6.0
 sanic==23.6.0
 sanic-routing==23.6.0
+selenium==4.16.0
 semver==2.13.0
 sh==1.14.2
 shortuuid==1.0.9

requirements/doc.txt

@@ -1,6 +1,6 @@
 click==8.1.3
 ghp-import==2.1.0
-importlib-metadata==4.11.3
+importlib-metadata>=4.11.3
 jinja2==3.1.2
 markdown==3.3.7
 markupsafe==2.1.1

requirements/idea-administrator.txt

@@ -1,4 +1,4 @@
-aiofiles==0.8.0
+aiofiles==23.2.1
 alembic==1.8.0
 arrow==1.2.1
 attrs==21.4.0
@@ -27,7 +27,7 @@ decorator==5.1.1
 defusedxml==0.7.1
 exceptiongroup==1.0.0rc8
 fastcounter==1.1.0
-greenlet==1.1.2
+greenlet>=1.1.2
 httptools==0.4.0
 idna==3.3
 ipaddress==1.0.23
@@ -39,7 +39,6 @@ markupsafe==2.1.1
 multidict==6.0.2
 mypy==0.950
 mypy-extensions==0.4.3
-opensearch-py==2.3.1
 orjson==3.6.5
 prettytable==3.3.0
 prometheus-client==0.14.1

requirements/idea-cluster-manager.txt

@@ -1,4 +1,4 @@
-aiofiles==0.8.0
+aiofiles==23.2.1
 alembic==1.8.0
 arrow==1.2.1
 banal==1.0.6
@@ -16,7 +16,7 @@ cryptography==41.0.4
 dataset==1.5.2
 decorator==5.1.1
 fastcounter==1.1.0
-greenlet==1.1.2
+greenlet>=1.1.2
 httptools==0.4.0
 idna==3.3
 jinja2==3.1.2
@@ -27,7 +27,6 @@ markupsafe==2.1.1
 multidict==6.0.2
 mypy==0.950
 mypy-extensions==0.4.3
-opensearch-py==2.3.1
 orjson==3.6.5
 prettytable==3.3.0
 prometheus-client==0.14.1

requirements/idea-dev-lambda.in

@@ -1,2 +1 @@
-opensearch-py
 cryptography

requirements/idea-dev-lambda.txt

@@ -3,7 +3,6 @@ cffi==1.15.1
 charset-normalizer==2.1.1
 cryptography==41.0.4
 idna==3.4
-opensearch-py==2.3.1
 pycparser==2.21
 requests==2.31.0
 urllib3==1.26.18

requirements/idea-scheduler.txt

@@ -1,4 +1,4 @@
-aiofiles==0.8.0
+aiofiles==23.2.1
 alembic==1.7.7
 arrow==1.2.1
 banal==1.0.6
@@ -17,7 +17,7 @@ cryptography==41.0.4
 dataset==1.5.2
 decorator==5.1.1
 fastcounter==1.1.0
-greenlet==1.1.2
+greenlet>=1.1.2
 httptools==0.4.0
 idna==3.3
 jinja2==3.1.2
@@ -27,7 +27,6 @@ markupsafe==2.1.1
 multidict==6.0.2
 mypy==0.950
 mypy-extensions==0.4.3
-opensearch-py==2.3.1
 orjson==3.6.5
 prettytable==3.3.0
 prometheus-client==0.14.1

requirements/idea-sdk.in

@@ -23,7 +23,6 @@ fastcounter
 psutil
 blinker
 troposphere
-opensearch-py
 Jinja2
 cryptography
 PyJWT