Update Python Lending Library and Item Tracker examples for compatibi…

…lity with Aurora Serverless v2

Because both of these examples make use of Data API, and Data API for Serverless v2
currently supports Aurora PostgreSQL but not Aurora MySQL, modernizing to Serverless v2
also means switching database engines to a recent major version of Aurora PostgreSQL.

Adapt Lending Library for using Serverless v2 in the demo:

* Update wording in README to focus less on Serverless cluster notion.
* Enhance the code that implements "wait for cluster to be created"
  Make it also wait for the associated DB instance.
* Create a PostgreSQL wrapper for the library app equivalent to the MySQL one.
  Using PostgreSQL data types, SQL idioms, and in particular the RETURNING
  clause for INSERT statements as the way to get back the auto-generated ID
  value(s) from the new rows.

Substantive changes are mainly around auto-increment columns.
Instead of adding the AUTO_INCREMENT keyword into the column definition,
the autoincrement attribute in the Python source just indicates which
column(s) to skip in INSERT statements. It's the responsibility of the
caller to specify one of the PostgreSQL auto-increment types like
SERIAL or BIGSERIAL.

* Add debugging output to see what's being executed for batch SQL statements.
* Add more debugging code around interpretation of results from batch SQL statement.
* Make the insert() operation use a RETURNING * clause.
  Since Data API for PostgreSQL doesn't include data in the generatedFields
  pieces in the result set.
* Make the INSERT statement for the Authors table work from a single big string.
  Supply all the VALUES data as part of the original SQL string and submitted to
  ExecuteStatement, not an array of parameters used with BatchExecuteStatement.

If the VALUES string is constructed with single-quoted values,
e.g. ('firstname','lastname'), then it's vulnerable to a syntax
error for names like O'Quinn that contain a single quote.
So I'm making the delimiters be $$first$$ and $$last$$ to
avoid any possibility of collisions.

* Add some more debugging output around submitting SQL to lend or return books.
  Also exception/debug/tracing code to verify
  exactly which operations fail and what the parameters
  and return values are around that vicinity.

* Change from IS NULL to IS NOT DISTINCT FROM NULL in get_borrowed_books() and return_book().
  Because the substitution at the point of 'null' isn't allowed
  in Postgres with JDBC protocol. Even though it is allowed in MySQL.

* Be more flexible in date/time-related types that get the DATE type hint.
  Don't cast today's date to a string, so it's recognized as a date/time type.

Set up CDK setup path for the cross-service resources to use Serverless v2 instead of Serverless v1:

* Create DatabaseCluster instead of ServerlessCluster.
* Include the 'enable Data API' / 'enable HTTP endpoint' flag.
  Added recently to DatabaseCluster in this pull request: aws/aws-cdk#29338
* Updated the CDK version to 2.132.1, a recent enough version that includes that ^^^ pull request.
* Switch from instanceProps to writer as the attribute of the DatabaseCluster. Based on deprecation
  of instanceProps that happened in Nov 2023.
* Changes to VPC and subnets to get example to work with existing network resources.
  Had to boost VPC and VPC subnets attributes out of the instance and up to the cluster level.
* Switched to Serverless v2 instance for the writer. Now serverless vs. provisioned is a
  different method call instead of asking for a different instance class.

Reformat Python code with 'black' linter.

For the aurora_serverless_app CloudFormation stack:

* Make the cluster and associated instance be Aurora PostgreSQL version 15
* Update CFN stack to refer to the Serverless v2 scaling config attribute.
* Add MinCapacity and MaxCapacity fields with numeric values.
* Take out Autopause attribute.
* See: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-rds-dbcluster-serverlessv2scalingconfiguration.html
* Take out engine mode entirely for cluster. (Serverless v2 uses provisioned engine mode, which is the default.)
* Add RDSDBInstance section. In Serverless v2, the Aurora cluster does have DB instances.
  It's the DB instances that are marked as serverless via the 'db.serverless' instance class.
* Add a DependsOn attribute so instance creation waits for the cluster to be created first.

In the Python item tracker code:

* Apply same Serverless v2 + Data API change to INSERT statement as in PHP and Java examples, which
  were updated in earlier pull requests.
* Turn the DDL statement into a query. Get the auto-generated ID value back from
  "records" instead of "generatedFields".

python/cross_service/aurora_item_tracker/README.md

@@ -44,9 +44,15 @@ python -m pip install -r requirements.txt
 
 ### Aurora Serverless DB cluster and Secrets Manager secret
 
-This example requires an Aurora Serverless DB cluster that contains a MySQL database. The
-database must be configured to use credentials that are contained in a Secrets Manager
-secret. 
+This example requires an Aurora DB cluster with the Data API feature enabled. As of March 2024,
+the available choices are:
+* Aurora PostgreSQL using Aurora Serverless v2 or provisioned instances in the cluster.
+* Aurora MySQL or Aurora PostgreSQL using a Serverless v1 cluster.
+
+For this example, we assume that the Aurora cluster uses a combination of Aurora PostgreSQL
+and Aurora Serverless v2.
+
+The database must be configured to use credentials that are contained in a Secrets Manager secret. 
 
 Follow the instructions in the 
 [README for the Aurora Serverless application](/resources/cdk/aurora_serverless_app/README.md) 
@@ -71,16 +77,16 @@ CloudFormation setup script:
 credentials, such as `arn:aws:secretsmanager:us-west-2:123456789012:secret:docexampleauroraappsecret8B-xI1R8EXAMPLE-hfDaaj`.
 * **DATABASE** — Replace with the name of the database, such as `auroraappdb`.  
 
-*Tip:* The caret `^` is the line continuation character for a Windows command prompt.
-If you run this command on another platform, replace the caret with the line continuation
+*Tip:* The caret `\` is the line continuation character for a Linux or Mac command prompt.
+If you run this command on another platform, replace the backslash with the line continuation
 character for that platform.
 
 ```
-aws rds-data execute-statement ^
-    --resource-arn "CLUSTER_ARN" ^
-    --database "DATABASE" ^
-    --secret-arn "SECRET_ARN" ^
-    --sql "create table work_items (iditem INT AUTO_INCREMENT PRIMARY KEY, description TEXT, guide VARCHAR(45), status TEXT, username VARCHAR(45), archived BOOL DEFAULT 0);"
+aws rds-data execute-statement \
+    --resource-arn "CLUSTER_ARN" \
+    --database "DATABASE" \
+    --secret-arn "SECRET_ARN" \
+    --sql "create table work_items (iditem SERIAL PRIMARY KEY, description TEXT, guide VARCHAR(45), status TEXT, username VARCHAR(45), archived BOOL DEFAULT false);"
 ```
 
 #### AWS Management Console
@@ -102,6 +108,19 @@ as `auroraappdb`.
 This opens a SQL query console. You can run any SQL queries here that you want. Run the 
 following to create the `work_items` table:
 
+For a PostgreSQL-compatible database:
+```sql
+create table work_items ( 
+  iditem SERIAL PRIMARY KEY, 
+  description TEXT, 
+  guide VARCHAR(45), 
+  status TEXT, 
+  username VARCHAR(45), 
+  archived BOOL DEFAULT false
+);
+
+```
+For a MySQL-compatible database:
 ```sql
 create table work_items ( 
   iditem INT AUTO_INCREMENT PRIMARY KEY, 

python/cross_service/aurora_item_tracker/storage.py

@@ -112,14 +112,26 @@ def add_work_item(self, work_item):
         """
         Adds a work item to the database.
 
+        Note: Wrapping the INSERT statement in a WITH clause and querying the auto-generated
+              ID value is required by a change in the Data API for Aurora Serverless v2.
+              The "generatedFields" fields in the return value for DML statements are all blank.
+              That's why the RETURNING clause is needed to specify the columns to return, and
+              the entire statement is structured as a query so that the returned value can
+              be retrieved from the "records" result set.
+
+              This limitation might not be permanent; the DML statement might be simplified
+              in future.
+
         :param work_item: The work item to add to the database. Because the ID
                           and archive fields are auto-generated,
                           you don't need to specify them when creating a new item.
         :return: The generated ID of the new work item.
         """
         sql = (
+            f"WITH t1 AS ( "
             f"INSERT INTO {self._table_name} (description, guide, status, username) "
-            f" VALUES (:description, :guide, :status, :username)"
+            f" VALUES (:description, :guide, :status, :username) RETURNING iditem "
+            f") SELECT iditem FROM t1"
         )
         sql_params = [
             {"name": "description", "value": {"stringValue": work_item["description"]}},
@@ -128,7 +140,10 @@ def add_work_item(self, work_item):
             {"name": "username", "value": {"stringValue": work_item["username"]}},
         ]
         results = self._run_statement(sql, sql_params=sql_params)
-        work_item_id = results["generatedFields"][0]["longValue"]
+// Old style, for Serverless v1:
+//        work_item_id = results["generatedFields"][0]["longValue"]
+// New style, for Serverless v2:
+        work_item_id = results["records"][0][0]["longValue"]
         return work_item_id
 
     def archive_work_item(self, iditem):

python/cross_service/aurora_rest_lending_library/README.md

@@ -7,7 +7,7 @@ Service (Amazon RDS) API and AWS Chalice to create a REST API backed by an
 Amazon Aurora database. The web service is fully serverless and represents
 a simple lending library where patrons can borrow and return books. Learn how to:
 
-* Create and manage a serverless Amazon Aurora database cluster.
+* Create and manage a serverless Amazon Aurora database. This example uses Aurora Serverless v2.
 * Use AWS Secrets Manager to manage database credentials.
 * Implement a data storage layer that uses Amazon RDS Data Service to move data into
 and out of the database.  
@@ -47,29 +47,15 @@ all must be run separately.
 
 ---
 
-### 1. Database deployment
-
-This database setup includes:
-* an Amazon Aurora serverless data cluster
-* an AWS Secrets Manager secret to hold the database user credentials.
-
-This infrastructure is defined in a [setup.ts](resources/cdk/aurora_serverless_app/setup.ts) (see [README.md](resources/cdk/aurora_serverless_app/README.md)).
-
-To execute this CDK stack, run:
-```
-cdk deploy
-```
-The output will show `ClusterArn`, `DbName`, and `SecretArn`. Make sure these values are reflected in the [config for this project](config.yml).
-
----
-
-### 2. Populate database
+    Creates an Amazon Aurora cluster and associated Aurora Serverless v2 database instance.
+    Also creates an AWS Secrets Manager secret to hold the database user credentials.
+    It fills the database with example data pulled from the [Internet Archive's Open Library](https://openlibrary.org). 
 
 Fill the database with example data pulled from the [Internet Archive's Open Library](https://openlibrary.org). 
 
 ```
-python library_demo.py populate_data
-```
+python library_demo.py deploy_database
+``` 
 
 The database is now ready and can be accessed through the 
 [AWS Console Query Editor](https://console.aws.amazon.com/rds/home?#query-editor:) 

python/cross_service/aurora_rest_lending_library/library_api/app.py

@@ -50,7 +50,7 @@ def index():
     """Briefly describes the REST API."""
     return {
         "description": "A simple lending library REST API that runs entirely on "
-        "serverless components."
+        "serverless components: Aurora Serverless v2, API Gateway"
     }
 
 
@@ -131,8 +131,15 @@ def add_patron():
 
     :return: The ID of the added patron.
     """
-    patron_id = get_storage().add_patron(app.current_request.json_body)
-    return {"Patrons.PatronID": patron_id}
+    try:
+        patron_id = get_storage().add_patron(app.current_request.json_body)
+        return {"Patrons.PatronID": patron_id}
+    except Exception as err:
+        logger.exception(
+            f"Got exception in add_patron() inside library_api/app.py: {str(err)}"
+        )
+        logger.exception(f"Returning None instead of patron_id.")
+        return None
 
 
 @app.route("/patrons/{patron_id}", methods=["DELETE"])
@@ -155,7 +162,15 @@ def list_borrowed_books():
 
     :return: The list of currently borrowed books.
     """
-    return {"books": get_storage().get_borrowed_books()}
+    try:
+        json_doc = {"books": get_storage().get_borrowed_books()}
+        return json_doc
+    except Exception as err:
+        logger.exception(
+            f"Exception while calling get_storage().get_borrowed_books(): {str(err)}"
+        )
+        logger.exception(f"Continuing with blank list of borrowed books...")
+        return {"books": []}
 
 
 @app.route("/lending/{book_id}/{patron_id}", methods=["PUT", "DELETE"])