Merge branch 'main' into features/37_move_init.sql_to_use_alembic

README.md

@@ -1,25 +1,48 @@
 # User and Financial Institutions Management API
 This app communicates with Keycloak to provide some user management functionality, as well as serving as `Institutions API` to retrieve information about institutions.
 
+---
+## Contact Us
+If you have an inquiry or suggestion for the user and financial institutions management API or any SBL related code, please reach out to us at <sblhelp@cfpb.gov>
+
 ---
 ### Dependencies
 - [Poetry](https://python-poetry.org/) is used as the package management tool. Once installed, just running `poetry install` in the root of the project should install all the dependencies needed by the app.
 - [Docker](https://www.docker.com/) is used for local development where ancillary services will run.
 - [jq](https://jqlang.github.io/jq/download/) is used for parsing API responses in the curl command examples shown below
 
 ---
-## Pre-requesites
+## Pre-requisites
 [SBL Project Repo](https://github.com/cfpb/sbl-project) contains the `docker-compose.yml` to run the ancillary services. 
 - Not all services need to run, this module `regtech-user-fi-management` is part of the docker compose file, which doesn't need to be ran in docker for local development.
 - Issuing `docker compose up -d pg keycloak` would start the necessary services (postgres, and keycloak)
+```bash
+$ cd ~/Projects/sbl-project
+$ docker compose up -d pg keycloak        
+[+] Running 3/3
+ ⠿ Network sbl-project_default       Created     0.2s
+ ⠿ Container sbl-project-pg-1        Started     2.6s
+ ⠿ Container sbl-project-keycloak-1  Started     13.4s
+```
+![Docker](images/sbl_project_svcs.png)
 
 ---
 ## Running the app
-Once the [Dependencies](#dependencies), and [Pre-requesites](#pre-requesites) have been satisfied, we can run the app by going into the `src` folder, then issue the poetry run command:
+Once the [Dependencies](#dependencies), and [Pre-requisites](#pre-requisites) have been satisfied:
+-  All dependencies installed
+-  Postgres and keycloak services started
+
+we can run the app by going into the `src` folder, then issue the poetry run command:
 ```bash
-cd src
-poetry run uvicorn main:app --reload --port 8888
+$ poetry run uvicorn main:app --reload --port 8888
+INFO:     Will watch for changes in these directories: ['/Projects/regtech-user-fi-management/src']
+INFO:     Uvicorn running on http://127.0.0.1:8888 (Press CTRL+C to quit)
+INFO:     Started reloader process [37993] using StatReload
+INFO:     Started server process [37997]
+INFO:     Waiting for application startup.
+INFO:     Application startup complete.
 ```
+
 ### Local development notes
 - [.env.template](.env.template) is added to allow VS Code to search the correct path for imports when writing tests, just copy the [.env.template](.env.template) file into `.env` file locally
 - [src/.env.template](./src/.env.template) is added as the template for the app's environment variables, appropriate values are already provided in [.env.local](./src/.env.local) for local development. If `ENV` variable with default of `LOCAL` is changed, copy this template into `src/.env`, and provide appropriate values, and set all listed empty variables in the environment.
@@ -41,6 +64,17 @@ curl 'localhost:8880/realms/regtech/protocol/openid-connect/token' \
 --data-urlencode 'client_id=regtech-client' | jq -r '.access_token'
 ```
 
+For local development, we can retrieve the access token and store it to local variable then use this variable on API calls.
+```bash
+export RT_ACCESS_TOKEN=$(curl 'localhost:8880/realms/regtech/protocol/openid-connect/token' \
+-X POST \
+-H 'Content-Type: application/x-www-form-urlencoded' \
+--data-urlencode 'username=user1' \
+--data-urlencode 'password=user' \
+--data-urlencode 'grant_type=password' \
+--data-urlencode 'client_id=regtech-client' | jq -r '.access_token')
+```
+
 ---
 ## Functionalities
 There are 2 major functionalities provided by this app, one serves as the integration with Keycloak, and the other to integrate with Institutions database to show institutions' information. Below are the routers for these functionalities. As mentioned above, authentication is required to access the endpoints.
@@ -110,13 +144,114 @@ There are 2 major functionalities provided by this app, one serves as the integr
     ```
 For both these routers, the needed roles to access each endpoint is decorated with the `@requires` decorator, i.e. `@requires(["query-groups", "manage-users"])`. Refer to [institutions router](./src/routers/institutions.py) for the decorator example; these roles corresponds to Keycloak's roles.
 
+---
+## Example of Local Development Flow
+- Install [Poetry](https://python-poetry.org/), [Docker](https://www.docker.com/) and [jq](https://jqlang.github.io/jq/download/)
+- Checkout [SBL Project Repo](https://github.com/cfpb/sbl-project) and [SBL User and Financial Institution Management API Repo](https://github.com/cfpb/regtech-user-fi-management)
+- Open a terminal, and run keycloak and postqres
+```bash
+# go to sbl-project repo root directory
+cd ~/Projects/sbl-project
+
+# bring up postgres and keycloak 
+docker compose up -d pg keycloak        
+[+] Running 3/3
+ ⠿ Network sbl-project_default       Created     0.2s
+ ⠿ Container sbl-project-pg-1        Started     2.6s
+ ⠿ Container sbl-project-keycloak-1  Started     13.4s
+```
+
+- Open another terminal, and run keycloak and postqres
+```bash
+# go to regtech-user-fi-management's src directory
+cd ~/Projects/regtech-user-fi-management/src
+
+# run local api server
+poetry run uvicorn main:app --reload --port 8888
+
+INFO:     Will watch for changes in these directories: ['/Users/harjatia/Projects/regtech-user-fi-management/src']
+INFO:     Uvicorn running on http://127.0.0.1:8888 (Press CTRL+C to quit)
+INFO:     Started reloader process [42182] using StatReload
+INFO:     Started server process [42186]
+INFO:     Waiting for application startup.
+INFO:     Application startup complete.
+INFO:     127.0.0.1:51374 - "GET /v1/admin/me HTTP/1.1" 200 OK
+```
+
+- Using first terminal or new terminal, run and test API Calls
+```bash
+# Retrieve and store token key
+export RT_ACCESS_TOKEN=$(curl 'localhost:8880/realms/regtech/protocol/openid-connect/token' \
+    -X POST \
+    -H 'Content-Type: application/x-www-form-urlencoded' \
+    --data-urlencode 'username=admin1' \
+    --data-urlencode 'password=admin' \
+    --data-urlencode 'grant_type=password' \
+    --data-urlencode 'client_id=regtech-client' | jq -r '.access_token')
+
+# test token key
+curl localhost:8888/v1/admin/me -H "Authorization: Bearer ${RT_ACCESS_TOKEN}" | jq -r '.'
+  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
+                                 Dload  Upload   Total   Spent    Left  Speed
+100   801  100   801    0     0   7545      0 --:--:-- --:--:-- --:--:--  8010
+{
+  "claims": {
+    "exp": 1697482762,
+    "iat": 1697482462,
+    "jti": "3380a7bb-765b-4197-a500-19eb424518ce",
+    "iss": "http://localhost:8880/realms/regtech",
+    "aud": [
+      "realm-management",
+      "account"
+    ],
+    "sub": "631dbab3-4dcf-46bf-b283-55c18a3722e6",
+    "typ": "Bearer",
+    "azp": "regtech-client",
+    "session_state": "c8d4b4c8-3d3f-41b3-aebc-b4cab3b314eb",
+    "allowed-origins": [
+      "*"
+    ],
+    "realm_access": {
+      "roles": [
+        "offline_access",
+        "uma_authorization",
+        "default-roles-regtech"
+      ]
+    },
+    "resource_access": {
+      "realm-management": {
+        "roles": [
+          "manage-users",
+          "query-groups"
+        ]
+      },
+      "account": {
+        "roles": [
+          "manage-account"
+        ]
+      }
+    },
+    "scope": "email profile",
+    "sid": "c8d4b4c8-3d3f-41b3-aebc-b4cab3b314eb",
+    "email_verified": false,
+    "preferred_username": "admin1",
+    "given_name": "",
+    "family_name": ""
+  },
+  "name": "",
+  "username": "admin1",
+  "email": "",
+  "id": "631dbab3-4dcf-46bf-b283-55c18a3722e6",
+  "institutions": []
+}
+```
+
 ---
 ## API Documentation
 This module uses the [FastAPI](https://fastapi.tiangolo.com/) framework, which affords us built-in [Swagger UI](https://swagger.io/tools/swagger-ui/), this can be accessed by going to `http://localhost:8888/docs`
 - _Note_: The `Try It Out` feature does not work within the Swagger UI due to the use of `AuthenticationMiddleware`
 
 ---
-
 ## Open source licensing info
 
 1. [TERMS](TERMS.md)

src/.env.local

@@ -13,4 +13,7 @@ INST_DB_USER=fi
 INST_DB_PWD=fi
 INST_DB_HOST=localhost:5432
 INST_DB_SCHEMA=public
-INST_CONN=postgresql+asyncpg://${INST_DB_USER}:${INST_DB_PWD}@${INST_DB_HOST}/${INST_DB_NAME}
+INST_CONN=postgresql+asyncpg://${INST_DB_USER}:${INST_DB_PWD}@${INST_DB_HOST}/${INST_DB_NAME}
+JWT_OPTS_VERIFY_AT_HASH="false"
+JWT_OPTS_VERIFY_AUD="false"
+JWT_OPTS_VERIFY_ISS="false"

src/config.py

@@ -0,0 +1,61 @@
+import os
+from typing import Dict, Any
+
+from pydantic import TypeAdapter
+from pydantic.networks import HttpUrl, PostgresDsn
+from pydantic.types import SecretStr
+from pydantic_settings import BaseSettings, SettingsConfigDict
+
+JWT_OPTS_PREFIX = "jwt_opts_"
+
+env_files_to_load = [".env"]
+if os.getenv("ENV", "LOCAL") == "LOCAL":
+    env_files_to_load.append(".env.local")
+
+
+class Settings(BaseSettings):
+    inst_conn: PostgresDsn
+    inst_db_schema: str = "public"
+    auth_client: str
+    auth_url: HttpUrl
+    token_url: HttpUrl
+    certs_url: HttpUrl
+    kc_url: HttpUrl
+    kc_realm: str
+    kc_admin_client_id: str
+    kc_admin_client_secret: SecretStr
+    kc_realm_url: HttpUrl
+    jwt_opts: Dict[str, bool | int] = {}
+
+    def __init__(self, **data):
+        super().__init__(**data)
+        self.set_jwt_opts()
+
+    def set_jwt_opts(self) -> None:
+        """
+        Converts `jwt_opts_` prefixed settings, and env vars into JWT options dictionary.
+        all options are boolean, with exception of 'leeway' being int
+        valid options can be found here:
+        https://github.com/mpdavis/python-jose/blob/4b0701b46a8d00988afcc5168c2b3a1fd60d15d8/jose/jwt.py#L81
+
+        Because we're using model_extra to load in jwt_opts as a dynamic dictionary,
+        normal env overrides does not take place on top of dotenv files,
+        so we're merging settings.model_extra with environment variables.
+        """
+        jwt_opts_adapter = TypeAdapter(int | bool)
+        self.jwt_opts = {
+            **self.parse_jwt_vars(jwt_opts_adapter, self.model_extra.items()),
+            **self.parse_jwt_vars(jwt_opts_adapter, os.environ.items()),
+        }
+
+    def parse_jwt_vars(self, type_adapter: TypeAdapter, setting_variables: Dict[str, Any]) -> Dict[str, bool | int]:
+        return {
+            key.lower().replace(JWT_OPTS_PREFIX, ""): type_adapter.validate_python(value)
+            for (key, value) in setting_variables
+            if key.lower().startswith(JWT_OPTS_PREFIX)
+        }
+
+    model_config = SettingsConfigDict(env_file=env_files_to_load, extra="allow")
+
+
+settings = Settings()

src/entities/engine/engine.py

@@ -1,14 +1,14 @@
-import os
 from sqlalchemy.ext.asyncio import (
     create_async_engine,
     async_sessionmaker,
     async_scoped_session,
 )
 from asyncio import current_task
+from config import settings
 
-DB_URL = os.getenv("INST_CONN")
-DB_SCHEMA = os.getenv("INST_DB_SCHEMA", "public")
-engine = create_async_engine(DB_URL, echo=True).execution_options(schema_translate_map={None: DB_SCHEMA})
+engine = create_async_engine(settings.inst_conn.unicode_string(), echo=True).execution_options(
+    schema_translate_map={None: settings.inst_db_schema}
+)
 SessionLocal = async_scoped_session(async_sessionmaker(engine, expire_on_commit=False), current_task)
 
 

src/entities/models/dto.py

@@ -1,4 +1,4 @@
-from typing import List, Dict, Any, Set, Optional
+from typing import List, Dict, Any, Set
 from pydantic import BaseModel
 from starlette.authentication import BaseUser
 
@@ -15,7 +15,7 @@ class FinancialInsitutionDomainDto(FinancialInsitutionDomainBase):
     lei: str
 
     class Config:
-        orm_mode = True
+        from_attributes = True
 
 
 class FinancialInstitutionBase(BaseModel):
@@ -26,7 +26,7 @@ class FinancialInstitutionDto(FinancialInstitutionBase):
     lei: str
 
     class Config:
-        orm_mode = True
+        from_attributes = True
 
 
 class FinancialInstitutionWithDomainsDto(FinancialInstitutionDto):
@@ -37,13 +37,13 @@ class DeniedDomainDto(BaseModel):
     domain: str
 
     class Config:
-        orm_mode = True
+        from_attributes = True
 
 
 class UserProfile(BaseModel):
     first_name: str
     last_name: str
-    leis: Optional[Set[str]]
+    leis: Set[str] | None = None
 
     def to_keycloak_user(self):
         return {"firstName": self.first_name, "lastName": self.last_name}

src/main.py

@@ -1,6 +1,4 @@
-import os
 import logging
-import env  # noqa: F401
 from http import HTTPStatus
 from fastapi import FastAPI, HTTPException, Request
 from fastapi.responses import JSONResponse
@@ -12,6 +10,8 @@
 
 from oauth2 import BearerTokenAuthBackend
 
+from config import settings
+
 log = logging.getLogger()
 
 app = FastAPI()
@@ -32,7 +32,9 @@ async def general_exception_handler(request: Request, exception: Exception) -> J
     )
 
 
-oauth2_scheme = OAuth2AuthorizationCodeBearer(authorizationUrl=os.getenv("AUTH_URL"), tokenUrl=os.getenv("TOKEN_URL"))
+oauth2_scheme = OAuth2AuthorizationCodeBearer(
+    authorizationUrl=settings.auth_url.unicode_string(), tokenUrl=settings.token_url.unicode_string()
+)
 
 app.add_middleware(AuthenticationMiddleware, backend=BearerTokenAuthBackend(oauth2_scheme))
 app.add_middleware(