Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for AWS IoT Core device #2226

Merged
merged 101 commits into from
Jun 23, 2023
Merged

Conversation

kbeaugrand
Copy link
Member

Description

What's new?

This PR concerns the addition of AWS IoT Core support to the portal.
On the Azure side, nothing should change in terms of functionality.

See #1783 to find out which features were introduced during this PR.

This introduces some breaks as the configuration has changed, so a new major release will be required.

What kind of change does this PR introduce?

  • Bugfix
  • Feature
  • Code style update (formatting, local variables)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • CI related changes
  • Documentation content changes
  • Tests
  • Other

@codecov
Copy link

codecov bot commented Jun 18, 2023

Codecov Report

Patch coverage: 85.16% and project coverage change: -3.91 ⚠️

Comparison is base (0ba44ea) 92.39% compared to head (d15718a) 88.49%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2226      +/-   ##
==========================================
- Coverage   92.39%   88.49%   -3.91%     
==========================================
  Files         277      296      +19     
  Lines        9129    11301    +2172     
==========================================
+ Hits         8435    10001    +1566     
- Misses        694     1300     +606     
Impacted Files Coverage Δ
.../IoTHub.Portal.Application/Helpers/ConfigHelper.cs 87.03% <ø> (ø)
.../IoTHub.Portal.Application/Helpers/DeviceHelper.cs 100.00% <ø> (ø)
....Portal.Application/Mappers/ConcentratorProfile.cs 100.00% <ø> (ø)
...l.Application/Mappers/DeviceModelCommandProfile.cs 100.00% <ø> (ø)
...IoTHub.Portal.Application/Mappers/DeviceProfile.cs 94.89% <ø> (ø)
...ortal.Application/Mappers/DevicePropertyProfile.cs 100.00% <ø> (ø)
...Hub.Portal.Application/Mappers/DeviceTagProfile.cs 100.00% <ø> (ø)
...plication/Mappers/EdgeDeviceModelCommandProfile.cs 100.00% <ø> (ø)
...rtal.Application/Mappers/EdgeDeviceModelProfile.cs 100.00% <ø> (ø)
...ub.Portal.Application/Mappers/EnumerableProfile.cs 100.00% <ø> (ø)
... and 157 more

... and 249 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@kbeaugrand kbeaugrand marked this pull request as ready for review June 18, 2023 19:15
@kbeaugrand kbeaugrand requested a review from a team as a code owner June 18, 2023 19:15
Comment on lines +59 to +62
catch (Exception e)
{
this.logger.LogError(e, "Sync Greengrass Deployents job has failed");
}

Check notice

Code scanning / CodeQL

Generic catch clause

Generic catch clause.
Comment on lines +60 to +63
catch (Exception e)
{
this.logger.LogError(e, "Sync Thing Types job has failed");
}

Check notice

Code scanning / CodeQL

Generic catch clause

Generic catch clause.
Comment on lines +80 to +83
catch (Exception e)
{
this.logger.LogError(e, "Sync Things job has failed");
}

Check notice

Code scanning / CodeQL

Generic catch clause

Generic catch clause.
@kbeaugrand kbeaugrand added the aws label Jun 19, 2023
@kbeaugrand kbeaugrand added this to the The Portal Keepers - S57 milestone Jun 19, 2023
@kbeaugrand kbeaugrand force-pushed the canditate/merge-from-main-vnext branch from 1a70685 to c335145 Compare June 20, 2023 05:16
@kbeaugrand
Copy link
Member Author

As discussed internally, it seems that PR #1964 seems to be missing since in this PR.
I'll update the branch to have it again

@kbeaugrand
Copy link
Member Author

@hocinehacherouf I cherry picked the PR missing to this branch. Please let me know if something other is missing

@kbeaugrand kbeaugrand added the breaking-change The code change introduce some breaking change to notice label Jun 20, 2023
@kbeaugrand kbeaugrand force-pushed the canditate/merge-from-main-vnext branch from fa5b941 to 6377e28 Compare June 22, 2023 05:37
Copy link

@github-advanced-security github-advanced-security bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CodeQL found more than 10 potential problems in the proposed changes. Check the Files changed tab for more details.

@kbeaugrand kbeaugrand force-pushed the canditate/merge-from-main-vnext branch from 6377e28 to e43a77c Compare June 23, 2023 05:39
delager and others added 14 commits June 23, 2023 09:39
- Add "CloudProvider" in configuration
- Add AmazonIoTClient and AmazonIotDataClient configurations when CloudProvider is AWS

* Update src/AzureIoTHub.Portal.Server/Startup.cs

Add OrdinalIgnoreCase string comparison for CloudProvider

Co-authored-by: Kevin BEAUGRAND <9513635+kbeaugrand@users.noreply.github.com>

* Update src/AzureIoTHub.Portal.Server/Startup.cs

- Local variables camelCased
- Keep asynchronous
- Let the DI container build instances when it is called

Co-authored-by: Kevin BEAUGRAND <9513635+kbeaugrand@users.noreply.github.com>

* #1793 update src\AzureIoTHub.Portal.Server\Startup.cs missing reference

- Missing reference

* #1793 Add CloudProvider for E2E tests

Add CloudProvider config for E2E tests
Add Azure in the current test pipeline (a new AWS test pipeline will be created in the next feature)

---------

Co-authored-by: Kevin BEAUGRAND <9513635+kbeaugrand@users.noreply.github.com>
…ider (#1972)

* #1925 & #1928 Specific config handler and throw error if no CloudProvider

- Add AWS & Azure specific config handlers
- Add AWSS3Storage configuration
- Throw exception if CloudProvider undefined or unknown

* #1925

- Missing TU for AWSS3StorageConnectionString in DevConfig

* #1925 Null reference

- Add ! for null reference possible
* #1929 Configuration logic by provider

- Add CloudProvider dependencies to the Server Startup
- Add CloudProvider dependencies to the Infrastructure Startup
- Azure first in the switch to unify

* #1929 Healthcheck config

- Move common healthcheck configuration

* #1929 Move AWS Client

- Move AWS client singletons into the infrastructure Layer
* #1942 Add CI for CloudFormation scripts

- Add new CI for AWS CloudFormation scripts
- Add label 'cfn-templates' on PR
- Add syntax checking of cfn scripts using scottbrenner/cfn-lint-action

* #1942 Remove schedule

No schedule needed for this workflow

* #1942 Add main branche

Co-authored-by: Kevin BEAUGRAND <9513635+kbeaugrand@users.noreply.github.com>

---------

Co-authored-by: Kevin BEAUGRAND <9513635+kbeaugrand@users.noreply.github.com>
With path option not possible with scottbrenner/cfn-lint-action@v2
Using with command instead
* #1943 Add AWS Deployment

- Add AWS Deployment
- Add AWS information in the different sections

* #1943 Update README.md

* #1943 CloudProvider choice

Co-authored-by: Kevin BEAUGRAND <9513635+kbeaugrand@users.noreply.github.com>

* #1943 Quick Start update

Co-authored-by: Kevin BEAUGRAND <9513635+kbeaugrand@users.noreply.github.com>

* #1943 Configure environment

Co-authored-by: Kevin BEAUGRAND <9513635+kbeaugrand@users.noreply.github.com>

* #1943 Start using

Co-authored-by: Kevin BEAUGRAND <9513635+kbeaugrand@users.noreply.github.com>

* #1943 Quick start update

Co-authored-by: Kevin BEAUGRAND <9513635+kbeaugrand@users.noreply.github.com>

---------

Co-authored-by: Kevin BEAUGRAND <9513635+kbeaugrand@users.noreply.github.com>
…ces images (#2032)

* AWS S3 store & expose device images

* s3 storage image + some unit testing

* #1935 code + tests

* #1935 code + tests

* AWS image cache control

* removing usersecret
… Runner (#1987)

* CloudFormation script initialization

* CloudFormation => PostgreSQL database & S3 storage

* CloudFormation(Draft) => PostgreSQL database & AWS S3 storage

* CloudFormation => some changes in AWS S3 storage & PostgreSQL database

* CloudFormation => AWS EC2 template

* CIDR IP adress putted as a param

* App Runner initialized

* CloudFormation => AppRunner service

* CloudFormation => AppRunner service => Add CLoudProvider

* S3 Storage and Database deployment tested

* Deploy pblm => error in assuming instance role

* AWS access key hided

* merge templates in one

* add secret Manager for the RuntimeEnvironmentSecrets

* instanceRoleArn pblm

* instanceRoleArn pblm

* instanceRoleArn pblm

* #1940 Update CloudFormation script

- Image retrieve from private registry
- Add the cloudformation.amazonaws.com service in the role

* #1940 Cloudformation script

- Add instance role
- Update Secret Manager

* #1940 CloudFormation script

- Multiple SecretsManager for environment variables
- PublicAccessBlockConfiguration for s3 Bucket
- Correct ImageIdentifier

* #1940 CloudFormation script

- Unique DBSubnetGroupName

* #1940 CloudFormation script

- Missing AuthenticationConfiguration

* #1940 CloudFormation script

- Give access to all UniqueSolutionPrefix-* secrets manager

* Add ACL public Read to allow publicRead device images

* Policy removed

---------

Co-authored-by: Kevin BEAUGRAND <contact@kbeaugrand.fr>
Co-authored-by: delager <raphael.delage@cgi.com>
Co-authored-by: DELAGE Raphaël <36408929+delager@users.noreply.github.com>
* create Thing Type table

* PostgreSQL DB migration

* Model + Repo + Service Think Type

* Remove Gneric Thing Type

* Test API Done

* Testing ThingType Service (DONE)

* Create Thing Type Controller tested

* some changes in the Cloud Provider

* Add Thing Type Button OK

* Adapt CreateDeviceModelTests with Azure cloud Provider

* #1802 DONE without adding Migration Files

* #1802 DONE with Migration Files Added

* #1802 Correct COnflict Problem

* #1802 PG Migration new class

* create Thing Type table

* PostgreSQL DB migration

* Model + Repo + Service Think Type

* Remove Gneric Thing Type

* Test API Done

* Testing ThingType Service (DONE)

* Create Thing Type Controller tested

* some changes in the Cloud Provider

* Add Thing Type Button OK

* Adapt CreateDeviceModelTests with Azure cloud Provider

* #1802 DONE without adding Migration Files

* correcting Migration Pblm

* correcting conflicts & bugs in Migration files

* Update AWSImageManager

* Fix formatting

* Fix Build Problems

* Fixes on the PR

* change tag dictionary to tag DTO

* Add Copyright CGI

* remove unsused variables

* Using AWS ThingTypeID for my DB

* handle image in Thing type creation

* handle Thing Type image

* uploading avatar in Thing Type (DONE)

* adding test for InsertAndGetIdAsync function

* adding some tests for test cov
ssgueye2 and others added 26 commits June 23, 2023 09:40
* Hide certificates when getting the magic ommand

* fix edge model update

* fix failed test

---------

Co-authored-by: Kevin BEAUGRAND <contact@kbeaugrand.fr>
* #2168 Refactoring Device & EdgeDevice sync job for AWS

- 1 job Sync things job => update device & edgedevice based on thingtype (iotEdge or not)
- Remove Sync GreenGrass Devices
- Update the job thingtype for isEdge method

* #2168 Add TU for isEdge & ConnectionState

* #2168 Fix TU
- Add missing mappings
- Fix AWSEdgeDeviceService
- Remove TableClientFactory
- Add missing service for Azure
* Hide certificates when getting the magic ommand

* Enable Devices and EdgeDevice in the platform

---------

Co-authored-by: Kevin BEAUGRAND <contact@kbeaugrand.fr>
Co-authored-by: Kevin BEAUGRAND <9513635+kbeaugrand@users.noreply.github.com>
* #2110 Refactor AWS External Device

- Remove AWS Exterrnal Device
- Calls to AWS APIs in the respective services
- Add methods to mutualize in IExternalDeviceService

* #2110 Add missing TU

* #2110 Add AWSExternalDeviceService TU
* Hide certificates when getting the magic ommand

* can remove devices that are not in AWS

* Fix tests => in next commit

* Fix failed tests

---------

Co-authored-by: Kevin BEAUGRAND <contact@kbeaugrand.fr>
* Hide certificates when getting the magic ommand

* Deleting a deployment will deprecate its thing type

---------

Co-authored-by: Kevin BEAUGRAND <contact@kbeaugrand.fr>
* Hide certificates when getting the magic ommand

* Remove AWS Thing type that was deparecated for 5mn ago

* refacto

---------

Co-authored-by: Kevin BEAUGRAND <contact@kbeaugrand.fr>
* Improve AWS exceptions (1)

* improve AWS exceptions (2)

* delete using

* Improve AWS exceptions (3)

* delete unused using instructions

* fix some code scanning warning

* fixed

* Failed tests to fix

* Improve AWS Exception

* Fix some warnings

* Update AWSEdgeDevicesService.cs

* DeleteDynamicGroups when deleting thing type
* merge create and edit device screen

* fix unit tests

* update unit tests

* fix solution project

* update unit tests

* update unit tests

* update unit tests

* update unit tests

* update unit tests

* update unit tests

---------

Co-authored-by: GuillaumeM <mourlonguillaume@gmail.com>
@kbeaugrand kbeaugrand force-pushed the canditate/merge-from-main-vnext branch from e43a77c to d15718a Compare June 23, 2023 07:49
@kbeaugrand kbeaugrand merged commit 928908b into main Jun 23, 2023
@kbeaugrand kbeaugrand deleted the canditate/merge-from-main-vnext branch June 23, 2023 09:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
aws breaking-change The code change introduce some breaking change to notice
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants