Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update all Azure APIs to their latest public version #559

Merged
merged 15 commits into from
Feb 7, 2024

Conversation

kmgalanakis
Copy link
Contributor

@kmgalanakis kmgalanakis commented Aug 10, 2023

Description of the Change

In this MR, I'm updating the Azure APIs version. More specifically I'm updating the Azure AI Vision Image Analyze endpoint version and Thumbnail Generate version to v3.2. I'm also handling the way the API error responses are handed since those changed in v3.2

Closes #553

How to test the Change

Changelog Entry

Changed - Updated the Azure AI Vision Image Analyze endpoint version and Thumbnail Generate version to v3.2. Note we recommend lowering the threshold values for the Descriptive Text Generator feature to 55% to ensure the best results.

Credits

Props @kmgalanakis

Checklist:

  • I agree to follow this project's Code of Conduct.
  • I have updated the documentation accordingly.
  • I have added tests to cover my change.
  • All new and existing tests pass.

@dkotter
Copy link
Collaborator

dkotter commented Aug 10, 2023

I tested this a bit when I put the issue together and one thing I noticed that we'll need to address in this PR is that the error's returned from the API have changed from v3.0 to v3.2. Can compare the v3.0 error responses to the v3.2 error responses to see that.

If helpful, here's the code I had added locally to handle this:

if ( 200 !== wp_remote_retrieve_response_code( $request ) ) {
    if ( isset( $body->error ) ) {
         $rtn = new WP_Error( $body->error->code ?? 'error', $body->error->message ?? esc_html__( 'An error occurred.', 'classifai' ), $body );
     } elseif ( isset( $body->message ) ) {
         $rtn = new WP_Error( $body->error ?? 'error', $body->message, $body );
     } else {
         $rtn = new WP_Error( 'error', esc_html__( 'An error occurred.', 'classifai' ), $body );
     }
} else {
    $rtn = $body;
}

@kmgalanakis
Copy link
Contributor Author

I tested this a bit when I put the issue together and one thing I noticed that we'll need to address in this PR is that the error's returned from the API have changed from v3.0 to v3.2. Can compare the v3.0 error responses to the v3.2 error responses to see that.

If helpful, here's the code I had added locally to handle this:

if ( 200 !== wp_remote_retrieve_response_code( $request ) ) {
    if ( isset( $body->error ) ) {
         $rtn = new WP_Error( $body->error->code ?? 'error', $body->error->message ?? esc_html__( 'An error occurred.', 'classifai' ), $body );
     } elseif ( isset( $body->message ) ) {
         $rtn = new WP_Error( $body->error ?? 'error', $body->message, $body );
     } else {
         $rtn = new WP_Error( 'error', esc_html__( 'An error occurred.', 'classifai' ), $body );
     }
} else {
    $rtn = $body;
}

Your code was mostly fine @dkotter. In the code for the second if condition, I just had to change $body->error to $body->code. For Response 415 the error format remained

{
    "code": "BadArgument",
    "message": "Invalid Media Type"
}

Thank you

@jeffpaul jeffpaul added this to the 2.3.0 milestone Aug 14, 2023
@dkotter dkotter modified the milestones: 2.3.0, 2.4.0 Aug 17, 2023
@dkotter
Copy link
Collaborator

dkotter commented Aug 17, 2023

Overall this looks good to me but I do think we need to figure out how to handle the lower confidence scores, as mentioned here. That will need to be resolved as part of this PR before we can merge this in.

@jeffpaul
Copy link
Member

jeffpaul commented Sep 8, 2023

@dkotter assuming the TLDR from @kmgalanakis is that with the v3.2 API in order to see similar results the threshold needs to drop from 70% to 50/55%, then I think the suggestion from @kmgalanakis to display a dismissable admin notice when we see a threshold still at the default 70% and IBM Watson NLU active that we recommend dropping the value(s) to 55%. So something like:

The latest version of ClassifAI comes with an update to the Language Processing option with IBM Watson that provides best results if you change the default threshold from 70% to 55%. Click here to adjust those settings.

If the IBM Watson NLU feature is not activated and the thresholds are at 70%, then let's change those defaults to 55%. Otherwise new installs will receive the new default of 55%.

@jeffpaul
Copy link
Member

jeffpaul commented Oct 9, 2023

@dkotter any thoughts on the potential handling as noted in my comment above?

@dkotter
Copy link
Collaborator

dkotter commented Oct 12, 2023

@dkotter assuming the TLDR from @kmgalanakis is that with the v3.2 API in order to see similar results the threshold needs to drop from 70% to 50/55%, then I think the suggestion from @kmgalanakis to display a dismissable admin notice when we see a threshold still at the default 70% and IBM Watson NLU active that we recommend dropping the value(s) to 55%. So something like:

The latest version of ClassifAI comes with an update to the Language Processing option with IBM Watson that provides best results if you change the default threshold from 70% to 55%. Click here to adjust those settings.

If the IBM Watson NLU feature is not activated and the thresholds are at 70%, then let's change those defaults to 55%. Otherwise new installs will receive the new default of 55%.

I think this all sounds fine. We can call this out in the changelog and in the release as well as add an admin notice to hopefully make this clear to all existing users.

@dkotter dkotter modified the milestones: 2.4.0, 2.5.0 Nov 7, 2023
@sksaju
Copy link
Member

sksaju commented Nov 22, 2023

@dkotter assuming the TLDR from @kmgalanakis is that with the v3.2 API in order to see similar results the threshold needs to drop from 70% to 50/55%, then I think the suggestion from @kmgalanakis to display a dismissable admin notice when we see a threshold still at the default 70% and IBM Watson NLU active that we recommend dropping the value(s) to 55%. So something like:

The latest version of ClassifAI comes with an update to the Language Processing option with IBM Watson that provides best results if you change the default threshold from 70% to 55%. Click here to adjust those settings.

If the IBM Watson NLU feature is not activated and the thresholds are at 70%, then let's change those defaults to 55%. Otherwise new installs will receive the new default of 55%.

I think this all sounds fine. We can call this out in the changelog and in the release as well as add an admin notice to hopefully make this clear to all existing users.

@jeffpaul @dkotter I've updated the default thresholds value and added admin notice which will only run when the IBM Watson is active/configured. please check it and let me know your feedback. Thank you

@sksaju sksaju marked this pull request as ready for review November 22, 2023 21:57
@sksaju sksaju requested review from jeffpaul and a team as code owners November 22, 2023 21:57
@jeffpaul jeffpaul mentioned this pull request Dec 6, 2023
22 tasks
includes/Classifai/Providers/Watson/NLU.php Outdated Show resolved Hide resolved
includes/Classifai/Providers/Watson/NLU.php Outdated Show resolved Hide resolved
@jeffpaul jeffpaul modified the milestones: 2.5.0, 2.6.0 Dec 12, 2023
@jeffpaul jeffpaul removed their request for review January 16, 2024 15:34
…e're changing here. Move admin notice into our Notifications class
…. This sends an ajax request that we listen for and store some data in user meta, which we then use to not show the notice again
@dkotter dkotter modified the milestones: 3.1.0, 3.0.0 Feb 1, 2024
@dkotter dkotter changed the title Feature/update azure apis versions Update all Azure APIs to their latest public version Feb 1, 2024
@dkotter dkotter requested review from dkotter, a team and Sidsector9 and removed request for dkotter and a team February 1, 2024 19:34
@dkotter
Copy link
Collaborator

dkotter commented Feb 1, 2024

I've gotten this PR ready for review. I think it would be a great addition to the v3 release as the v3.2 version of the AI Vision API brings support for quite a few new languages which I know has come up as a request.

I know one of the things we were concerned with was lower confidence scores. In testing, I do see that for Descriptive Text. As an example, I tested on v3.0 with an image of a dog and got a response of a close up of a dog with 96% confidence. On v3.2, I got a dog with its mouth open with 57% confidence. So I think the results are better (or at least just as good) but the confidence scores have decreased. As such, I've changed our default on that from 75% to 55% and we output an admin notice that lets people know they should decrease that threshold.

But for Image Tagging, I'm not seeing a decrease in the confidence scores. Results seem better but the actual scores seem to be the same, so I've left that threshold alone.

@dkotter dkotter requested review from iamdharmesh and removed request for Sidsector9 February 6, 2024 20:26
Copy link
Member

@iamdharmesh iamdharmesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code LGTM and PR tests well. Thanks all for the great work here.

@dkotter dkotter merged commit 9f4ee48 into develop Feb 7, 2024
12 checks passed
@dkotter dkotter deleted the feature/update-azure-apis-versions branch February 7, 2024 14:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update the Azure APIs to their latest versions
5 participants