Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates to the OpenAI Embeddings Provider #758

Merged
merged 23 commits into from
May 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
fa6144e
Switch to the new small embeddings model; keep our embedding data to …
dkotter Feb 19, 2024
bc7b158
Ensure our vector arrays are the same length before running calculati…
dkotter Feb 19, 2024
9234d59
If we trim the embedding data, make sure we also normalize it
dkotter Feb 19, 2024
794017c
Remove code that isn't needed
dkotter Feb 19, 2024
d4c3f84
Update docblock
dkotter Feb 21, 2024
2df624c
Merge branch 'develop' into feature/update-embedding-model
dkotter Mar 26, 2024
5ad469f
Merge branch 'develop' into feature/update-embedding-model
dkotter Apr 9, 2024
cc72cbf
Add helper methods to get the API URL, model name and max tokens, pas…
dkotter Apr 9, 2024
729e2cc
Add a helper method to get the number of dimensions each embedding sh…
dkotter Apr 9, 2024
59d95da
Remove lines of code that aren't used anymore
dkotter Apr 9, 2024
b38346b
Don't remove abbreviations when normalizing content
dkotter Apr 9, 2024
9f11d56
Add a generic generate_embedding method and use that when generating …
dkotter Apr 9, 2024
3fa96e1
Add a helper method to get the max number of terms and add a filter a…
dkotter Apr 10, 2024
cc5d800
Iterate over all term chunks when run similarity calculations
dkotter Apr 10, 2024
5a95615
Ensure our set_terms method properly accounts for the new embedding a…
dkotter Apr 10, 2024
c53904f
Rename our similarity function to be more clear. Fix some undefined e…
dkotter Apr 10, 2024
8a00548
Remove code that isn't functional and doesn't appear to ever have bee…
dkotter Apr 10, 2024
b5fe959
Clean up the Notifications class a bit
dkotter Apr 10, 2024
2f9a023
Add a notice to trigger regeneration of embeddings. Only show our not…
dkotter Apr 10, 2024
03531a7
Show success message once embeddings are regenerated. Clean up code a…
dkotter Apr 11, 2024
4de6136
Remove test that we no longer need as we aren't stripping abbreviatio…
dkotter Apr 16, 2024
7a31aa2
Merge branch 'develop' into feature/update-embedding-model
dkotter May 8, 2024
b4e85bd
Make sure we force update all terms
dkotter May 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
258 changes: 168 additions & 90 deletions includes/Classifai/Admin/Notifications.php
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
namespace Classifai\Admin;

use Classifai\Features\DescriptiveTextGenerator;
use Classifai\Features\Classification;

class Notifications {

Expand All @@ -24,21 +25,33 @@ public function can_register(): bool {
* Register the actions needed.
*/
public function register() {
add_action( 'classifai_activation_hook', [ $this, 'add_activation_notice' ] );
add_action( 'admin_notices', [ $this, 'maybe_render_notices' ], 0 );
add_action( 'admin_notices', [ $this, 'thresholds_update_notice' ] );
add_action( 'admin_notices', [ $this, 'v3_migration_completed_notice' ] );
add_action( 'admin_enqueue_scripts', [ $this, 'add_dismiss_script' ] );
add_action( 'wp_ajax_classifai_dismiss_notice', [ $this, 'ajax_maybe_dismiss_notice' ] );
}

/**
* Respond to the activation hook.
* Render any needed admin notices.
*/
public function maybe_render_notices() {
// Only show these notices to admins.
if ( ! current_user_can( 'manage_options' ) ) {
return;
}

$this->render_registration_notice();
$this->render_activation_notice();
$this->thresholds_update_notice();
$this->v3_migration_completed_notice();
$this->render_embeddings_notice();
}

/**
* Render a registration notice, if needed.
*/
public function render_registration_notice() {
$registration_settings = get_option( 'classifai_settings' );
// phpcs:ignore WordPress.Security.NonceVerification.Recommended
$page = isset( $_GET['page'] ) ? sanitize_text_field( wp_unslash( $_GET['page'] ) ) : '';
$page = isset( $_GET['page'] ) ? sanitize_text_field( wp_unslash( $_GET['page'] ) ) : ''; // phpcs:ignore WordPress.Security.NonceVerification.Recommended

if (
'classifai' === $page &&
Expand All @@ -53,103 +66,44 @@ public function maybe_render_notices() {
</div>
<?php
}

$needs_setup = get_transient( 'classifai_activation_notice' );
if ( $needs_setup ) {
$onboarding = new Onboarding();
if ( $onboarding->is_onboarding_completed() ) {
delete_transient( 'classifai_activation_notice' );
return;
}

// Prevent showing the default WordPress "Plugin Activated" notice.
unset( $_GET['activate'] ); // phpcs:ignore WordPress.Security.NonceVerification
?>
<div data-notice="plugin-activation" class="notice notice-success is-dismissible">
<div id="classifai-activation-notice">
<div class="classifai-logo">
<img src="<?php echo esc_url( CLASSIFAI_PLUGIN_URL . 'assets/img/classifai.png' ); ?>" alt="<?php esc_attr_e( 'ClassifAI', 'classifai' ); ?>" />
</div>
<h3 class="classifai-activation-message">
<?php esc_html_e( 'Congratulations, the ClassifAI plugin is now activated.', 'classifai' ); ?>
</h3>
<a class="classifai-button" href="<?php echo esc_url( admin_url( 'admin.php?page=classifai_setup' ) ); ?>">
<?php esc_html_e( 'Start setup', 'classifai' ); ?>
</a>
</div>
</div>
<?php
delete_transient( 'classifai_activation_notice' );
}
}

/**
* Print out a script to dismiss a notice.
*
* This allows us to save that a user has dismissed a notice.
*
* Influenced by https://github.com/WPTT/admin-notices/blob/af52f563398b42cff82d38eefa55c8121d698ebe/src/Dismiss.php#L77
* Render an activation notice, if needed.
*/
public function add_dismiss_script() {
$nonce = wp_create_nonce( 'classifai_dismissible_notice' );
$admin_ajax_url = esc_url( admin_url( 'admin-ajax.php' ) );

$script = <<<EOD
jQuery( function() {
const dismissBtns = document.querySelectorAll( '.classifai-dismissible-notice' );

if ( ! dismissBtns.length ) {
return;
}

// Add an event listener to the dismiss buttons.
dismissBtns.forEach( function( dismissBtn ) {
dismissBtn.addEventListener( 'click', function( event ) {
const id = dismissBtn.getAttribute( 'data-notice' );

if ( ! id ) {
return;
}

const httpRequest = new XMLHttpRequest();
let postData = '';

// Build the data to send in our request.
// Data has to be formatted as a string here.
postData += 'notice_id=' + id;
postData += '&action=classifai_dismiss_notice';
postData += '&nonce=$nonce';

httpRequest.open( 'POST', '$admin_ajax_url' );
httpRequest.setRequestHeader( 'Content-Type', 'application/x-www-form-urlencoded' )
httpRequest.send( postData );
});
});
});
EOD;

wp_add_inline_script( 'common', $script, 'after' );
}
public function render_activation_notice() {
$needs_setup = get_transient( 'classifai_activation_notice' );

/**
* Verify ajax request and dismiss the notice.
*
* Influenced by https://github.com/WPTT/admin-notices/blob/af52f563398b42cff82d38eefa55c8121d698ebe/src/Dismiss.php#L133
*/
public function ajax_maybe_dismiss_notice() {
if ( ! isset( $_POST['action'] ) || 'classifai_dismiss_notice' !== $_POST['action'] ) {
if ( ! $needs_setup ) {
return;
}

if ( ! isset( $_POST['notice_id'] ) ) {
$onboarding = new Onboarding();
if ( $onboarding->is_onboarding_completed() ) {
delete_transient( 'classifai_activation_notice' );
return;
}

check_ajax_referer( 'classifai_dismissible_notice', 'nonce' );
// Prevent showing the default WordPress "Plugin Activated" notice.
unset( $_GET['activate'] ); // phpcs:ignore WordPress.Security.NonceVerification
?>

$notice_id = sanitize_text_field( wp_unslash( $_POST['notice_id'] ) );
<div data-notice="plugin-activation" class="notice notice-success is-dismissible">
<div id="classifai-activation-notice">
<div class="classifai-logo">
<img src="<?php echo esc_url( CLASSIFAI_PLUGIN_URL . 'assets/img/classifai.png' ); ?>" alt="<?php esc_attr_e( 'ClassifAI', 'classifai' ); ?>" />
</div>
<h3 class="classifai-activation-message">
<?php esc_html_e( 'Congratulations, the ClassifAI plugin is now activated.', 'classifai' ); ?>
</h3>
<a class="classifai-button" href="<?php echo esc_url( admin_url( 'admin.php?page=classifai_setup' ) ); ?>">
<?php esc_html_e( 'Start setup', 'classifai' ); ?>
</a>
</div>
</div>

update_user_meta( get_current_user_id(), "classifai_dismissed_{$notice_id}", true );
<?php
delete_transient( 'classifai_activation_notice' );
}

/**
Expand Down Expand Up @@ -235,6 +189,7 @@ public function v3_migration_completed_notice() {
return;
}
?>

<div class="notice notice-info is-dismissible classifai-dismissible-notice classifai-migation-notice" data-notice="<?php echo esc_attr( $key ); ?>">
<p>
<?php
Expand All @@ -251,6 +206,129 @@ public function v3_migration_completed_notice() {
?>
</p>
</div>

<?php
}

/**
* Render a notice about needing to regenerate embeddings.
*/
public function render_embeddings_notice() {
// Bail if no need to show the notice.
if ( get_option( 'classifai_hide_embeddings_notice', false ) ) {
return;
}

// Ensure the feature exists.
if ( ! class_exists( 'Classifai\Features\Classification' ) ) {
return;
}

$feature_instance = new Classification();

// Don't show the notice if the feature is not enabled.
if ( ! $feature_instance->is_feature_enabled() ) {
return;
}

// Don't show the notice if the provider is not OpenAI Embeddings.
$provider = $feature_instance->get_settings( 'provider' );
if ( 'openai_embeddings' !== $provider ) {
return;
}

$key = 'embedding_regen_completed';

// Don't show the notice if the user has already dismissed it.
if ( get_user_meta( get_current_user_id(), "classifai_dismissed_{$key}", true ) ) {
return;
}
?>

<div class="notice notice-warning is-dismissible classifai-dismissible-notice" data-notice="<?php echo esc_attr( $key ); ?>">
<p>
<?php
echo wp_kses_post(
sprintf(
// translators: %1$s: Feature specific message; %2$s: URL to Feature settings.
__( 'ClassifAI has updated to the <code>text-embedding-3-small</code> embeddings model. <br>This requires regenerating any stored embeddings for functionality to work properly. <br><a href="%1$s">Click here to do that</a>, noting this will make multiple API requests to OpenAI.', 'classifai' ),
wp_nonce_url( admin_url( 'tools.php?page=classifai&tab=language_processing&feature=feature_classification' ), 'regen_embeddings', 'embeddings_nonce' )
)
);
?>
</p>
</div>

<?php
}

/**
* Print out a script to dismiss a notice.
*
* This allows us to save that a user has dismissed a notice.
*
* Influenced by https://github.com/WPTT/admin-notices/blob/af52f563398b42cff82d38eefa55c8121d698ebe/src/Dismiss.php#L77
*/
public function add_dismiss_script() {
$nonce = wp_create_nonce( 'classifai_dismissible_notice' );
$admin_ajax_url = esc_url( admin_url( 'admin-ajax.php' ) );

$script = <<<EOD
jQuery( function() {
const dismissNotices = document.querySelectorAll( '.classifai-dismissible-notice' );

if ( ! dismissNotices.length ) {
return;
}

// Add an event listener to the dismiss buttons.
dismissNotices.forEach( function( dismissNotice ) {
let dismissBtn = dismissNotice.querySelector( '.notice-dismiss' );
dismissBtn.addEventListener( 'click', function( event ) {
const id = dismissNotice.getAttribute( 'data-notice' );

if ( ! id ) {
return;
}

const httpRequest = new XMLHttpRequest();
let postData = '';

// Build the data to send in our request.
// Data has to be formatted as a string here.
postData += 'notice_id=' + id;
postData += '&action=classifai_dismiss_notice';
postData += '&nonce=$nonce';

httpRequest.open( 'POST', '$admin_ajax_url' );
httpRequest.setRequestHeader( 'Content-Type', 'application/x-www-form-urlencoded' )
httpRequest.send( postData );
});
});
});
EOD;

wp_add_inline_script( 'common', $script, 'after' );
}

/**
* Verify ajax request and dismiss the notice.
*
* Influenced by https://github.com/WPTT/admin-notices/blob/af52f563398b42cff82d38eefa55c8121d698ebe/src/Dismiss.php#L133
*/
public function ajax_maybe_dismiss_notice() {
if ( ! isset( $_POST['action'] ) || 'classifai_dismiss_notice' !== $_POST['action'] ) {
return;
}

if ( ! isset( $_POST['notice_id'] ) ) {
return;
}

check_ajax_referer( 'classifai_dismissible_notice', 'nonce' );

$notice_id = sanitize_text_field( wp_unslash( $_POST['notice_id'] ) );

update_user_meta( get_current_user_id(), "classifai_dismissed_{$notice_id}", true );
}
}
1 change: 0 additions & 1 deletion includes/Classifai/Command/ClassifaiCommand.php
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@
use Classifai\Providers\Watson\APIRequest;
use Classifai\Providers\Watson\Classifier;
use Classifai\Normalizer;
use Classifai\Providers\Watson\PostClassifier;
use Classifai\Providers\OpenAI\Embeddings;

use function Classifai\Providers\Watson\get_username;
Expand Down
3 changes: 1 addition & 2 deletions includes/Classifai/Helpers.php
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
use Classifai\Features\Classification;
use Classifai\Providers\Provider;
use Classifai\Admin\UserProfile;
use Classifai\Providers\Watson\NLU;
use Classifai\Providers\OpenAI\Embeddings;
use Classifai\Services\Service;
use Classifai\Services\ServicesManager;
Expand Down Expand Up @@ -589,7 +588,7 @@ function get_classification_feature_enabled( string $classify_by ): bool {
$settings = ( new Classification() )->get_settings();

return filter_var(
$settings[ $classify_by ],
isset( $settings[ $classify_by ] ) ?? false,
FILTER_VALIDATE_BOOLEAN
);
}
Expand Down
3 changes: 0 additions & 3 deletions includes/Classifai/Normalizer.php
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,6 @@ public function normalize_content( $post_content, $post_title = '', $post_id = f
/* Strip HTML entities */
$post_content = preg_replace( '/&#?[a-z0-9]{2,8};/i', '', $post_content );

/* Strip abbreviations */
$post_content = preg_replace( '/[A-Z][A-Z]+/', '', $post_content );

/* Replace HTML linebreaks with newlines */
$post_content = preg_replace( '#<br\s?/?>#', "\n\n", $post_content );

Expand Down
Loading
Loading