Events to queue new jobs not working

[tecsinn]

Howdy,

I purchased WP2Static and have two little questions:

1. How should the "queue a new job every time a post or page is (saved|deleted)" work? I have both activated, but updating posts or pages will not result in a new line in Job Queue/History, and the static export (we're exporting to S3) also does not update. If I hit Manually Enqueue Jobs Now, I see the jobs enqueued and also the S3 export eventually updates. I'm not new to programming, but new to wordpress, and unsure where to look further to debug this. I also purchased the WP Cron Status plugin to see what cron is actually doing, and it's the same picture—I see my manually enqueued WP2Static jobs being worked on, but nothing triggered by a post/page update. Any hints on what one should see in these two locations would help (e.g. "for manually enqueued jobs you should see X, and for auto jobs triggerd by post updates you should see Y").
2. I was using the old version previously, but switched to the paid new one because of encoding problems in the export. The new version fixed that, but I'm really missing the path rewrite portion of WP2Static to hide the typical wordpress (and theme) directories. I also didn't find this as an extra. Was I just too blind?

Thanks a lot for this plugin, Leon!

[leonstafford]

Hi @tecsinn very sorry for the slow reply!

I think there's a fix in V7 already to sort out the jobs queuing, but the next version also brings a manual "Run a job now" type button, like the old one, to make things easier.

I was hoping for next round of deploys for V7 last Friday/weekend, but delayed myself a bit, updating a few other related projects.

I have some updated code for doing the content modification sitting as an extra add-on for V7, but it's a little way off. Just today, I was updating the V6 codebase around this functionality, which will be continued as a different project, to allow both to advance. I'm hoping to deal with the unicode issues in V6 now that the codebase is a bit more easily testable from today. Simple Cyrillic in page content seems like it should be OK, but filenames are still likely an issue, both for crawling and S3 deployments.

I know it's frustrating, but if you can hang tight a little longer, the next V7 core + addons release should be much improved and may see V6 also sorting out the unicode issue.

As a 3rd contender, I've also updated a fork of SimplyStatic (https://github.com/WP2Static/simplerstatic) to run on PHP 7.3+ and also pave way for easier continued development (they all currently have some unique benefits, so like children, I must love them all :D).

[leonstafford]

^ for what should appear in the Jobs queue, we'd expect to see (if all 4 checkboxes are checked):

deploy
post_process
crawl
detect

If there's already a set of those in waiting status, it won't add any more, it just "squashes" any duplicates together, so you'll only deploy the latest version of the site.

[tecsinn]

No problem about the delay @leonstafford, you're doing a great job here in the issues :) Thanks for the info!

I'll give simplerstatic a go. And here's a question that's OT for this thread but I don't know where to ask: If my site has server side 30x redirects, cant WP2Static/Simplerstatic export that as an *.html with a meta redirect?

P.S.
simplerstatic.com backend is down, cloudflare says 🙈

[john-shaffer]

If my site has server side 30x redirects, cant WP2Static/Simplerstatic export that as an *.html with a meta redirect?

This is probably better handled by the individual addons. The S3 addon, for example, could deploy a 0 byte file with Website-Redirect-Location set. The zip addon could do as you say. The tricky part is detecting the redirects.

[leonstafford]

Yes, per-deployment specialized redirect handlers would be best. It looks like SimplerStatic/SimplyStatic are recording all 3xx's at the moment, I need to check if it stores both locations against the URL for working with. And providing those as lists to the user after a crawl are useful, along with 4xx's to fix (that should now be working in "Static HTML Output" >=6.6.8 (sorry for all the new names!), where I noticed a bug that was detecting the 404's, but not quite getting them into the Logs > 404's file

Having easily copy pastable logs of 3xx/4xx's will help short-term to paste into a Netlify _redirects, CloudFront behaviour/Lambda fn, etc... until a deploy-specific option is available in each add-on

[tecsinn]

@john-shaffer great idea! could you nudge me into a direction where I could look into implementing this?

@leonstafford which version of wp2static do you recommend using at the moment? i'm a bit confused :) our setup: wordpress on k8s, always private. static site crawled and stored on s3. public site delivered via a copy of the s3 bucket where the static export went (so we can have rollbacks)

[leonstafford]

@tecsinn - this repo's (WP2Static) will be the best for your use case, with most options for continuous integration / scripted workflows. @john-shaffer has got some great improvements coming in the next S3 add-on for WP2Static, too, I just need to hustle a bit more this week and get them merged in for us all!

Re redirects, the current place is in the src/Crawler.php:156:
public function crawlURL( string $url ) : ?string { ....

You could probably jump in there, where it's detecting a 404 and do something for any 3xx responses. Ughh, looking at that function now, it's pretty naive and really needs improvement, but I can't promise how soon I'll get to that, so making some modifications yourself may be best if time critical, then if you share back what you come up with, it might get worked back into the codebase, but definitely that area needs stuff like hooks for different status codes, so that other add-ons could be made to easily handle different error codes...

[john-shaffer]

@john-shaffer great idea! could you nudge me into a direction where I could look into implementing this?

I have a working implementation:

    public static function put_redirect( \Aws\S3\S3Client $s3, string $bucket,
                                         string $key, string $redirect_to ) : object {
        $result = $s3->putObject(
            [
                'Bucket' => $bucket,
                'Key' => $key,
                'ACL' => 'public-read',
                'WebsiteRedirectLocation' => $redirect_to
            ]
        );
        return $result;
    }

where the first two arguments are

        $s3 = \WP2StaticS3\Deployer::s3_client();
        $bucket = \WP2StaticS3\Controller::getValue( 's3Bucket' );

The question is, where are you getting a list of redirects from? I don't think that the Crawler will detect every redirect in most cases. It certainly should keep track of any that it finds, but what about other sources like plugins?

IMO all the redirect detection should live in wp2static, probably with a new table to store every redirect it knows about, and possibly allowing manual additions. Then the addons can just read this table and deploy as best they can. I'll try to look at implementing the table if I have time.

[tecsinn]

Thanks you two, I love the energy here 🚀

@john-shaffer That's great! I didn't look at how WP2Static stores the crawl results, but it should be fairly easy to detect from a 3xx status code, right? A table of crawl results with response headers (and body) would be very useful indeed.

@leonstafford I'll have a look, thank you very much. What's the typical update flow in the wordpress world if one wants to use master of a plugin? Somehow via the web interface, or just clone on the server?

[john-shaffer]

@john-shaffer That's great! I didn't look at how WP2Static stores the crawl results, but it should be fairly easy to detect from a 3xx status code, right? A table of crawl results with response headers (and body) would be very useful indeed.

Yes, that would be useful. But most redirects won't be detected by crawling. I consider it a bug if anything on my site references a URL that is a redirect (looking at you, Yoast SEO).

@leonstafford I'll have a look, thank you very much. What's the typical update flow in the wordpress world if one wants to use master of a plugin? Somehow via the web interface, or just clone on the server?

In wp-content/plugins, git clone the repo, than cd into the new plugin dir and run composer i. I'm not sure whether the most recent wp2static is compatible with the addons. I'm running 43944a8 with the latest wp2static-addon-3.

[john-shaffer]

Redirects were implemented in #567.

Leon implemented the "Process queue" button in an earlier release, and #794 should fully address the original issue here.

[john-shaffer]

Should be addressed by 68d6bd0. Will be in 7.2.0.