Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update migrate:osci-publication to use migration file from osci-migration #433

Merged
merged 36 commits into from
Aug 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
6f16729
Rename instances of Digital Publications Sections to Articles [PUB-191]
web-dev-trev May 29, 2024
99c349a
Fix interactive features slug bug [WEB-2886]
web-dev-trev Jun 26, 2024
d057c6e
Adapt migrate:osci-publication to use sqlite + pure HTML blocks
cbutcosk Jun 12, 2024
f2641e8
Use blocks to fill article contents
cbutcosk Jun 13, 2024
6890ef0
Populate block position
cbutcosk Jun 13, 2024
fd2481b
Fix figure patch and add figure caption
cbutcosk Jun 13, 2024
fcdc49f
Fix block order
cbutcosk Jun 13, 2024
bc19551
Refine f/n migrate: use 'text' for DigitalPubArticle type
cbutcosk Jun 14, 2024
c0b98a3
Refactor `migrate:osci-publication` command to access sqlite DB using…
nikhiltri Jun 21, 2024
a84105e
Add default nested set values for imported articles [PUB-204]
nikhiltri Jun 25, 2024
22ad400
Clean up linting errs
cbutcosk Jun 26, 2024
e4b7fb6
Media operations from test.jpeg + figure-type-aware block serializations
cbutcosk Jun 27, 2024
ffbea99
Remove over-aggressive json encodings
cbutcosk Jun 27, 2024
9ea4ff5
Fix lint errs
cbutcosk Jun 27, 2024
253d27a
Correct block types + caption usage for 360, rti_viewer
cbutcosk Jun 27, 2024
3993576
Break out layered image as a block type
cbutcosk Jun 27, 2024
e8fe6d6
Use fallback_url for IIP stub blocks, apply nikhil's fix
cbutcosk Jul 2, 2024
3f6abd0
Join layers to fig refs, refine layer, image, and video block handling
cbutcosk Jul 12, 2024
71229c4
Clean up uploaded media filenames, media w/h from bytes not data
cbutcosk Jul 12, 2024
011b8a3
Change block sizes to 'small'
nikhiltri Jul 12, 2024
8749cca
Clean up linting, switch and order fig layer annotations on fig options
cbutcosk Jul 12, 2024
3493792
Use first image block's image as the hero image on a migrated article…
nikhiltri Jul 17, 2024
75e8c48
Refine iip, svg overlay, layers handling, image url syntaxes
cbutcosk Jul 19, 2024
d7ca1b8
Fix lint errs
cbutcosk Jul 19, 2024
453daa1
Soft-remove IIP protocol handling
cbutcosk Jul 19, 2024
0862daa
Round svg image h/w, media retry / fail on errors
cbutcosk Jul 19, 2024
929b59b
Add `migrate:osci-media` for iip assets, use assets in digipubarticles
cbutcosk Jul 24, 2024
15c4b0a
Use config value for s3 urls
cbutcosk Jul 24, 2024
0b98bea
Break up figure block configuration info config functions
cbutcosk Jul 24, 2024
d60db57
Improve console output while migrating OSCI publications [PUB-217]
nikhiltri Jul 22, 2024
efc40b3
Add migrate all publications commands [PUB-225]
nikhiltri Jul 25, 2024
8868b9c
Lint code [PUB-225]
nikhiltri Jul 25, 2024
17130f5
Use http facade for URL fetching, refine fetch error handling
cbutcosk Jul 25, 2024
2b497f7
Remove dead-code http context + fix lint error
cbutcosk Jul 25, 2024
daa53bc
Housekeeping -- code quality fixes, comments, correct media-one arg sig
cbutcosk Jul 25, 2024
38f7371
More lints
cbutcosk Jul 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions app/Console/Commands/MigrateOSCIPublication.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
<?php

namespace App\Console\Commands;

use Illuminate\Console\Command;

class MigrateOSCIPublication extends Command
{
/**
* The name and signature of the console command.
*
* @var string
*/
protected $signature = 'migrate:osci-publication';

/**
* The console command description.
*
* @var string
*/
protected $description = 'Migrate all OSCI publications from a migration file to website DigitalPublications';

protected $publications = [
'albright',
'americansilver',
'caillebotte',
'digitalwhistler',
'ensor',
'gauguin',
'manet',
'matisse',
'modernseries',
'modernseries2',
'monet',
'pissarro',
'renoir',
'romanart',
'whistler_linkedvisions',
'whistlerart',
];

/**
* Execute the console command.
*
* @return int
*/
public function handle()
{
foreach ($this->publications as $pub) {
$this->call('migrate:osci-publication-one', ['id' => $pub]);
}
}
}
53 changes: 53 additions & 0 deletions app/Console/Commands/MigrateOSCIPublicationMedia.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
<?php

namespace App\Console\Commands;

use Illuminate\Console\Command;

class MigrateOSCIPublicationMedia extends Command
{
/**
* The name and signature of the console command.
*
* @var string
*/
protected $signature = 'migrate:osci-media';

/**
* The console command description.
*
* @var string
*/
protected $description = 'Migrate all OSCI publications\' IIP and 360º media from OSCI servers to S3';

protected $publications = [
'albright',
'americansilver',
'caillebotte',
'digitalwhistler',
'ensor',
'gauguin',
'manet',
'matisse',
'modernseries',
'modernseries2',
'monet',
'pissarro',
'renoir',
'romanart',
'whistler_linkedvisions',
'whistlerart',
];

/**
* Execute the console command.
*
* @return int
*/
public function handle()
{
foreach ($this->publications as $pub) {
$this->call('migrate:osci-media-one', ['id' => $pub]);
}
}
}
98 changes: 98 additions & 0 deletions app/Console/Commands/MigrateOSCIPublicationMediaOne.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
<?php

namespace App\Console\Commands;

use Imagick;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Storage;

$PTIF_LAYER_MAX = 10000;

class MigrateOSCIPublicationMediaOne extends Command
{
/**
* The name and signature of the console command.
*
* @var string
*/
protected $signature = 'migrate:osci-media-one {id : Publication ID}';

/**
* The console command description.
*
* @var string
*/
protected $description = 'Migrate an OSCI publications\' IIP and 360º media from OSCI servers to S3';

/**
* Execute the console command.
*
* @return int
*/
public function handle()
{
$pubId = $this->argument('id');
$this->comment('Starting ' . $pubId . '...');

# TODO: Query 360s too
$assetsQuery = <<<EOT
SELECT DISTINCT json_extract(layers.value,'$._image_ident') AS image_ident
FROM figure_layers,
json_each(figure_layers.data,'$.layers') AS layers
WHERE json_extract(layers.value,'$._type')='iip'
AND figure_layers.package=:pubId
ORDER BY json_extract(layers.value,'$._height') ASC
EOT;

$assets = DB::connection('osci_migration')->select($assetsQuery, ['pubId' => $pubId]);

foreach ($assets as $asset) {
// TODO: Check the asset type and handle 360 spins differently
$key = trim($asset->image_ident, '/');
$asset_key = preg_replace('/^\/?osci\//', '', $asset->image_ident);
$jpg_key = preg_replace('/\.ptif$/', '.jpg', $key);

if (!Storage::disk('osci_s3')->fileExists($asset_key)) {
$this->info("OSCI_S3_BUCKET did not contain {$asset_key}!");
continue;
}

if (Storage::disk('osci_s3')->fileExists($jpg_key)) {
$this->info("OSCI_S3_BUCKET already contains {$jpg_key}, skipping this asset");
continue;
}

$this->info("Compressing {$asset_key}");

$content = Storage::disk('osci_s3')->get($asset_key);

$image = new Imagick();
$image->readImageBlob($content);

// Iterate zoom layers to find a 10k-pixels-a-side page geometry
while ($image->hasPreviousImage()) {
$size = $image->getImagePage();

if ($size['height'] > 10000 && $size['width'] > 10000) {
$image->nextImage();
break;
}
$image->previousImage();
}

$image->setImageFormat('jpeg');
$compressed = $image->getImageBlob();

$res = Storage::disk('osci_s3')->put($jpg_key, $compressed);

if (!$res) {
$this->info("There was an error uploading the compressed file to {$jpg_key}");
continue;
}

echo "Uploaded {$jpg_key}\n";
}
$this->comment('...' . $pubId . ' done!');
}
}
Loading
Loading