Import through CLI to user's root without specifying album #1221

anoteng · 2022-02-11T11:20:00Z

Detailed description of the problem [REQUIRED]

Hi, I am not sure if this is a bug, a feature request or a support question. I have a large collection of photos (>35k files) I'd like to import into Lychee. I have successfully imported most of them through the web interface, but some folders are missing. Importing through the web interface takes a lot of hours, and usually something will fail or I have to move/disconnect my laptop before it finishes. Any chrome based browser will after a while reach about 8GB of memory use, and then fail with an out of memory crash. Works fine in Firefox though, with a memory use hovering around 500-800MB.

I would like to do a full import of the whole collection through the CLI, but it seems to fail due to no album with ID 0 present. Is there a way to make the CLI import create albums when needed like the web interface does, and use a users root as the starting point?
Also, is there some way of seeing album IDs in the web interface?

Steps to reproduce the issue

Steps to reproduce the behavior:

Go to the folder where lychee is installed
type: php artisan lychee:sync --skip_duplicates -vvv <image folder>
See error:

Start syncing.
/mnt/nas/lene_bilder/: 0%
/mnt/nas/lene_bilder/Thumbs.db: Unsupported file type
/mnt/nas/lene_bilder/index.php: Unsupported file type
/mnt/nas/lene_bilder/: 50%
/mnt/nas/lene_bilder/: 100%
Illuminate\Database\Eloquent\ModelNotFoundException: No query results for model [App\Models\Album] 0 in /opt/lychee/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Builder.php:444
Stack trace:
#0 /opt/lychee/app/Actions/Album/Create.php(38): Illuminate\Database\Eloquent\Builder->findOrFail()
#1 /opt/lychee/app/Actions/Album/Create.php(20): App\Actions\Album\Create->set_parent()
#2 /opt/lychee/app/Actions/Import/Exec.php(310): App\Actions\Album\Create->create()
#3 /opt/lychee/app/Console/Commands/Sync.php(81): App\Actions\Import\Exec->do()
#4 /opt/lychee/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): App\Console\Commands\Sync->handle()
#5 /opt/lychee/vendor/laravel/framework/src/Illuminate/Container/Util.php(40): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#6 /opt/lychee/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure()
#7 /opt/lychee/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\Container\BoundMethod::callBoundMethod()
#8 /opt/lychee/vendor/laravel/framework/src/Illuminate/Container/Container.php(653): Illuminate\Container\BoundMethod::call()
#9 /opt/lychee/vendor/laravel/framework/src/Illuminate/Console/Command.php(136): Illuminate\Container\Container->call()
#10 /opt/lychee/vendor/symfony/console/Command/Command.php(298): Illuminate\Console\Command->execute()
#11 /opt/lychee/vendor/laravel/framework/src/Illuminate/Console/Command.php(121): Symfony\Component\Console\Command\Command->run()
#12 /opt/lychee/vendor/symfony/console/Application.php(1005): Illuminate\Console\Command->run()
#13 /opt/lychee/vendor/symfony/console/Application.php(299): Symfony\Component\Console\Application->doRunCommand()
#14 /opt/lychee/vendor/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun()
#15 /opt/lychee/vendor/laravel/framework/src/Illuminate/Console/Application.php(94): Symfony\Component\Console\Application->run()
#16 /opt/lychee/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(129): Illuminate\Console\Application->run()
#17 /opt/lychee/artisan(37): Illuminate\Foundation\Console\Kernel->handle()
#18 {main}
Done syncing.

### Output of the diagnostics  [REQUIRED]
*(Settings => Diagnostics or https://example.com/Diagnostics or `php artisan lychee:diagnostics`)*
Diagnostics
-----------
Warning: Dropbox import not working. dropbox_key is empty.
Warning: You may experience problems when uploading a photo of large size. Take a look in the FAQ for details.
Warning: You may experience problems when uploading a photo of large size. Take a look in the FAQ for details.
Warning: Default timezone not properly set; you might experience strange results when importing photos without explicit EXIF timezone

System Information
------------------
Lychee Version (git):            fix-1218 (2ea4623) - Branch is not master, cannot compare.
DB Version:                      4.5.0

composer install:                --no-dev
APP_ENV:                         production
APP_DEBUG:                       true

System:                          Linux
PHP Version:                     8.1
PHP User agent:                  Lychee/4 (https://lycheeorg.github.io/)
Timezone:                        UTC
Max uploaded file size:          2M
Max post size:                   8M
Max execution time:              0
MySQL Version:                   8.0.28-0ubuntu0.20.04.3

Imagick:                         1
Imagick Active:                  1
Imagick Version:                 1690
GD Version:                      2.3.0

Config Information
------------------
version:                         040500
check_for_updates:               1
sorting_Photos_col:              taken_at
sorting_Photos_order:            ASC
sorting_Albums_col:              max_taken_at
sorting_Albums_order:            ASC
imagick:                         1
skip_duplicates:                 1
small_max_width:                 0
small_max_height:                360
medium_max_width:                1920
medium_max_height:               1080
lang:                            nb-no
layout:                          1
image_overlay_type:              desc
default_license:                 none
compression_quality:             90
full_photo:                      1
delete_imported:                 1
Mod_Frame:                       1
Mod_Frame_refresh:               30
thumb_2x:                        1
small_2x:                        1
medium_2x:                       1
landing_page_enable:             0
landing_owner:                   John Smith
landing_title:                   John Smith
landing_subtitle:                Bilder
landing_facebook:                https://www.facebook.com/JohnSmith
landing_flickr:                  https://www.flickr.com/JohnSmith
landing_twitter:                 https://www.twitter.com/JohnSmith
landing_instagram:               https://instagram.com/JohnSmith
landing_youtube:                 https://www.youtube.com/JohnSmith
landing_background:              dist/cat.jpg
site_title:                      Lychee v4
site_copyright_enable:           1
site_copyright_begin:            2004
site_copyright_end:              2022
additional_footer_text:
display_social_in_gallery:       0
public_search:                   0
SL_enable:                       0
SL_for_admin:                    0
public_recent:                   0
recent_age:                      1
public_starred:                  0
downloadable:                    0
photos_wraparound:               1
map_display:                     0
zip64:                           1
map_display_public:              0
map_provider:                    Wikimedia
force_32bit_ids:                 0
map_include_subalbums:           0
update_check_every_days:         3
has_exiftool:                    0
share_button_visible:            0
import_via_symlink:              0
has_ffmpeg:                      1
location_decoding:               0
location_decoding_timeout:       30
location_show:                   1
location_show_public:            0
rss_enable:                      0
rss_recent_days:                 7
rss_max_items:                   100
prefer_available_xmp_metadata:   0
editor_enabled:                  1
lossless_optimization:           0
swipe_tolerance_x:               150
swipe_tolerance_y:               250
local_takestamp_video_formats:   .avi|.mov
log_max_num_line:                1000
unlock_password_photos_with_url_param: 0
nsfw_visible:                    1
nsfw_blur:                       0
nsfw_warning:                    0
nsfw_warning_admin:              0
map_display_direction:           1
album_subtitle_type:             oldstyle
upload_processing_limit:         4
public_photos_hidden:            1
new_photos_notification:         0
legacy_id_redirection:           1

Browser and system

Ubuntu server 20.04 with backported PHP8, photos and storage folders connected through NFS mount.

The text was updated successfully, but these errors were encountered:

ildyria · 2022-02-11T11:23:06Z

Album with ID 0 should be root i.e. no album.

nagmat84 · 2022-02-11T11:49:15Z

As the root album does not explicitly exist on the DB, there is no album with ID 0. The correct value is null. This is probably a regression due the re-factoring. The old code was very sloppy and treated null and 0 as equal most of the time.

In particular, the DB column album_id from photos to albums must properly set to null, otherwise the foreign constraint fails.

I will have a look into that this weekend.

nagmat84 · 2022-02-11T12:04:26Z

@anoteng While the 0 vs. null problem is a bug and definitely something we must solve, I would like to draw your attention to the warning messages of the diagnostic output.

In particular, I recommend to take the warning about the timezone setting seriously and use the waiting time while we fix the bug to accurately set your timezone before you continue to import further photos. Otherwise you might have imported 10th of thousands of photos in the end only to notice that the takedate of the photos is wrong and you have to restart over.

anoteng · 2022-02-11T12:25:16Z

Thanks. Timezone is set to UTC in the config, which corresponds to the HW clock of the server. Should it be set to the default timezone of the photos?
Max post and max filesize is set to 500M in public/.htaccess
Where do php artisan lychee:diagnostics get those vlaues from?

nagmat84 · 2022-02-11T15:37:54Z

Timezone is set to UTC in the config [...]. Should it be set to the default timezone of the photos?

Exactly.

Max post and max filesize is set to 500M in public/.htaccess Where do php artisan lychee:diagnostics get those vlaues from?

Lychee just receives these values from he PHP runtime environment. In principle, .htaccess should be fine. Of course, .htaccess is not used when you invoke PHP from the CLI. However, for CLI the values for max file size and max post size are irrelevant. You should check the diagnostic via the web front-end to see the values which are used by the web server.

kamil4 · 2022-02-11T16:33:10Z

Also, you should switch your Lychee tree back to master (and pull), since the fix for #1218 has been merged in the meantime.

anoteng · 2022-02-11T16:36:20Z

Yeah, I did that after posting 😁 fre. 11. feb. 2022, 17:33 skrev Kamil Iskra ***@***.***>:

…

Also, you should switch your Lychee tree back to master (and pull), since the fix for #1218 <#1218> has been merged in the meantime. — Reply to this email directly, view it on GitHub <#1221 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJCQYRNPER4C2OT6X7RG4DU2U25DANCNFSM5OD6O37A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

anoteng · 2022-02-11T17:45:30Z

Damn you guys work fast! Confirmed fix. fre. 11. feb. 2022, 17:42 skrev Matthias Nagel ***@***.***>:

…

Closed #1221 <#1221> via #1222 <#1222>. — Reply to this email directly, view it on GitHub <#1221 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJCQYX4ELXP7MOM6KL5HRTU2U37NANCNFSM5OD6O37A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

ildyria · 2022-02-11T23:16:59Z

Damn you guys work fast! Confirmed fix. fre. 11. feb. 2022, 17:42 skrev Matthias Nagel @.***>:

Obligatory xkcd:

ildyria added the bug Something isn't working label Feb 11, 2022

nagmat84 linked a pull request Feb 11, 2022 that will close this issue

Fixes #1221 #1222

Merged

nagmat84 closed this as completed in #1222 Feb 11, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Import through CLI to user's root without specifying album #1221

Import through CLI to user's root without specifying album #1221

anoteng commented Feb 11, 2022 •

edited

Loading

ildyria commented Feb 11, 2022

nagmat84 commented Feb 11, 2022

nagmat84 commented Feb 11, 2022 •

edited

Loading

anoteng commented Feb 11, 2022

nagmat84 commented Feb 11, 2022

kamil4 commented Feb 11, 2022

anoteng commented Feb 11, 2022 via email

anoteng commented Feb 11, 2022 via email

ildyria commented Feb 11, 2022

Import through CLI to user's root without specifying album #1221

Import through CLI to user's root without specifying album #1221

Comments

anoteng commented Feb 11, 2022 • edited Loading

Detailed description of the problem [REQUIRED]

Steps to reproduce the issue

Browser and system

ildyria commented Feb 11, 2022

nagmat84 commented Feb 11, 2022

nagmat84 commented Feb 11, 2022 • edited Loading

anoteng commented Feb 11, 2022

nagmat84 commented Feb 11, 2022

kamil4 commented Feb 11, 2022

anoteng commented Feb 11, 2022 via email

anoteng commented Feb 11, 2022 via email

ildyria commented Feb 11, 2022

anoteng commented Feb 11, 2022 •

edited

Loading

nagmat84 commented Feb 11, 2022 •

edited

Loading