Updating nightly version nukes all my services

[starchild]

Avoid duplicates

• I agree to follow the Code of Conduct that this project adheres to.
• I have searched the issues tracker for a bug report similar to mine, in vain

Ferdium Version

6.4.1

What Operating System are you using?

macOS

Operating System Version

Ventura 13.2.1 (22D68)

What arch are you using?

arm64 (including Apple Silicon)

Last Known Working Ferdium version

No response

Expected Behavior

update should keep the app working as usual

Actual Behavior

All services are deleted and it's not possible to add them back. I have to reinstall the dmg from the website.

Steps to reproduce

click "restart and install update" on the nightly build update button
that's it

Debug link

No response

Screenshots

No response

Additional information

No response

[SpecialAro]

Hi @starchild

Taking into account that the last nightly is 6.5.0-nightly.X can you confirm which version caused this? Were you trying to switch from stable to nightly?

Also, are you using an accountless option or server option? If server, which server?

[starchild]

I had nightly already enabled. I've since switched to stable only so the upgrade button no longer appears.

Not sure about the other details - how can I check?

It started last week sometime actually, so it's been there for a while I guess...

[veresvit]

@starchild Revert back to version 6.5.0-nightly.5 - this works again as expected. I had the same problem with the nightly.6 and I was afraid to update since then. The whole app just appears empty after update. When revert back to nightly 5 all works as before,

[starchild]

@starchild Revert back to version 6.5.0-nightly.5 - this works again as expected. I had the same problem with the nightly.6 and I was afraid to update since then. The whole app just appears empty after update. When revert back to nightly 5 all works as before,

@veresvit yeah, I just downloaded the latest stable release and am using that. I'm just no longer doing nightly versions. No need really.

[ylluminate]

Just tried to update to Nightly 19 and lost services again:

Reverted back to Nightly 5 and sanity restored.

[SpecialAro]

Just tried to update to Nightly 19 and lost services again:

Reverted back to Nightly 5 and sanity restored.

Could you provide the logs for the non-working version (does nightly.6 works?) - CTRL + ALT + I to open the developers console. Then copy and paste the logs on Console.

EDIT: @ylluminate https://github.com/ferdium/ferdium-app/releases/tag/v6.5.0-nightly.6 if you can try it, please. Also, write in the console window.location.href - if you get a thing other than file:///C:/Program%20Files/Ferdium/resources/app.asar/index.html#/ it might be an issue

[Skowt]

A new update popup showed today and after clicking it I had this same issue. Here's some of the logs that popped up:

[Skowt]

Seems to not be related to nightly releases now:

[APz]

I encountered the same issue, on amd64 Mac version, upgrading from previous stable onto 6.5.0.

Downgrading to 6.4.1 seems to restore functionality, so the data files aren't fortunately corrupted.

[ylluminate]

@SpecialAro I did try 6 before and it had the same problem. 5 is the only nightly that presently works. I have not been able to test further for you and give my logs due to an "upgrade" to macOS 14 and having it create a bug that I am still recovering from and now being backlogged with work. I just don't feel "safe" updating to a newer version yet for risk of losing my settings entirely somehow through some misstep and my ridiculous schedule at present.

[bkimmett]

This is still broken for me upgrading from stable 6.4.1 to stable 6.5.1:

[SpecialAro]

@bkimmett would you mind to write the following on the console and take a screenshot of the result?
document.location.href

[bkimmett]

@bkimmett would you mind to write the following on the console and take a screenshot of the result? document.location.href

I got back file:///Applications/Ferdium.app/Contents/Resources/app.asar/index.html.

[SpecialAro]

@bkimmett would you mind to write the following on the console and take a screenshot of the result? document.location.href

I got back file:///Applications/Ferdium.app/Contents/Resources/app.asar/index.html.

Can you try to do document.location.href = 'file:///Applications/Ferdium.app/Contents/Resources/app.asar/index.html#/' and report back? I might have an idea of what is going on

[bkimmett]

@bkimmett would you mind to write the following on the console and take a screenshot of the result? document.location.href

I got back file:///Applications/Ferdium.app/Contents/Resources/app.asar/index.html.

Can you try to do document.location.href = 'file:///Applications/Ferdium.app/Contents/Resources/app.asar/index.html#/' and report back? I might have an idea of what is going on

Nothing different seems to happen:

These are the same messages I was getting before.

Did you mean to say 'app.asar.unpacked'?

[SpecialAro]

@bkimmett would you mind to write the following on the console and take a screenshot of the result? document.location.href

I got back file:///Applications/Ferdium.app/Contents/Resources/app.asar/index.html.

Can you try to do document.location.href = 'file:///Applications/Ferdium.app/Contents/Resources/app.asar/index.html#/' and report back? I might have an idea of what is going on

Nothing different seems to happen:

These are the same messages I was getting before.

Did you mean to say 'app.asar.unpacked'?

It should be just app.asar, but could you try to see what you get on the build that works 6.4.1 for instance?

I'm really sorry for making all this questions but I'm unable to reproduce this issue myself so I can only rely on your inputs...

Also, would you care to PM me on discord? I think I can provide you with a build that probably fixes this (or if you want I can tell you the commit you should revert)

[SpecialAro]

I'm blaming commit 970738c for this issue, that was introduced in PR #1344

[bkimmett]

So here's something interesting.

In the regular, working Ferdium (6.4.1), I see this in the log:

So I think those error messages were unrelated.

Entering document.location.href on yhe older version gives:

file:///Applications/Ferdium%206.4.app/Contents/Resources/app.asar/index.html (I renamed the bundle so that both versions can coexist in my applications folder).

I'd be happy to DM you on Discord. What's your username?

[SpecialAro]

So I think those error messages were unrelated.

Yes, I think they might be! The issue might be another one. Are you able to build code in your local or do you prefer that I send you a test version through discord? Search for "specialaro"

[bkimmett]

Sent you a friend request. Also, test version would be greatly appreciated. Most of my compiling code I do through homebrew these days XD

[SpecialAro]

Can anyone on this issue try to build this #1389 on an arm64 (either Windows or macOS). It think this can fix it (even though we tested - me and @bkimmett - and it didn't work)

[Alphrag]

Hey guys, if any of you can try the build I made for PR1389 and report back on it, that would be really useful. Details there: #1389 (comment)

(Note: There is no need to build anything yourself, and it can coexist next to the official Ferdium version without breaking it, so it is safe to try that way)

[bkimmett]

I tried it. It still doesn't work, alas, but I brought error logs! #1389 (comment)

Sadly, when I tried to follow it around in the debugger I couldn't get anywhere. However, I do note - as it's complaining about error 500 trying to access the server, I should note that I am specifically using Ferdium serverless.

So perhaps the error is in the app's "server lite" component somewhere?

[SpecialAro]

And the nightly version .5 still works and .6 doesn't?

@Alphrag would you be able to rebase my PR (or make a new one) to the nightly .7 (for instance) of the previous stable and see if this works for users? If so, it could mean that there are more than one issue we need to tackle - not only the DeepLink but some other as well.

(Sorry that I'm asking you to do that but I'm currently without active access to a computer 🙈)

[ylluminate]

I'm sorry to say @SpecialAro @Alphrag the new 6.5.2-nightly.1-arm64 still results in the same problem:

Console output: https://p153.p0.n0.cdn.getcloudapp.com/items/WnuprYJm/8d87111f-2908-4efb-83ab-399439654763.txt

[SpecialAro]

I'm sorry to say @SpecialAro @Alphrag the new 6.5.2-nightly.1-arm64 still results in the same problem:

Console output: https://p153.p0.n0.cdn.getcloudapp.com/items/WnuprYJm/8d87111f-2908-4efb-83ab-399439654763.txt

That nightly doesn't have the fixed I provided with the PR (that is currently on draft mode). @Alphrag I think we can rule out your that revert as being the problem.

[Alphrag]

Sorry, didn't get access to a computer with the repo setup for a few days.
I have now put multiple releases on my fork at https://github.com/Alphrag/ferdium-app/releases and I would like people to test them. They are called 6.5.1-fix.n where n is one of 6, 11, 14, 16, 21 and correspond to the appropriate releases of Ferdium of the form 6.5.0-nightly.n with PR1334 and 1361 removed.

Please test them, starting with the highest version first and going down if it is broken until you get one that works fine. This will greatly help find the culprit.

I will also add another one that will be called 6.5.1-osupdate.22 which will have both PR back in and will be compiled on the MacOS 13 instead of 12 in case this helps and no other problems are picked up on the other releases.

[ylluminate]

I'm sorry, is there a method to import my standard Ferdium settings into your new-name build so as to facilitate realistic testing @Alphrag?

[Alphrag]

I'm sorry, is there a method to import my standard Ferdium settings into your new-name build so as to facilitate realistic testing @Alphrag?

@ylluminate Yeah, the easiest is to open your normal Ferdium, and export your config (Help menu -> Import/Export Configuration Data to download the .ferdium-data file) and then importing it into Ferdium-Alphrag (same process as above, but import the file you just downloaded instead). This should set up all workspaces/services as you have them in Ferdium.

If you want to get your settings the same, you will however have to do it by hand (and it is not even sure it will fully work) by copying the file preferences located in the Ferdium folder located at ~/Library/Application Support/ into the folder of Ferdium-Alphrag in the same place. Thanks for trying it out!

[ylluminate]

@Alphrag thank you very much - I attempted to use 6.5.1-fix.21-arm64.dmg and unfortunately got the following after startup:

I then also attempted to import my 2023-10-08_160900 - export.ferdium-data from the main Ferdium export from 6.5.0-nightly.5 and it seems to have failed:

http://localhost:46570/transfer
Could not import your services into our system.
Error: Error: Knex: run
$ npm install sqlite3 --save
Cannot find module '/Applications/Ferdium-Alphrag.app/Contents/Resources/app.asar/node_modules/sqlite3/lib/binding/napi-v6-darwin-unknown-arm64/node_sqlite3.node'
Require stack:
- /Applications/Ferdium-Alphrag.app/Contents/Resources/app.asar/node_modules/sqlite3/lib/sqlite3-binding.js
- /Applications/Ferdium-Alphrag.app/Contents/Resources/app.asar/node_modules/sqlite3/lib/sqlite3.js
- /Applications/Ferdium-Alphrag.app/Contents/Resources/app.asar/node_modules/knex/lib/dialects/sqlite3/index.js
- /Applications/Ferdium-Alphrag.app/Contents/Resources/app.asar/node_modules/knex/lib/config-resolver.js
- /Applications/Ferdium-Alphrag.app/Contents/Resources/app.asar/node_modules/knex/lib/knex.js
- /Applications/Ferdium-Alphrag.app/Contents/Resources/app.asar/node_modules/knex/lib/index.js
- /Applications/Ferdium-Alphrag.app/Contents/Resources/app.asar/node_modules/knex/knex.js
- /Applications/Ferdium-Alphrag.app/Contents/Resources/app.asar/node_modules/@adonisjs/lucid/src/Database/index.js
- /Applications/Ferdium-Alphrag.app/Contents/Resources/app.asar/node_modules/@adonisjs/lucid/src/Database/Manager.js
- /Applications/Ferdium-Alphrag.app/Contents/Resources/app.asar/node_modules/@adonisjs/lucid/providers/LucidProvider.js
- /Applications/Ferdium-Alphrag.app/Contents/Resources/app.asar/node_modules/require-stack/src/index.js
- /Applications/Ferdium-Alphrag.app/Contents/Resources/app.asar/node_modules/require-stack/index.js
- /Applications/Ferdium-Alphrag.app/Contents/Resources/app.asar/node_modules/@adonisjs/fold/src/Ioc/index.js
- /Applications/Ferdium-Alphrag.app/Contents/Resources/app.asar/node_modules/@adonisjs/fold/index.js
- /Applications/Ferdium-Alphrag.app/Contents/Resources/app.asar/internal-server/start.js
- /Applications/Ferdium-Alphrag.app/Contents/Resources/app.asar/electron/ipc-api/localServer.js
- /Applications/Ferdium-Alphrag.app/Contents/Resources/app.asar/electron/ipc-api/index.js
- /Applications/Ferdium-Alphrag.app/Contents/Resources/app.asar/index.js
-

[Alphrag]

@ylluminate Oh thanks, that is quite a good thing to see for the error! It means that when building the app for arm, the sqlite3 component which should be rebuilt to have the appropriate bindings is not correctly build. Does any of the other versions between fix.6 and fix.21 work?

@SpecialAro or @balajiv113 I think we had a similar problem at some point and I have a vague memory of having to do an extra command to force the build of the bindings for arm, but I don't remember exactly what we did and where. Any ideas?

[ylluminate]

@Alphrag no, I'm sorry, identical failures.

[Alphrag]

We are considering only running the mac builds for x64 for now (avoiding arm) in order to release the different improvement that are necessary to fix CVEs and to upgrade the server. We will then triage again the arm builds in different ways to try to fix the problem before reintroducing them completely. I assume that these will run fine on arm mac, even though they might need to go through Rosetta.
Can anyone confirm that at least these work appropriately?

[bkimmett]

Oh man, I forgot about Rosetta! Let me check.

[bkimmett]

I can confirm the nightly (Mac x64) version loads all my services properly!!

[Alphrag]

To verify whether the problem came from any updates we made on top of nightly.5 or if it is a problem during the build itself, could people please try the arm asset of release 6.5.1-fixreset.5 on my fork?

@bkimmett Thanks for confirming that at least, temporarily stopping arm builds won't mean preventing you guys from having a working version anyway.

[ylluminate]

@Alphrag well, this was interesting. No error this time - "Get started" persists and no error appears below it. If I attempt to import the exported Ferdium config, it acts like it works with a response of Your account has been imported.... BUT no accounts or such appear in the Ferdium-Alphrag app itself... So this seems to be progress, but strange (perhaps another bug).

EDIT: after a third reimport and waiting a little bit it seems like the services finally came up randomly and spontaneously. Not sure what's going on there. I did notice that the custom icons were not saved/reimported and other settings were not restores (such as the horizontal layout and sizes of tabs).

[bkimmett]

Fixreset.5 (arm) doesn't work, but in a different way. It doesn't display the red banner at the bottom. But if I click 'Get Started', nothing loads in the available services panel, so it's a wash.

[ylluminate]

Notice my update above as well regarding it finally importing my services. So I think this does fix it.

[SpecialAro]

Notice my update above as well regarding it finally importing my services. So I think this does fix it.

In fact we haven't fixed a thing, we just mimicked the nightly.5 release to see if the problem was with the build or with the nightly.6 itself (or even with the sqlite3 biddings).

Thank you so much (@ylluminate and @bkimmett) for testing out this... We are so sorry but we have no way around to fix it (since neither me or @Alphrag have a ARM processor) other than ask you to test stuff... So thank you!

[Alphrag]

@ylluminate and @bkimmett Thank you very much for doing all this testing, we could not find the problem without you.
There is one PR from nightly.6 that I did not suspect would cause a problem but that I have decided to separate from the rest. So I have now published two new releases: one with the PR upgrading some npm modules tagged fixreset.5-modules and another one containing the other commits that were added to nightly.5 to get nightly.6 which is tagged fixreset.5-minor.
Let me know which of the two breaks/works, and hopefully we will be able to put back all the subsequent commits barring the culprit, to get back to an up-to-date working version.

[bkimmett]

fixreset.5-modules breaks!

[bkimmett]

And fixreset.5-minor works.

[Alphrag]

fixreset.5-modules breaks!

Looks like we've found the culprit here!
I have thus reverted this PR on top of the latest official release Ferdium v6.5.2-nightly.3 on my fork for a patched version. If this version works, we will revert it on the official repository, otherwise, I will restart the dissection to find the next breaking change.

[ylluminate]

Darn, this still yields the problem:

[Alphrag]

So... It is more complicated than just that PR being problematic, since it resides deep into how the modules are put inside during the build. However, upgrading to the latest electron-builder seems to work fine on the computer I tested.
@ylluminate and @bkimmett Could you test the osupdate-22 release I did last week please to confirm that?

If it does, we will need to fix something on the windows build since it currently fails there, but hopefully that will be easier.

Quick explication of what is going on:
The indirect dependency @electron/rebuild went from v3.2.13 in which our modules where rebuild when needed to v3.3.0 where it would call node-pre-gyp. However, this change meant we were no longer getting the bindings of sqlite3 properly included. In electron-builder v24.6.1 they have moved away from @electron/rebuild to go back to building it directly, bypassing this problem.
Note: the build failing for windows is a completely different problem linked to the interaction between electron-builder and pnpm that we have met in a slightly different way in the past (cf #862)

[ylluminate]

@Alphrag WORKS! Services load up that I'd previously exported and imported... 🎉

[bkimmett]

Could you test the osupdate-22 release I did last week please to confirm that?

Seems to work! No red banner, no Error 500s in the console.

[ylluminate]

@Alphrag thank you so much for your work in this area. You are a huge help. Any chance we'll see this mainlined in the near term with a new major/minor build release? Would very much like some of the other updates to see where some other issues stand that I've been having for a while...

[Alphrag]

@ylluminate Well, we think that we have managed to fix all the problems with @SpecialAro, so we are testing one last version 6.5.2-fullfix.3 before making the changes go onto the main repo.

[ylluminate]

Fires up fine. I don't see any issue @Alphrag @SpecialAro.