[Migration] MongoDB Atlas Sync depracated.

[GeoffreyChen777]

Hi folks, @igoogolx @charlieJ107

Thanks you both again for your consistent contribution to Paperlib.

We have a really big trouble. #607

MongoDB Atlas announced that they are deprecating the device sync and device sdk. Paperlib highly relies on these two functions.

• device sdk: the interface of our database.
• device sync cross device data sync.

We need to migrate these two functions before Sep 2025.

As it's a really big change, I wish you could help me with it.

The first step is finding a alternative solution.

Really appreciate.

(We can discuss in either English or Chinese, if you prefer. I believe we both speak Chinese)

[charlieJ107]

Shall we keep the current repositories interface and implement them with a RESTful API, so we can keep the sync features and allow the further extension

[GeoffreyChen777]

@charlieJ107 I think we can propose our own ideas, and we discuss them, finally we choose the best solution. I remember you have some ideas about the database interface before.

Here I provide some key features in the current Paperlib database that we should consider for the alternative solution:

• Document-based data schema. Currently, all data structure in Paperlib like a JSON object. Thus it would be better to still use document-based database, such as mongodb etc.
• Reactive. The current Paperlib UI is basically a database dashboard. We show the database with different query conditions in the UI.
  • When we change the database, the UI is refreshed automatically. For example, we change the title of a paper, we just need to update the title in the database. The UI will then show the update without manually refreshing.
  • When we change the query condition, the UI is also refreshed automatically.
  • When users use sync database, they update the database on the device A, the updates should be synced to the database on the device B, and refresh the UI of device B.
• Easy query.
  • We have a feature called smart-filter. It basically use the realm query language to filter the database.
  • The advanced searching also use the realm query language.
  • The tags/folders in the sidebar are also based on the realm query. There is a query sentence behind each tag/folder. When a user clicks a tag/folder in the sidebar, we send the corresponding query sentence to the database and get the required data.
• Sync. This is a really important feature.
  • Database should be automatically synced between different devices.
  • Local first. Users can still use a local copy of the online database if there is no internet connection. After the user connects the internet, the local copy should be merged with the online database.
  • Automatic conflict handeling.

[charlieJ107]

I believe the first issue we need to resolve is determining how much we want to change the current codebase and how many breaking changes we are willing to tolerate. Based on this, I propose two potential solutions with their pros and cons.

Option 1: Maintain the existing codebase, build a compatibility layer

The core of this solution is to remove the deprecated Realm features and build a compatibility layer to integrate new data storage and access methods into the current codebase.

Pros:

1. Minimized workload: Most of the work is focused on the data layer, while other parts of the system remain unchanged.
2. Existing architecture remains intact: The plugin system and software architecture can largely remain unchanged, reducing risk.
3. Lower learning curve: The development team won’t need to adapt to a new architecture, allowing for faster implementation.

Cons:

1. Technical debt: The compatibility layer might increase future maintenance complexity and add to the technical debt.
2. Limited flexibility: The current architecture’s limitations might hinder feature expansion in the future.
3. Sync and query challenges: We will need to implement sync and query language compatibility, which could become a bottleneck.

Option 2: Full project refactor, front-end/back-end separation

This solution involves a complete project refactor with a front-end/back-end separation architecture, decoupling data access from other services, and redesigning the system to be more flexible and modern.

Pros:

1. Decoupling architecture: It provides greater flexibility for future feature expansions and reduces dependencies.
2. Solves foundational issues: It addresses deep-rooted system problems, such as self-hosted and team-sharing functionalities.
3. Long-term growth: The new architecture lays a foundation for future scalability and performance improvements.
4. Sync and reactive needs are met: A back-end service can resolve sync and UI auto-refresh issues effectively.

Cons:

1. Significant workload: It requires a large-scale refactor, meaning a longer development time and more work.
2. Breaking changes: The new architecture may cause incompatibility with the existing system, leading to major changes.
3. High development cost: The team will need to learn and adapt to the new architecture, increasing the development difficulty.

My Recommendation

I think we need to reach a consensus on our short-term and long-term goals. If the goal is to quickly address the issue, Option 1 can serve as a temporary solution. However, in the long run, I favor Option 2 as it offers more flexibility and future scalability. We should carefully balance the costs and benefits of a full refactor and have thorough discussions on how to decouple existing components and adopt best practices to ensure the robustness of the system.

[GeoffreyChen777]

@charlieJ107

Thanks for your ideas.

As we have one year for this, it would be OK to choose option 2.

And, I have already finished the front-end/back-end separation work in the branch upgrade-and-change-build-system. Currently, the electron renderer process is just a pure web page, without any node integration. All services are running in two utility processes, one is for the extension-related services, and the other one is for other services including the database.

Thus, if we choose option 2, we will still focus mainly on implementing the database.

if we decide to implement the database part by ourselves, I think maybe we can mix option 1 and option 2:

• based on the current database service interface, we implement a compatibility layer to connect to some databases. I think we don't need to implement a database from scratch. It's too complicated.
• we consider some future features, such as team-sharing, if the current interface is not the optimal design, we change it. Otherwise, we keep it.

[GeoffreyChen777]

I'm investigating this database: https://rxdb.info/

It's an open-source database with a sync function.

[charlieJ107]

Thank you for your great work on refactoring and decoupling the front end and back end in the new branch. I think we can now focus on designing a scalable and flexible data access architecture. My suggestion is to refer to the common practice of Apps on mobile devices and adopt a two-layer data architecture with multiple parallel Data Sources and a unified Data Repository. Taking the Paper data as an example, there is usually a unified PaperRepository to provide data to other components in the application. In PaperRepository, a LocalDataSource and a RemoteDataSource are maintained. For LocalDataSource, the common implementation on Android Apps is to use the local Sqlite database, while RemoteDatasource uses the HTTP or other protocols to access the API on the cloud to implement data interaction. The Repository implements data synchronization and conflict handling of the two DataSources. I haven't studied the RxDB you mentioned in detail, but I believe that even if you change to any other local database, you can provide a LocalDataSource implementation, thereby decoupling the implementation of the local database from other components in the App. Even if RxDB or any local database implementation is unavailable again for some reason in the future, all we need to do is replace a LocalDataSource implementation.

The main purpose of RemoteDataSource is data synchronization. We previously synchronized data directly through Realm and MongoDB Atlas App Service. My suggestion is that we implement a data backend web application based on RESTful API and use production-level database programs to store data in this web application. We can also synchronize attachments, including Paper's PDFs and other attachments, uniformly through this backend.

I am happy to contribute an independent Web API application for synchronization backend. This web service is designed to run on the cloud, interact with the application through the HTTP-based Restful API, and provide a relatively simple web page to view the content in the database online (of course, the plug-in system cannot be implemented in the Web, but the client can store the data processed by the plug-in system in this API backend). We can provide a version that users can manage by themselves. Users with certain operation and maintenance experience can host their own storage backend. I can also develop an extended version to provide official synchronization services from Paperlib. This extended version will expand the account system belonging to Paperlib and lay the foundation for future team sharing and other functions.

The API backend idea is still in the prototype stage, but if it works (in other words, Paperlib has ambitions to offer official services), I'd be happy to discuss the details. Let me know what others think of this part.

[GeoffreyChen777]

Hi, thanks for your introduction.

I think, for paperlib, we don't need a real realtime sync database. We can tolerate some latency.

If I understand correctly, RxDB they use the server sent event to send signal to client APPs.
https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events

I guess it's a lightweight solution?

Your design sounds ok. We can start from it.

And about the deployment of the server database. Yes that's what I want. Professional users can deploy their own server. Other users can use the official server, maybe we can also provide free and paid plan in the future.

[charlieJ107]

Our current Sync API uses a RESTful API to implement data interaction, and currently considers polling to obtain database updates. Regarding the specific update method, I just thought of a way for the client to know whether it needs to obtain updated data and refresh the rendering. I will provide a Diff API in the API, which the client can use to call Get (perhaps with a local Diff ID), and the server will return the latest Diff ID. When the Diff ID carried in the request does not match the latest ID on the server (the server data has been updated), the Diff API will also return all the data that has changed since the requested ID and the server's latest ID (including data types, such as updated Paper or updated Tag) and the ID of the updated data entry. This way, the client can actively send a Get request again to get the corresponding updated data according to the ID in turn. I'm still wondering whether to set up a separate field for Diff to indicate whether new data has been created, so that the client can actively refresh the entire table; the alternative is to let the client determine whether to refresh the entire table based on the Diff obtained and by checking its own database records. But I think this Diff API can be used to obtain data updates in a lightweight manner, so that the client can update the data on demand without having to refresh the database each time it polls to trigger UI rendering.

[GeoffreyChen777]

the Diff API will also return all the data that has changed since the requested ID and the server's latest ID (including data types, such as updated Paper or updated Tag) and the ID of the updated data entry.

seems that you need to maintain a log to record all changes to the server db right?

[charlieJ107]

Yeah, some sort of that. It can also be implement in other backend tech stack (say some cloud native log service) but add a table in DB will be more easier to implement so we can make our sync service available asap. Since the diff API is unlikely changing if we changed the backend implement we can transfer from DB table to a more demure way later. Currently I'm planning to add a middleware that create a new record to the diff table whenever a write operation is made to the DB.

[GeoffreyChen777]

Cool, I think with this log DB, we can even implement more detailed conflict handling in the future.

[GeoffreyChen777]

I just checked the atlas, currently we cannot even create a new sync service. It means that for all new users, Paperlib cannot do the cross device sync.

[ZanderZhao]

Can I deploy MongoDB Atlas Sync in the cloud by myself, what should I fill in that address, I use docker mongodb/mongodb-atlas-local, and I found that filling in the ip+ port is not working

[GeoffreyChen777]

It’s not the sync server. Currently there is no way to self host a MongoDB sync server. We are developing our own sync server. Just wait for it.