Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mongo driver migration to an officially supported one + moving Orion to CentOS 8 #3622

Merged
merged 100 commits into from
Apr 9, 2021

Conversation

fgalan
Copy link
Member

@fgalan fgalan commented Feb 7, 2020

This PR is related with #3132 and provides the "wrapping modules" described in that issue.

At the present point, apart from a first version of the modules themselves (in the mongoDriver/ library) only a small part of the CB logic (in particular, the one related with subscription creation, update and retreival) has been migrated to use the new library.

The idea is to continue migrating more CB logic, enhancing mongoDriver/ in the process, if needed. This could be done either ini feature/3132_poc_isolating_driver_usage directly or as PR on that branch (contributions welcome! :)

Pay attention to the "FIXME OLD-DR" marks. They should be solved and removed before merging this PR into master.

Pending references to old driver outside mongoDriver (grep '#include "mongo/' -R src/* | grep -v mongoDriver | wc -l)

  • 35

@fgalan
Copy link
Member Author

fgalan commented Feb 7, 2020

Note that we have some part of CB using old mongo::BSON* classes and the other using orion::BSON* classes, we keep to separate connection pools:

$ contextBroker -fg -logLevel DEBUG -t 100 -logForHumans | grep "Connection info"
DEBUG@12:46:59  mongoConnectionPool.cpp[129]: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@12:46:59  mongoConnectionPool.cpp[129]: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@12:46:59  mongoConnectionPool.cpp[129]: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@12:46:59  mongoConnectionPool.cpp[129]: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@12:46:59  mongoConnectionPool.cpp[129]: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@12:46:59  mongoConnectionPool.cpp[129]: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@12:46:59  mongoConnectionPool.cpp[129]: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@12:46:59  mongoConnectionPool.cpp[129]: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@12:46:59  mongoConnectionPool.cpp[129]: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@12:46:59  mongoConnectionPool.cpp[129]: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@12:46:59  mongoConnectionPool.cpp[116]: orion:: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@12:46:59  mongoConnectionPool.cpp[116]: orion:: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@12:46:59  mongoConnectionPool.cpp[116]: orion:: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@12:46:59  mongoConnectionPool.cpp[116]: orion:: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@12:46:59  mongoConnectionPool.cpp[116]: orion:: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@12:46:59  mongoConnectionPool.cpp[116]: orion:: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@12:46:59  mongoConnectionPool.cpp[116]: orion:: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@12:46:59  mongoConnectionPool.cpp[116]: orion:: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@12:46:59  mongoConnectionPool.cpp[116]: orion:: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@12:46:59  mongoConnectionPool.cpp[116]: orion:: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000

Eventually, the old one will be removed and only the one with orion:: will remain.

@fgalan fgalan force-pushed the feature/3132_poc_isolating_driver_usage branch from 4335e0e to 287144b Compare July 21, 2020 11:27
@fgalan
Copy link
Member Author

fgalan commented Jul 21, 2020

At commit 287144b initial implementation of mongoDriver and following files migrated:

  • src/lib/apiTypesV2/HttpInfo.cpp
  • src/lib/apiTypesV2/HttpInfo.h
  • src/lib/mongoBackend/MongoCommonSubscription.cpp
  • src/lib/mongoBackend/MongoCommonSubscription.h
  • src/lib/mongoBackend/MongoCommonUpdate.cpp
  • src/lib/mongoBackend/MongoGlobal.cpp (not fully)
  • src/lib/mongoBackend/MongoGlobal.h (not fully)
  • src/lib/mongoBackend/mongoCreateSubscription.cpp
  • src/lib/mongoBackend/mongoGetSubscriptions.cpp
  • src/lib/mongoBackend/mongoSubCache.cpp (probably not fully)
  • src/lib/mongoBackend/mongoSubCache.h (probably not fully)
  • src/lib/mongoBackend/mongoUpdateSubscription.cpp

Pending references to old driver outside mongoDriver (grep '#include "mongo/' -R src/* | grep -v mongoDriver | wc -l)

  • 27

Functional tests log (from my local)

Total test time: 3321.71 seconds

WARNING: 5 test cases disabled:
  o  2: 0000_bad_requests/exit.test
  o  30: 0000_ipv6_support/ipv4_only.test
  o  31: 0000_ipv6_support/ipv6_only.test
  o  390: 0917_queryContext_behaves_differently/query_with_and_without_forwarding.test
  o  625: 1310_suspect_200OK/suspect_200OK.test

@fgalan fgalan force-pushed the feature/3132_poc_isolating_driver_usage branch from a190514 to c389579 Compare July 28, 2020 16:29
@fgalan
Copy link
Member Author

fgalan commented Jul 28, 2020

At commit c389579, almost old the code migrated (only some parts in MongoGlobal remain):

  • src/lib/apiTypesV2/HttpInfo.cpp
  • src/lib/cache/subCache.h
  • src/lib/mongoBackend/MongoCommonRegister.cpp
  • src/lib/mongoBackend/MongoCommonRegister.h
  • src/lib/mongoBackend/MongoCommonUpdate.cpp
  • src/lib/mongoBackend/MongoCommonUpdate.h
  • src/lib/mongoBackend/MongoGlobal.cpp (not fully)
  • src/lib/mongoBackend/MongoGlobal.h (not fully)
  • src/lib/mongoBackend/compoundResponses.cpp
  • src/lib/mongoBackend/compoundResponses.h
  • src/lib/mongoBackend/compoundValueBson.cpp
  • src/lib/mongoBackend/compoundValueBson.h
  • src/lib/mongoBackend/dateExpiration.cpp
  • src/lib/mongoBackend/dateExpiration.h
  • src/lib/mongoBackend/location.cpp
  • src/lib/mongoBackend/location.h
  • src/lib/mongoBackend/mongoDiscoverContextAvailability.cpp
  • src/lib/mongoBackend/mongoGetSubscriptions.cpp
  • src/lib/mongoBackend/mongoQueryTypes.cpp
  • src/lib/mongoBackend/mongoRegisterContext.cpp
  • src/lib/mongoBackend/mongoRegistrationCreate.cpp
  • src/lib/mongoBackend/mongoRegistrationDelete.cpp
  • src/lib/mongoBackend/mongoRegistrationGet.cpp
  • src/lib/mongoBackend/mongoSubCache.cpp
  • src/lib/mongoBackend/mongoUpdateSubscription.cpp
  • src/lib/ngsi/ContextAttribute.cpp
  • src/lib/ngsi/ContextAttribute.h
  • src/lib/ngsi/ContextElementResponse.cpp
  • src/lib/ngsi/ContextElementResponse.h
  • src/lib/ngsi/Metadata.cpp
  • src/lib/ngsi/Metadata.h
  • src/lib/rest/StringFilter.cpp
  • src/lib/rest/StringFilter.h

Pending references to old driver outside mongoDriver (grep '#include "mongo/' -R src/* | grep -v mongoDriver | wc -l)

  • 8

Functional tests log (from travis):

Total test time: 2548.58 seconds

WARNING: 5 test cases disabled:
  o  2: 0000_bad_requests/exit.test
  o  30: 0000_ipv6_support/ipv4_only.test
  o  31: 0000_ipv6_support/ipv6_only.test
  o  390: 0917_queryContext_behaves_differently/query_with_and_without_forwarding.test
  o  625: 1310_suspect_200OK/suspect_200OK.test

Unit test are expected to fail. Probably it is the same problem at commit 287144b, solved using DISABLED.
However, we are not paying too much attention to this by the moment, we'll fix one the new driver
gets in place.

@fgalan fgalan force-pushed the feature/3132_poc_isolating_driver_usage branch from d377716 to 7df0fb4 Compare July 30, 2020 12:00
@fgalan
Copy link
Member Author

fgalan commented Jul 30, 2020

At commit 7df0fb4, old code migrated.

Pending references to old driver outside mongoDriver (grep '#include "mongo/' -R src/* | grep -v mongoDriver | wc -l)

  • 0

In addition, the following files has been removed from mongoBackend (no longer used):

  • mongoConnectionPool.cpp|h
  • connectionOperations.cpp|h
  • safeMongo.cpp|h

With regards to this previous comment now the situation is:

$ contextBroker -fg -logLevel DEBUG -t 100 -logForHumans | grep "Connection info"
DEBUG@13:50:23  mongoConnectionPool.cpp[117]: orion:: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@13:50:23  mongoConnectionPool.cpp[117]: orion:: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@13:50:23  mongoConnectionPool.cpp[117]: orion:: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@13:50:23  mongoConnectionPool.cpp[117]: orion:: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@13:50:23  mongoConnectionPool.cpp[117]: orion:: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@13:50:23  mongoConnectionPool.cpp[117]: orion:: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@13:50:23  mongoConnectionPool.cpp[117]: orion:: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@13:50:23  mongoConnectionPool.cpp[117]: orion:: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@13:50:23  mongoConnectionPool.cpp[117]: orion:: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000
DEBUG@13:50:23  mongoConnectionPool.cpp[117]: orion:: Connection info: dbName='orion', rplSet='', timeout=4666723172467343.000000

Functional tests log (https://travis-ci.org/github/telefonicaid/fiware-orion/jobs/713271841)):

Total test time: 2510.69 seconds

WARNING: 5 test cases disabled:
  o  2: 0000_bad_requests/exit.test
  o  30: 0000_ipv6_support/ipv4_only.test
  o  31: 0000_ipv6_support/ipv6_only.test
  o  390: 0917_queryContext_behaves_differently/query_with_and_without_forwarding.test
  o  625: 1310_suspect_200OK/suspect_200OK.test

So the codebase is now ready to start with the driver migration work :) That will be done in a branch of feature/3132_poc_isolating_driver_usage

The following tasks are deferred after driver migration:

  • Enable some functions in connectionOperations.cpp, related with write concern (setWriteConcern, getWriteConern) and auth (connectionAuth). By the moment, Orion cannot authenticat on MongoDB nor change de write cocern.
  • Re-enable de unit tests
  • Review all remaining FIXME OLD-DR comments. Many of them are related with usage of the streaming API

@fgalan fgalan removed the help wanted label Feb 8, 2021
@fgalan
Copy link
Member Author

fgalan commented Feb 12, 2021

Stage one done in Stage 1 done in PR #3772

Note that in that PR we are the C official driver instead of the C++ official driver. Originally the plan was to use the C++ driver. However, at the end we have used the C driver given the following reasons:

  • We didn't manage to compile the new C++ driver in CentOS 7 (probably due to the driver is using "modern C++" and the gcc compiler in CentOS 7 is too old)
  • The C++ driver is in fact based in mongo C driver (which is installed as a prerequeriment of it). Thus, using mongo C driver directly removes a layer of software.
  • The high-level abstractions that C++ driver could be providing are in fact provided by our own mongoDriver/ module. We don't need the C++ driver to provide such abstractions. In other words, we are not losing the high-level abstractions that a C++ driver would provide as we are ourselves providing these abstractions to client code in mongoDriver/ API

@fgalan
Copy link
Member Author

fgalan commented Feb 12, 2021

Having completed stage 1 (the most difficult one :), the plan from now on is as follows. Each stage in the plan will be done in a separate PR (using task/3132_* pull branch as branch of this feature/3132_poc_isolating_driver_usage)

After merging this PR #3622, some additional tasks have to be done:

  • Remove old driver from Dockerfile for CI Remove regression stuff related with :rpm7 (.github/workflows and ci/rpm7/ if I remember correctly... do a grep -R rpm7 to find possible leftovers after that) -> after merging the PR, no rpm7 leftover remains (grep rpm7 * -R | wc -l results 0). Nothing to do
  • Re-enable disabled unit tests, using the new driver (this seems to be a heavy work task, it will be probably moved to an issue -> deferred at the end to issue mongodriver follow up: re-enable disabled unit tests #3775)

@fgalan fgalan changed the title [WIP] mongoDriver/ encapsulation library [WIP] mongo driver migration to an officially supported one Feb 12, 2021
…_second_stage

FIX code style and disable mongo unit test based in old driver
@fgalan fgalan mentioned this pull request Feb 18, 2021
@fgalan fgalan force-pushed the feature/3132_poc_isolating_driver_usage branch from 8f367a3 to 861665d Compare March 22, 2021 13:11
Update procedure for Ubuntu 20.04 LTS (amd64 and aarch64)
@fgalan
Copy link
Member Author

fgalan commented Mar 22, 2021

  • In Ubuntu (doc/manuals/admin/build_source.md#ubuntu-1804-lts), taking into account the new driver and the move to MongoDB 4.4. I did some early modifications on the documentation, but not sure if they are complete and a review from your side would be great.

I sent a PR #3819. It updataes the procedure to directly run Orion on Ubuntu (amd64 and aarch64). It would be great if you could review this PR.

Merged. Thanks!

That covers the first item. Regarding the second item:

  • In Raspberry Pi (docker/raspberry_pi.md and docker/raspberry_pi.jp.md). Probably this ones don't need too much adaptation, as they are based in Orion running in docker (and Dockefile has been already adapted by me in feature/3132_poc_isolating_driver_usage branch). Maybe just change docker build --build-arg=centos7 -t orion . to docker build --build-arg=centos8 -t orion . ?

It is not needed any adaptation in documents at the end?

@fisuda
Copy link
Contributor

fisuda commented Mar 22, 2021

I'm working on the second item. I will send a PR for updating the documentation.

@fisuda
Copy link
Contributor

fisuda commented Mar 26, 2021

  • In Raspberry Pi (docker/raspberry_pi.md and docker/raspberry_pi.jp.md). Probably this ones don't need too much adaptation, as they are based in Orion running in docker (and Dockefile has been already adapted by me in feature/3132_poc_isolating_driver_usage branch). Maybe just change docker build --build-arg=centos7 -t orion . to docker build --build-arg=centos8 -t orion . ?

I sent a PR #3825. It updates documentation to support CentOS 8 aarch64 for Raspberry Pi. It would be great if you could review this PR.

@fgalan
Copy link
Member Author

fgalan commented Apr 9, 2021

06d93e5 is a "candidate commit" to merge this PR.

Valgrind test pass:

$ ./valgrindTestSuite.sh
...
10 functional tests failed (not a leak, just a func-test failure):
   015: get_entity_dates_with_options.test (exit code 1)
   028: https.test (exit code 1)
   352: attribute_dates.test (exit code 1)
   356: filter_attr_datemodified_sub.test (exit code 1)
   362: filter_datecreated_sub.test (exit code 1)
   365: get_entity_dates.test (exit code 1)
   516: sub_from_active_to_fail.test (exit code 1)
   800: 2064_attrs_blacklist_complex.test (exit code 1)
   923: 2750_reset_metrics.test (exit code 1)
   1156: log_notifications.test (exit code 1)
---------------------------------------

Total test time: 21448.25 seconds (357 minutes)
Great, all valgrind tests ran without any memory leakage

Functional tests with cache:

$ ./testHarness.sh --cache
...
Total test time: 3259.53 seconds

WARNING: 5 test cases disabled:
  o  2: 0000_bad_requests/exit.test
  o  30: 0000_ipv6_support/ipv4_only.test
  o  31: 0000_ipv6_support/ipv6_only.test
  o  374: 0917_queryContext_behaves_differently/query_with_and_without_forwarding.test
  o  610: 1310_suspect_200OK/suspect_200OK.test

Functional test without cache:

$ ./testHarness.sh --noCache
...
Total test time: 3274.52 seconds

2 test cases OK in the second attempt:
  o  0876_entity_dates/get_entity_dates.test
  o  3068_cprs_full_functional_v2/fwd_v2_ngsiv2_query_one_forward_query.test

WARNING: 5 test cases disabled:
  o  2: 0000_bad_requests/exit.test
  o  30: 0000_ipv6_support/ipv4_only.test
  o  31: 0000_ipv6_support/ipv6_only.test
  o  374: 0917_queryContext_behaves_differently/query_with_and_without_forwarding.test
  o  610: 1310_suspect_200OK/suspect_200OK.test

Copy link
Collaborator

@mapedraza mapedraza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mapedraza mapedraza merged commit 05a0132 into master Apr 9, 2021
@fgalan fgalan deleted the feature/3132_poc_isolating_driver_usage branch April 9, 2021 16:15
@fgalan
Copy link
Member Author

fgalan commented Apr 9, 2021

The merging of this PR is an important milestone in the Orion Context Broker development. We are getting rid of the old legacy driver (a very toxic and dangerous dependency that has been around for too many years) and start using an officially supported one (the Mongo C driver).

Special thanks to @mapedraza and @fisuda for their contributions to this PR's branch. Also, special thanks to @bazile-clyde @kevinAlbs, from the MongoDB team, to help me to understand how the Mongo C driver works :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants