Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make protocol util a singleton per thread #1743

Merged
merged 25 commits into from
Jan 10, 2020

Conversation

larohra
Copy link
Contributor

@larohra larohra commented Jan 6, 2020

Description

Recently we discovered a bug in the Guest Agent where the WireClient object different for different handlers which was due to the fact that a new ProtocolUtil object was being instantiated for every Hanlder (UpdateHandler, ExtHandler, etc).
This PR addresses this issue by making sure that a single ProtocolUtil object is used for each thread. It was made a SingletonPerThread to avoid the concurrency issues that we face if we make it a generic singleton (and also because the majority of the processing is done in the ExtHandler thread and not in the other threads)

Issue #


PR information

  • The title of the PR is clear and informative.
  • There are a small number of commits, each of which has an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, see this page.
  • Except for special cases involving multiple contributors, the PR is started from a fork of the main repository, not a branch.
  • If applicable, the PR references the bug/issue that it fixes in the description.
  • New Unit tests were added for the changes made and Travis.CI is passing.

Quality of Code and Contribution Guidelines


This change is Reviewable

@codecov
Copy link

codecov bot commented Jan 6, 2020

Codecov Report

Merging #1743 into develop will increase coverage by 0.07%.
The diff coverage is 77.77%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #1743      +/-   ##
===========================================
+ Coverage     68.5%   68.58%   +0.07%     
===========================================
  Files           82       82              
  Lines        11737    11743       +6     
  Branches      1645     1646       +1     
===========================================
+ Hits          8041     8054      +13     
+ Misses        3350     3345       -5     
+ Partials       346      344       -2
Impacted Files Coverage Δ
azurelinuxagent/common/event.py 84.66% <ø> (ø) ⬆️
azurelinuxagent/daemon/resourcedisk/default.py 37.98% <0%> (ø) ⬆️
azurelinuxagent/daemon/resourcedisk/freebsd.py 12% <0%> (ø) ⬆️
azurelinuxagent/ga/env.py 51.88% <0%> (ø) ⬆️
azurelinuxagent/agent.py 44.65% <0%> (ø) ⬆️
azurelinuxagent/common/version.py 69.17% <100%> (ø) ⬆️
azurelinuxagent/common/protocol/util.py 83.41% <100%> (ø) ⬆️
azurelinuxagent/common/logger.py 90.79% <100%> (ø) ⬆️
azurelinuxagent/ga/monitor.py 89.24% <100%> (ø) ⬆️
azurelinuxagent/ga/exthandlers.py 85.44% <100%> (+1%) ⬆️
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c03ed38...60d34e2. Read the comment docs.

azurelinuxagent/common/protocol/wire.py Outdated Show resolved Hide resolved
azurelinuxagent/common/protocol/wire.py Outdated Show resolved Hide resolved
azurelinuxagent/ga/monitor.py Show resolved Hide resolved
monitor_handler.start()
time.sleep(1)
self.assertTrue(monitor_handler.is_alive())
with patch.object(monitor_handler, 'protocol'):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this patching? Are you forcefully returning a mock when looking at self.protocol?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or you intend to test it with protocol to be None

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I'm forcefully returning a mock for the monitor_thread.protocol object. This was not required before my changes because we weren't calling the protocol.update_goal_state() before my changes. This change in behaviour was breaking the test due to which I had to mock it

@vrdmr vrdmr added this to the v2.2.46 milestone Jan 6, 2020
azurelinuxagent/common/SingletonPerThread.py Outdated Show resolved Hide resolved
azurelinuxagent/common/protocol/util.py Show resolved Hide resolved
azurelinuxagent/common/protocol/wire.py Outdated Show resolved Hide resolved
azurelinuxagent/ga/monitor.py Show resolved Hide resolved
azurelinuxagent/common/protocol/wire.py Outdated Show resolved Hide resolved
tests/common/test_SingletonPerThread.py Outdated Show resolved Hide resolved
tests/common/test_SingletonPerThread.py Outdated Show resolved Hide resolved
self.assertEqual(obj1.uuid, obj2.uuid)

def test_it_should_have_multiple_instances_for_multiple_threads(self):
output = Queue()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just curious: why output is a local and errors an instance member?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mainly because the error checking is done on a class level in the _setup_multithread_and_execute class but the checks for output are specific to the test.

tests/common/test_SingletonPerThread.py Outdated Show resolved Hide resolved
tests/ga/test_exthandlers.py Outdated Show resolved Hide resolved
Copy link
Contributor

@pgombar pgombar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of nits and questions. Thanks for the change!

azurelinuxagent/common/SingletonPerThread.py Outdated Show resolved Hide resolved
azurelinuxagent/common/SingletonPerThread.py Outdated Show resolved Hide resolved
tests/common/test_SingletonPerThread.py Outdated Show resolved Hide resolved
tests/protocol/test_protocol_util.py Show resolved Hide resolved
@narrieta narrieta removed this from the v2.2.46 milestone Jan 8, 2020
last_incarnation = None
if os.path.isfile(incarnation_file):
last_incarnation = fileutil.read_file(incarnation_file)
last_incarnation = self.get_goal_state().incarnation
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we do this until we remove the cache on disk? the current code always reads the file; get_goal_state may return the in-memory value

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should keep this change on how we check for last_incarnation because without this the objects would again go out of sync as soon as some other thread updates the goal_state.
I can change the get_goal_state() to self.goal_state until the file cache is removed though.

narrieta
narrieta previously approved these changes Jan 8, 2020
pgombar
pgombar previously approved these changes Jan 8, 2020
Copy link
Contributor

@pgombar pgombar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with minor comments

tests/ga/test_monitor.py Outdated Show resolved Hide resolved
tests/ga/test_remoteaccess_handler.py Outdated Show resolved Hide resolved
tests/tools.py Show resolved Hide resolved
@larohra larohra dismissed stale reviews from pgombar and narrieta via cc0a924 January 9, 2020 00:18
pgombar
pgombar previously approved these changes Jan 9, 2020
pgombar
pgombar previously approved these changes Jan 9, 2020
narrieta
narrieta previously approved these changes Jan 9, 2020
@larohra larohra dismissed stale reviews from narrieta and pgombar via 0c12a1f January 10, 2020 02:16
Copy link
Member

@narrieta narrieta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one minor comment; otherwise LGTM

@larohra larohra merged commit e780d76 into Azure:develop Jan 10, 2020
@larohra larohra deleted the SingletonWireClient branch January 10, 2020 19:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants