Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a way to get the log from the docker container. #147

Merged
merged 21 commits into from
Mar 13, 2018

Conversation

allenh1
Copy link
Contributor

@allenh1 allenh1 commented Mar 5, 2018

This creates a logger mechanism for the docker container. We create a temporary directory using TempfileManager, then send the stderrand stdout streams into the file in append mode.

This allows us to grab the output even when commands return an error code, which is more desirable than just grabbing the output of successful commands (imo).

ToDo: parameterize the verbosity, add tests.

connects to #146

@allenh1
Copy link
Contributor Author

allenh1 commented Mar 7, 2018

Coverage report.

$ coverage run --source=superflore -m nose && coverage report -m
....................................................
----------------------------------------------------------------------
Ran 52 tests in 13.115s

OK
Name                                               Stmts   Miss  Cover   Missing
--------------------------------------------------------------------------------
superflore/CacheManager.py                            20      0   100%
superflore/PackageMetadata.py                         22      0   100%
superflore/TempfileManager.py                         34      6    82%   48-56
superflore/__init__.py                                10      4    60%   5-8
superflore/docker.py                                  68     17    75%   42, 56-69, 75, 94-95, 105-110
superflore/exceptions.py                              18      2    89%   33, 38
superflore/generate_installers.py                     63      0   100%
superflore/generators/__init__.py                      0      0   100%
superflore/generators/bitbake/__init__.py              3      1    67%   3
superflore/generators/bitbake/gen_packages.py        102     79    23%   40-122, 129-168, 175-183, 189
superflore/generators/bitbake/ros_meta.py             27     18    33%   24-27, 30-35, 38-51, 54-55
superflore/generators/bitbake/run.py                 113     90    20%   43-187
superflore/generators/bitbake/yocto_recipe.py        120    100    17%   44-67, 70, 73-77, 80-92, 95-99, 102-104, 107-108, 116-119, 128-187
superflore/generators/ebuild/__init__.py               3      1    67%   3
superflore/generators/ebuild/ebuild.py               185     19    90%   96-99, 126-131, 146, 176-180, 186, 193, 206-210
superflore/generators/ebuild/gen_packages.py         140    113    19%   46-119, 125-137, 143-188, 193-210, 213, 216
superflore/generators/ebuild/metadata_xml.py          33      0   100%
superflore/generators/ebuild/overlay_instance.py      35     24    31%   26-31, 34-45, 50-69, 72-73
superflore/generators/ebuild/run.py                  121     95    21%   48-188, 192-196
superflore/parser.py                                  13      0   100%
superflore/repo_instance.py                           70     51    27%   29-49, 56-65, 68-76, 82-83, 89, 95-96, 102, 105-120, 123
superflore/rosdep_support.py                          34      2    94%   83-84
superflore/test_integration/__init__.py                0      0   100%
superflore/test_integration/gentoo/__init__.py         3      1    67%   3
superflore/test_integration/gentoo/build_base.py      28     19    32%   26-28, 33, 38-52
superflore/test_integration/gentoo/main.py            31     25    19%   24-75
superflore/utils.py                                  189     65    66%   50-56, 60-63, 67-82, 86-97, 101-104, 114, 174, 191-194, 196, 202, 208-209, 214, 226-247
--------------------------------------------------------------------------------
TOTAL                                               1485    732    51%

Copy link
Member

@tfoote tfoote left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this looks good. Two quick comments to look at.

cmd_string = self.get_command(tmp)
if show_cmd:
msg = "Running container with command string '%s'..."
info(msg % self.get_command())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You shouldn't regenerate the command here but use cmd_string

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue with that is the fact that get_command() without a log file will return the list, but, otherwise, shows a &&> /tmp/[tmp directory/log.txt file.

I could perhaps just do a .replace, but I don't think it's any more efficient at that point.

Or, if you want, I can just have it show the temporary log file mapping.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to show the real values. If you mention the mounting right after that would allow people to debug easily.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do

@@ -69,3 +69,17 @@ def test_pull(self):
docker_instance.pull('allenh1', 'ros_gentoo_base')
docker_instance.add_bash_command("echo Hello, Gentoo")
docker_instance.run()

def test_get_command(self):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to add a test of the contents of the log output too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, that's a good idea. I'll get that going right now, then!

@allenh1
Copy link
Contributor Author

allenh1 commented Mar 12, 2018

@tfoote That great JSON error has resurfaced in the last test I added (because of the fact it has to run the container). I'm giving that one more go before I remove it from the CI tests.

@allenh1
Copy link
Contributor Author

allenh1 commented Mar 12, 2018

$ python3 -m coverage run --source=superflore -m nose && python3 -m coverage report -m
.....................................................
----------------------------------------------------------------------
Ran 53 tests in 14.881s

OK
Name                                            Stmts   Miss  Cover   Missing
-----------------------------------------------------------------------------
superflore/CacheManager                            20      0   100%   
superflore/PackageMetadata                         22      0   100%   
superflore/TempfileManager                         34      6    82%   48-56
superflore/__init__                                10      4    60%   5-8
superflore/docker                                  68     17    75%   42, 56-69, 75, 94-95, 105-110
superflore/exceptions                              18      2    89%   33, 38
superflore/generate_installers                     63      0   100%   
superflore/generators/__init__                      0      0   100%   
superflore/generators/bitbake/__init__              3      1    67%   3
superflore/generators/bitbake/gen_packages        102     79    23%   40-122, 129-168, 175-183, 189
superflore/generators/bitbake/ros_meta             27     18    33%   24-27, 30-35, 38-51, 54-55
superflore/generators/bitbake/run                 113     90    20%   43-187
superflore/generators/bitbake/yocto_recipe        120    100    17%   44-67, 70, 73-77, 80-92, 95-99, 102-104, 107-108, 116-119, 128-187
superflore/generators/ebuild/__init__               3      1    67%   3
superflore/generators/ebuild/ebuild               185     19    90%   96-99, 126-131, 146, 176-180, 186, 193, 206-210
superflore/generators/ebuild/gen_packages         140    113    19%   46-119, 125-137, 143-188, 193-210, 213, 216
superflore/generators/ebuild/metadata_xml          33      0   100%   
superflore/generators/ebuild/overlay_instance      35     24    31%   26-31, 34-45, 50-69, 72-73
superflore/generators/ebuild/run                  121     95    21%   48-188, 192-196
superflore/parser                                  13      0   100%   
superflore/repo_instance                           70     51    27%   29-49, 56-65, 68-76, 82-83, 89, 95-96, 102, 105-120, 123
superflore/rosdep_support                          34      2    94%   83-84
superflore/test_integration/__init__                0      0   100%   
superflore/test_integration/gentoo/__init__         3      1    67%   3
superflore/test_integration/gentoo/build_base      28     19    32%   26-28, 33, 38-52
superflore/test_integration/gentoo/main            31     25    19%   24-75
superflore/utils                                  189     65    66%   50-56, 60-63, 67-82, 86-97, 101-104, 114, 174, 191-194, 196, 202, 208-209, 214, 226-247
-----------------------------------------------------------------------------
TOTAL                                            1485    732    51%   

@allenh1 allenh1 requested a review from tfoote March 12, 2018 20:34
Copy link
Member

@tfoote tfoote left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Two quick comments left on the log path output.

cmd_string = self.get_command(tmp)
if show_cmd:
msg = "Running container with command string '%s'..."
info(msg % self.get_command())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to show the real values. If you mention the mounting right after that would allow people to debug easily.

)
ok("Docker container exited.")
except docker.errors.ContainerError:
err("Docker container exited with errors.")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be good to state where the log file can be found for future reference here. In case someone wants to run it through grep or something.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was considering adding a --log-file flag. I think I'll go ahead and do that. As of now, the file gets removed (since it's in a tempfile).

@allenh1 allenh1 force-pushed the add-docker-output branch from 9eb04d8 to 1471d5e Compare March 13, 2018 21:27
@allenh1 allenh1 force-pushed the add-docker-output branch from 2798ea6 to 30189c5 Compare March 13, 2018 21:43
@allenh1
Copy link
Contributor Author

allenh1 commented Mar 13, 2018

Ok, that should just about do it for this one!

$ coverage run --source=superflore -m nose && coverage report -m
.....................................................
----------------------------------------------------------------------
Ran 53 tests in 17.698s

OK
Name                                               Stmts   Miss  Cover   Missing
--------------------------------------------------------------------------------
superflore/CacheManager.py                            20      0   100%
superflore/PackageMetadata.py                         22      0   100%
superflore/TempfileManager.py                         34      6    82%   48-56
superflore/__init__.py                                10      4    60%   5-8
superflore/docker.py                                  78     23    71%   42, 56-69, 75, 93-95, 102, 107-108, 119-127
superflore/exceptions.py                              18      2    89%   33, 38
superflore/generate_installers.py                     63      0   100%
superflore/generators/__init__.py                      0      0   100%
superflore/generators/bitbake/__init__.py              3      1    67%   3
superflore/generators/bitbake/gen_packages.py        102     79    23%   40-122, 129-168, 175-183, 189
superflore/generators/bitbake/ros_meta.py             27     18    33%   24-27, 30-35, 38-51, 54-55
superflore/generators/bitbake/run.py                 113     90    20%   43-187
superflore/generators/bitbake/yocto_recipe.py        120    100    17%   44-67, 70, 73-77, 80-92, 95-99, 102-104, 107-108, 116-119, 128-187
superflore/generators/ebuild/__init__.py               3      1    67%   3
superflore/generators/ebuild/ebuild.py               185     19    90%   96-99, 126-131, 146, 176-180, 186, 193, 206-210
superflore/generators/ebuild/gen_packages.py         140    113    19%   46-119, 125-137, 143-188, 193-210, 213, 216
superflore/generators/ebuild/metadata_xml.py          33      0   100%
superflore/generators/ebuild/overlay_instance.py      35     24    31%   26-31, 34-45, 50-69, 72-73
superflore/generators/ebuild/run.py                  121     95    21%   48-188, 192-196
superflore/parser.py                                  13      0   100%
superflore/repo_instance.py                           70     51    27%   29-49, 56-65, 68-76, 82-83, 89, 95-96, 102, 105-120, 123
superflore/rosdep_support.py                          34      2    94%   83-84
superflore/test_integration/__init__.py                0      0   100%
superflore/test_integration/gentoo/__init__.py         3      1    67%   3
superflore/test_integration/gentoo/build_base.py      28     19    32%   26-28, 33, 38-54
superflore/test_integration/gentoo/main.py            32     26    19%   24-80
superflore/utils.py                                  189     65    66%   50-56, 60-63, 67-82, 86-97, 101-104, 114, 174, 191-194, 196, 202, 208-209, 214, 226-247
--------------------------------------------------------------------------------
TOTAL                                               1496    739    51%

You can now specify where to send the log.

$ superflore-check-ebuilds --ros-distro kinetic --log-file ~/kinetic_catkin_log.txt --pkgs catkin
>>>> testing gentoo package integrity
>>>> Running container with command string 'bash -c 'emaint sync -r ros-overlay &>> /home/allenh1/kinetic_catkin_log.txt && emerge ros-kinetic/catkin &>> /home/allenh1/kinetic_catkin_log.txt''...
>>>> Docker container exited.
>>>> Log file: '/home/allenh1/kinetic_catkin_log.txt'
>>>>   'ros-kinetic/catkin': building

@allenh1 allenh1 requested a review from tfoote March 13, 2018 21:58
@allenh1 allenh1 merged commit 967b96d into master Mar 13, 2018
@allenh1 allenh1 deleted the add-docker-output branch March 13, 2018 23:51
zffgithub pushed a commit to zffgithub/superflore that referenced this pull request Apr 11, 2023
…cture#147)

* Added a `--verbose` flag

* Added a `--log-file` argument to specify the name and path to store the log file.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants