Replace Dockerfile Alpine linux with Ubuntu #885
Conversation
…sted with a sample request. Seems to work fine
|
what about using debian stable slim instead? |
What are the advantages? Is that image as lightweight as Alpine? |
|
Is not as lightweight, but is slimmer than ubuntu, and usually debian is more conservative (and consequently more stable) than ubuntu |
Dockerfile
Outdated
| @@ -1,20 +1,25 @@ | |||
| FROM alpine:3.8 AS build | |||
| FROM ubuntu:14.04 AS build | |||
There was a problem hiding this comment.
Why 14.04? According to https://wiki.ubuntu.com/Releases, it just went out of support.
There was a problem hiding this comment.
ohhh I didn't check Ubuntu's wiki. I just went for the version I considered most stable from the Docker's supported Ubuntu image list https://hub.docker.com/_/ubuntu/. Version 19.04 was just released this month. Do you think we should go for a more stable one? Version 18.04 maybe? From https://hub.docker.com/_/ubuntu/:
Supported tags and respective Dockerfile links
- 18.04, bionic-20190424, bionic, latest (bionic/Dockerfile)
- 18.10, cosmic-20190418, cosmic (cosmic/Dockerfile)
- 19.04, disco-20190423, disco, rolling (disco/Dockerfile)
- 14.04, trusty-20190425, trusty (trusty/Dockerfile)
- 16.04, xenial-20190425, xenial (xenial/Dockerfile)
There was a problem hiding this comment.
I would go with 18.04, the even numbered releases are LTS (Long Term Support). As per https://wiki.ubuntu.com/Releases 19.04 goes off support in 2020, while 18.04 is supported until 2023. I am not sure why 18.10 doesn't have the long supported life
Dockerfile
Outdated
| WORKDIR /go/src/github.com/prebid/prebid-server/ | ||
| RUN apk add -U --no-cache go git dep musl-dev | ||
| RUN \ | ||
| sed -i 's/# deb/deb/g' /etc/apt/sources.list && \ |
There was a problem hiding this comment.
I confess I don't understand exactly the purpose of lines 4-9. Two questions:
- The above line will presumably remove uncomment all lines with "# deb". It occurs to me that it may be better to make sure the pattern is anchored at the beginning of the line (i.e., "^# deb") to avoid uncommenting any real comments that may trail actual code on a line.
- I'm guessing these commands are all run every time the file is executed. Is it safe and necessary to always execute all these commands? In particular, I'm wondering about the commands on line 9.
There was a problem hiding this comment.
The above line will presumably remove uncomment all lines with "# deb"
Yes, from the original /etc/apt/sources.list file that comes with the image. Not all of the deb lines are originally not commented out. If we uncomment them all we have access to more repositories.
There was a problem hiding this comment.
It occurs to me that it may be better to make sure the pattern is anchored at the beginning of the line (i.e., "^# deb") to avoid uncommenting any real comments that may trail actual code on a line
It's not necessary, the file is pretty small and the regular expression "# deb" only matches repository URLs. File /etc/apt/sources.list AFTER sed edition:
root@2794e056a135:/# cat /etc/apt/sources.list
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://archive.ubuntu.com/ubuntu/ bionic main restricted
deb-src http://archive.ubuntu.com/ubuntu/ bionic main restricted
## Major bug fix updates produced after the final release of the
## distribution.
deb http://archive.ubuntu.com/ubuntu/ bionic-updates main restricted
deb-src http://archive.ubuntu.com/ubuntu/ bionic-updates main restricted
## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team. Also, please note that software in universe WILL NOT receive any
## review or updates from the Ubuntu security team.
deb http://archive.ubuntu.com/ubuntu/ bionic universe
deb-src http://archive.ubuntu.com/ubuntu/ bionic universe
deb http://archive.ubuntu.com/ubuntu/ bionic-updates universe
deb-src http://archive.ubuntu.com/ubuntu/ bionic-updates universe
## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team, and may not be under a free licence. Please satisfy yourself as to
## your rights to use the software. Also, please note that software in
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb http://archive.ubuntu.com/ubuntu/ bionic multiverse
deb-src http://archive.ubuntu.com/ubuntu/ bionic multiverse
deb http://archive.ubuntu.com/ubuntu/ bionic-updates multiverse
deb-src http://archive.ubuntu.com/ubuntu/ bionic-updates multiverse
## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
deb http://archive.ubuntu.com/ubuntu/ bionic-backports main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu/ bionic-backports main restricted universe multiverse
## Uncomment the following two lines to add software from Canonical's
## 'partner' repository.
## This software is not part of Ubuntu, but is offered by Canonical and the
## respective vendors as a service to Ubuntu users.
deb http://archive.canonical.com/ubuntu bionic partner
deb-src http://archive.canonical.com/ubuntu bionic partner
deb http://security.ubuntu.com/ubuntu/ bionic-security main restricted
deb-src http://security.ubuntu.com/ubuntu/ bionic-security main restricted
deb http://security.ubuntu.com/ubuntu/ bionic-security universe
deb-src http://security.ubuntu.com/ubuntu/ bionic-security universe
deb http://security.ubuntu.com/ubuntu/ bionic-security multiverse
deb-src http://security.ubuntu.com/ubuntu/ bionic-security multiverse
There was a problem hiding this comment.
I'm guessing these commands are all run every time the file is executed.
Yes
Is it safe and necessary to always execute all these commands? In particular, I'm wondering about the commands on line 9.
It's not uncommon for a Dockerfile to have these commands. If any of these fail, the entire build fails, and a proper error code is returned. I'd say it is as safe as the previous version
There was a problem hiding this comment.
Line 9 cleans the cache and temp folders that got populated from the package update and installation instructions above
There was a problem hiding this comment.
The /etc/apt/sources.list file itself discusses it bruefly. This is the original version of the file, before the sed command on our Dockerfile kicks in
root@c23c1615c75b:/# cat /etc/apt/sources.list
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://archive.ubuntu.com/ubuntu/ bionic main restricted
# deb-src http://archive.ubuntu.com/ubuntu/ bionic main restricted
## Major bug fix updates produced after the final release of the
## distribution.
deb http://archive.ubuntu.com/ubuntu/ bionic-updates main restricted
# deb-src http://archive.ubuntu.com/ubuntu/ bionic-updates main restricted
## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team. Also, please note that software in universe WILL NOT receive any
## review or updates from the Ubuntu security team.
deb http://archive.ubuntu.com/ubuntu/ bionic universe
# deb-src http://archive.ubuntu.com/ubuntu/ bionic universe
deb http://archive.ubuntu.com/ubuntu/ bionic-updates universe
# deb-src http://archive.ubuntu.com/ubuntu/ bionic-updates universe
## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team, and may not be under a free licence. Please satisfy yourself as to
## your rights to use the software. Also, please note that software in
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb http://archive.ubuntu.com/ubuntu/ bionic multiverse
# deb-src http://archive.ubuntu.com/ubuntu/ bionic multiverse
deb http://archive.ubuntu.com/ubuntu/ bionic-updates multiverse
# deb-src http://archive.ubuntu.com/ubuntu/ bionic-updates multiverse
## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
deb http://archive.ubuntu.com/ubuntu/ bionic-backports main restricted universe multiverse
# deb-src http://archive.ubuntu.com/ubuntu/ bionic-backports main restricted universe multiverse
## Uncomment the following two lines to add software from Canonical's
## 'partner' repository.
## This software is not part of Ubuntu, but is offered by Canonical and the
## respective vendors as a service to Ubuntu users.
# deb http://archive.canonical.com/ubuntu bionic partner
# deb-src http://archive.canonical.com/ubuntu bionic partner
deb http://security.ubuntu.com/ubuntu/ bionic-security main restricted
# deb-src http://security.ubuntu.com/ubuntu/ bionic-security main restricted
deb http://security.ubuntu.com/ubuntu/ bionic-security universe
# deb-src http://security.ubuntu.com/ubuntu/ bionic-security universe
deb http://security.ubuntu.com/ubuntu/ bionic-security multiverse
# deb-src http://security.ubuntu.com/ubuntu/ bionic-security multiverse
There was a problem hiding this comment.
Let me double check again whether or not what we are installing is available in the default repository URL's (it wasn't according to many tests I performed, but let me look further into it because maybe we don't need to uncomment them all)
There was a problem hiding this comment.
Going back to my comment about line 9: my concern was that we're nuking the contents of /tmp and /var/tmp which are public directories that could also contain data from other programs, users, etc. How can we be sure that randomly nuking those directories is safe?
There was a problem hiding this comment.
/var/lib/apt/lists
According to https://unix.stackexchange.com/questions/217369/clear-apt-get-list
rm /var/lib/apt/lists/* will remove the package lists. No repositories will be deleted, all that can happen is that tools like apt-cache cannot get package information and apt-get install will fail unless you run apt-get update again
/var/tmp and /tmp
Both /var/tmp and /tmp are short term and long term temporary files folders. In our context, the temporary files stored there so far are those of the package installation process and won't be relevant again because installation won't be run again until a new container is created from scratch. Up to line 6, prebid-server has not even run for the first time so no relevant files for its operation will be there.
* Use the correct labels for cache performance metric (prebid#904) * PubMatic Adslot validation (prebid#886) * testing a few changes to validate adslot, more to follow * Changed AdSlot to optional parameter, modified validation and test cases for the same * removed imp id from adslot validation error msgs * assign banner size * remove TrimSpace where it is not needed as recommended in the review * Implementation of Categories Http fetcher (prebid#882) * Implementation of Categories Http fetcher -Added code to fetch data using http -Added previously loaded categories to run http request just at the first time * Moved common Category struct to stored_request * Added comments * Minor refactoring * Minor refactoring -Added verification if category exist in map * Minor refactoring * Minor refactoring -strings.Replace changed to strings.TrimSuffix * Improved error handling on Beachfront adapter (prebid#873) * Removed a redundant error message that was causing some confusion. * trying to turn '[]' into null and 200 inti 404 * not a goot path * looking at the status codes * I think I have covers all these bases. * checking for empty array response a failing sanely * removed an attempt to pointlessly reset the status code. * corrected and added test cases * Fix Rubicon bidder for multi-format impression processing (prebid#902) * Replace Dockerfile Alpine linux with Ubuntu (prebid#885) * Ubuntu dockerfile works, includes backups and out file * Ubuntu Dockerfile creates image successfully and prebid-server was tested with a sample request. Seems to work fine * Ubuntu 18.04 now builds, validates.sh, and curls sample request successfully * Removed sed command * [fix] broken Go 1.9 compatibility (prebid#910) This CL addresses Go 1.9 compatibility issue along with adding back Go 1.9 to travis testing setup to prevent such bugs. Issue: prebid#895 * Fixed "invalid BidType: " error for lifestreet adapter (prebid#893) * Fixed "invalid BidType: " error for lifestreet adapter * After run gofmt * Used standard bid.ext type to get bid.ext.prebid.type * [Currency support] Activate multi-currencies support (prebid#894) This CL is the last piece to activate currency conversion support in PBS. It activates currency support per default. Currency rates will be fetched once per hour (PrebidJS file is updated once a day). Issue: prebid#280 * ImproveDigital adapter: (prebid#887) - Add support for video - Add support for mobile/app - Add support for multibid responses - Extend optional parameters * Add a PI exemption environment variable to PBS (prebid#916) * Refactored to official name of config item * Changes suggested by Mansi Nahar * [Sharethrough] Add new Sharethrough Adapter (prebid#903) * wip * wip * wip * exploration for sharethrough prebid-server * WIP: updating sharethrough adapter to latest prebid version Co-authored-by: Chris Nguyen <cnguyen@sharethrough.com> * WIP: adding butler params to the request #164291358 Co-authored-by: Josh Becker <jbecker@sharethrough.com> * Manage bid sizes [#164291358] Co-authored-by: Josh Becker <jbecker@sharethrough.com> * Manage GDPR [#164291358] Co-authored-by: Josh Becker <jbecker@sharethrough.com> * Populate prebid-server version if provided [#164291358] Co-authored-by: Josh Becker <jbecker@sharethrough.com> * Refactor gdpr data extraction from request [#164291358] Co-authored-by: Eddy Pechuzal <epechuzal@sharethrough.com> * Add instant play capability [#164291358] Co-authored-by: Eddy Pechuzal <epechuzal@sharethrough.com> * Split in multiple files [#164291358] Co-authored-by: Eddy Pechuzal <epechuzal@sharethrough.com> * Add s2s-win beacon todo? replace server name by server id? [#164291358] Co-authored-by: Eddy Pechuzal <epechuzal@sharethrough.com> * Removing `server` param in s2s-win beacon (will be added in imp req) [#164291358] * Clean up code + enable syncer (prebid#4) [#165257793] * Proper error handling (prebid#6) * Proper error handling [#165745574] * Address review (baby clean up) + catch error that was missed [#165745574] * Implement Unit Tests (prebid#7) * Proper error handling [#165745574] * Address review (baby clean up) + catch error that was missed [#165745574] * Implement unit tests for utils [#165891351] * Add UT for utils + butler [#165891351] Co-authored-by: Michael Duran <mduran@sharethrough.com> * Attempt for testing Bidder interface function implementations [#165891351] Co-authored-by: Michael Duran <mduran@sharethrough.com> * Finalizing Unit tests [#165891351] Co-authored-by: Chris Nguyen <cnguyen@sharethrough.com> Co-authored-by: Josh Becker <jbecker@sharethrough.com> * Fixing sharethrough.yaml capabilities [#165891351] Co-authored-by: Josh Becker <jbecker@sharethrough.com> * Send supplyId to imp req instead of hbSource (prebid#5) [#165477915] * Finalize PR (prebid#8) [#164911891] Co-authored-by: Josh Becker <jbecker@sharethrough.com> * Remove test setting * Add Sharethrough in syncer_test * Update deserializing of third party partners * Refactor/optimize UserAgent parsing (prebid#9) following josephveach's review in prebid#903 * Addressing June 3rd review from prebid#903 Optimizations, clean up suggested by @mansinahar * Addressing June 4th review from prebid#903 (prebid#10) * Addressing June 4th review from prebid#903 Clean up canAutoPlayVideo + hardcode bhVersion to unknown for now... * Removing hbVersion butler param since it's not accessible * Fix adMarkup error handling * [Mgid] Add new Mgid Adapter (prebid#907) * new mgid adapter * increase coverage * remove extra imports * Cache validation fix (prebid#911) * Cache validation fix if no bids returned - don't throw cache error, just return empty result * Cache validation fix - optimization: do not run auction logic if no bids returned * Cache validation fix: minor refactoring * Cache validation fix: minor refactoring * Cache validation fix: added unit test for no bids returned * Cache validation fix: minor refactoring * Remove hard coded targeting keys (prebid#923)
* Ubuntu dockerfile works, includes backups and out file * Ubuntu Dockerfile creates image successfully and prebid-server was tested with a sample request. Seems to work fine * Ubuntu 18.04 now builds, validates.sh, and curls sample request successfully * Removed sed command
* Ubuntu dockerfile works, includes backups and out file * Ubuntu Dockerfile creates image successfully and prebid-server was tested with a sample request. Seems to work fine * Ubuntu 18.04 now builds, validates.sh, and curls sample request successfully * Removed sed command
* Ubuntu dockerfile works, includes backups and out file * Ubuntu Dockerfile creates image successfully and prebid-server was tested with a sample request. Seems to work fine * Ubuntu 18.04 now builds, validates.sh, and curls sample request successfully * Removed sed command
We don't often have to ssh into a running PBS pod in production... but when we do, Alpine is a pain. It uses musl libc, which is different in subtle ways from the standard glibc. For example: Davidoff says Alpine made a mess of their DNS resolving.
We also find that some standard shell commands are missing. Depending on the problem, this can make debugging painful. We should change our dockerfile to use Ubuntu over Alpine, since it's more standard and widely used.
Deployment notes:
"The change would be the creation of the docker image. For the first iteration we should make docker images for both OSes, so we can switch between the two with no other variables to confirm that the ubuntu version is working properly.
Next release we make both, and maybe push ubuntu to just one datacenter."