Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exceptions in make initdb #1511

Closed
Carreau opened this issue Dec 1, 2016 · 14 comments
Closed

Exceptions in make initdb #1511

Carreau opened this issue Dec 1, 2016 · 14 comments

Comments

@Carreau
Copy link
Contributor

Carreau commented Dec 1, 2016

After clearing up my system on all docker-container, docker images and updating almost everything, a make initdb give me the following:

  File "/usr/local/lib/python3.5/site-packages/urllib3/connection.py", line 138, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.5/site-packages/urllib3/util/connection.py", line 98, in create_connection
    raise err
  File "/usr/local/lib/python3.5/site-packages/urllib3/util/connection.py", line 88, in create_connection
    sock.connect(sa)
OSError: [Errno 113] No route to host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/elasticsearch/connection/http_urllib3.py", line 109, in perform_request
    response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
  File "/usr/local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 643, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.5/site-packages/urllib3/util/retry.py", line 311, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.5/site-packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/usr/local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 594, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 361, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.5/http/client.py", line 1106, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/lib/python3.5/http/client.py", line 1151, in _send_request
    self.endheaders(body)
  File "/usr/local/lib/python3.5/http/client.py", line 1102, in endheaders
    self._send_output(message_body)
  File "/usr/local/lib/python3.5/http/client.py", line 934, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.5/http/client.py", line 877, in send
    self.connect()
  File "/usr/local/lib/python3.5/site-packages/urllib3/connection.py", line 163, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.5/site-packages/urllib3/connection.py", line 147, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fd0b2063278>: Failed to establish a new connection: [Errno 113] No route to host
{"logger": "elasticsearch", "level": "WARNING", "event": "POST http://elasticsearch:9200/_bulk [status:N/A request:1.647s]", "thread": 140534582384384}
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/urllib3/connection.py", line 138, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.5/site-packages/urllib3/util/connection.py", line 98, in create_connection
    raise err
  File "/usr/local/lib/python3.5/site-packages/urllib3/util/connection.py", line 88, in create_connection
    sock.connect(sa)
OSError: [Errno 113] No route to host

During handling of the above exception, another exception occurred:
(.. repeat last many times)

Though warehouse seem to work.

That's on MacOS if it changes anything.

@di
Copy link
Member

di commented Dec 1, 2016

@Carreau Can you include everything after the make initdb command? I'm curious if this is happening after the docker-compose run web python -m warehouse search reindex command.

@di
Copy link
Member

di commented Dec 1, 2016

This might be because we recently upgraded to ElasticSearch 5.0 in #1494. Docker doesn't seem to rebuild containers when the docker-compose.yml file changes, so if you docker rm warehouse_elasticsearch_1, and then make initdb, it will recreate the container (now at 5.0) and should work.

@Carreau
Copy link
Contributor Author

Carreau commented Dec 1, 2016

Thanks @di ,

No it does not works, and as a output is ... a bit long I pasted it in a gist.

I also try to nike all container and images docker rm $(docker ps -a -q)) and docker rmi $(docker images), plus manually clean all the rest. It did redowload all from scratch for 10-15 minutes, so my guess is my images are clean.

@di
Copy link
Member

di commented Dec 1, 2016

@Carreau Hmm. After running make serve, is the warehouse_elasticsearch_1 container running? What is the output of docker ps?

@Carreau
Copy link
Contributor Author

Carreau commented Dec 1, 2016

Hum you are right it is not running and I have these line in the middle of my logs:

rabbitmq_1       | =INFO REPORT==== 1-Dec-2016::21:08:08 ===
rabbitmq_1       | Server startup complete; 0 plugins started.
warehouse_elasticsearch_1 exited with code 1
web_1            | [2016-12-01 21:21:01 +0000] [1] [INFO] Starting gunicorn 19.6.0
web_1            | [2016-12-01 21:21:01 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)

it does not affect me for what I am trying to accomplish right now. I'll try to keep an eye to see if I can figure out what make it fail.

@di
Copy link
Member

di commented Dec 1, 2016

Since those logs are all mixed between containers, the output of just docker-compose up elasticsearch might be more helpful.

@Carreau
Copy link
Contributor Author

Carreau commented Dec 1, 2016

Ha yes, that's immediately more obvious:

elasticsearch_1  | OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000085330000, 2060255232, 0) failed; error='Cannot allocate memory' (errno=12)
elasticsearch_1  | #
elasticsearch_1  | # There is insufficient memory for the Java Runtime Environment to continue.
elasticsearch_1  | # Native memory allocation (mmap) failed to map 2060255232 bytes for committing reserved memory.
elasticsearch_1  | # Can not save log file, dump to screen..
elasticsearch_1  | #
elasticsearch_1  | # There is insufficient memory for the Java Runtime Environment to continue.
elasticsearch_1  | # Native memory allocation (mmap) failed to map 2060255232 bytes for committing reserved memory.
elasticsearch_1  | # Possible reasons:
elasticsearch_1  | #   The system is out of physical RAM or swap space
elasticsearch_1  | #   In 32 bit mode, the process size limit was hit
elasticsearch_1  | # Possible solutions:
elasticsearch_1  | #   Reduce memory load on the system
elasticsearch_1  | #   Increase physical memory or swap space
elasticsearch_1  | #   Check if swap backing store is full
elasticsearch_1  | #   Use 64 bit Java on a 64 bit OS
elasticsearch_1  | #   Decrease Java heap size (-Xmx/-Xms)
elasticsearch_1  | #   Decrease number of Java threads
elasticsearch_1  | #   Decrease Java thread stack sizes (-Xss)
elasticsearch_1  | #   Set larger code cache with -XX:ReservedCodeCacheSize=
elasticsearch_1  | # This output file may be truncated or incomplete.
elasticsearch_1  | #
elasticsearch_1  | #  Out of Memory Error (os_linux.cpp:2630), pid=1, tid=0x00007f223174f700
elasticsearch_1  | #
elasticsearch_1  | # JRE version:  (8.0_111-b14) (build )
elasticsearch_1  | # Java VM: OpenJDK 64-Bit Server VM (25.111-b14 mixed mode linux-amd64 compressed oops)
elasticsearch_1  | # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
elasticsearch_1  | #

@Carreau
Copy link
Contributor Author

Carreau commented Dec 1, 2016

Ok, After stopping Docker-machine and pushing the Max RAM in VirtualBox.app to 4Gb I get the following:

elasticsearch_1  | [2016-12-01T21:51:03,397][INFO ][o.e.n.Node               ] [VzDvGr5] version[5.0.2], pid[1], build[f6b4951/2016-11-24T10:07:18.101Z], OS[Linux/4.4.27-boot2docker/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_111/25.111-b14]
elasticsearch_1  | [2016-12-01T21:51:14,582][INFO ][o.e.p.PluginsService     ] [VzDvGr5] loaded module [aggs-matrix-stats]
elasticsearch_1  | [2016-12-01T21:51:14,599][INFO ][o.e.p.PluginsService     ] [VzDvGr5] loaded module [ingest-common]
elasticsearch_1  | [2016-12-01T21:51:14,601][INFO ][o.e.p.PluginsService     ] [VzDvGr5] loaded module [lang-expression]
elasticsearch_1  | [2016-12-01T21:51:14,603][INFO ][o.e.p.PluginsService     ] [VzDvGr5] loaded module [lang-groovy]
elasticsearch_1  | [2016-12-01T21:51:14,607][INFO ][o.e.p.PluginsService     ] [VzDvGr5] loaded module [lang-mustache]
elasticsearch_1  | [2016-12-01T21:51:14,607][INFO ][o.e.p.PluginsService     ] [VzDvGr5] loaded module [lang-painless]
elasticsearch_1  | [2016-12-01T21:51:14,609][INFO ][o.e.p.PluginsService     ] [VzDvGr5] loaded module [percolator]
elasticsearch_1  | [2016-12-01T21:51:14,611][INFO ][o.e.p.PluginsService     ] [VzDvGr5] loaded module [reindex]
elasticsearch_1  | [2016-12-01T21:51:14,613][INFO ][o.e.p.PluginsService     ] [VzDvGr5] loaded module [transport-netty3]
elasticsearch_1  | [2016-12-01T21:51:14,614][INFO ][o.e.p.PluginsService     ] [VzDvGr5] loaded module [transport-netty4]
elasticsearch_1  | [2016-12-01T21:51:14,620][INFO ][o.e.p.PluginsService     ] [VzDvGr5] no plugins loaded
elasticsearch_1  | [2016-12-01T21:51:16,451][WARN ][o.e.d.s.g.GroovyScriptEngineService] [groovy] scripts are deprecated, use [painless] scripts instead
elasticsearch_1  | [2016-12-01T21:51:40,056][INFO ][o.e.n.Node               ] [VzDvGr5] initialized
elasticsearch_1  | [2016-12-01T21:51:40,065][INFO ][o.e.n.Node               ] [VzDvGr5] starting ...
elasticsearch_1  | [2016-12-01T21:51:42,204][INFO ][o.e.t.TransportService   ] [VzDvGr5] publish_address {172.17.0.7:9300}, bound_addresses {[::]:9300}
elasticsearch_1  | [2016-12-01T21:51:42,278][INFO ][o.e.b.BootstrapCheck     ] [VzDvGr5] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
elasticsearch_1  | ERROR: bootstrap checks failed
elasticsearch_1  | max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
elasticsearch_1  | [2016-12-01T21:51:42,366][INFO ][o.e.n.Node               ] [VzDvGr5] stopping ...
elasticsearch_1  | [2016-12-01T21:51:42,816][INFO ][o.e.n.Node               ] [VzDvGr5] stopped
elasticsearch_1  | [2016-12-01T21:51:42,830][INFO ][o.e.n.Node               ] [VzDvGr5] closing ...
elasticsearch_1  | [2016-12-01T21:51:42,974][INFO ][o.e.n.Node               ] [VzDvGr5] closed
warehouse_elasticsearch_1 exited with code 78

So now I probably need to figure out how to increase the max virtual_memory. It seem to be a know problem though I'm unsure how to fix it on my side.

@di
Copy link
Member

di commented Dec 3, 2016

@Carreau If you haven't seen it already, the thread at docker-library/elasticsearch#98 might help you.

A better alternative, if possible, might be switching from using VirtualBox as the host OS to using "Docker for Mac" instead: https://docs.docker.com/engine/installation/mac/

I'm going to close this issue for now because this isn't a problem with Warehouse per se, but please feel free to keep it updated if you have additional difficulties, or reopen if we need to update some documentation, etc!

@di di closed this as completed Dec 3, 2016
@Carreau
Copy link
Contributor Author

Carreau commented Dec 3, 2016

Thanks, yes I saw that issue, havent figured out exactly how to fix it on my system but its not preventing me to work for now. So no issue. Thanks.

@pradyunsg
Copy link
Contributor

Even I stumbled upon this.

From https://www.elastic.co/guide/en/elasticsearch/guide/current/_file_descriptors_and_mmap.html:

Ensure that you configure the maximum map count so that there is ample virtual memory available for mmapped files. This can be set temporarily:

sysctl -w vm.max_map_count=262144

Or you can set it permanently by modifying vm.max_map_count setting in your /etc/sysctl.conf.

@di maybe this can be added to the Getting Started page in the Warehouse docs?

@pradyunsg
Copy link
Contributor

Ignore that. I spoke prematurely.

@pradyunsg
Copy link
Contributor

Okay. So, make initdb was unsuccessful due to running out of memory. Moving a slider in the Docker For Mac to increase the memory to 4GB, it now works.

@di
Copy link
Member

di commented Jan 18, 2018

@pradyunsg Yes, this has been a recurring issue. I've created #2811 to document the memory requirements.

fitodic pushed a commit to fitodic/warehouse that referenced this issue Jul 28, 2018
miketheman referenced this issue in miketheman/warehouse Feb 23, 2022
The current settings for the elasticsearch container place the started
node in "production, single node cluster" mode.
This performs boostrap tests, one of which tests for virtual memory via
the `sysctl` setting of `vm.max_map_count`.

In certain Docker-in-Docker development environments, the end user may
not have access to the underlying host settings, and cannot modify this
setting.
An alternative is to set the `discovery-type` to a single node, placing
the service in "development mode", turning any bootstrap checks from
errors to warnings.

Refs: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/system-config.html#dev-vs-prod
Refs: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/bootstrap-checks.html

Originally reported in #1511, related to pypi#4370

Signed-off-by: Mike Fiedler <miketheman@gmail.com>
di pushed a commit that referenced this issue Feb 23, 2022
* chore(dev): set web to depend on db

When running `make tests`, the current expectation is that the user has
executed `make serve` and the database container is running in already.

This brings in all the other runtime dependencies, when all the tests
need is a running database instance.

Setting the `depends_on` flag tells docker-compose to start the db
container first, any time we run a `web` container, and saves us the
step of starting it ourselves in another shell.

Signed-off-by: Mike Fiedler <miketheman@gmail.com>

* chore(dev): set elasticsearch to single-node

The current settings for the elasticsearch container place the started
node in "production, single node cluster" mode.
This performs boostrap tests, one of which tests for virtual memory via
the `sysctl` setting of `vm.max_map_count`.

In certain Docker-in-Docker development environments, the end user may
not have access to the underlying host settings, and cannot modify this
setting.
An alternative is to set the `discovery-type` to a single node, placing
the service in "development mode", turning any bootstrap checks from
errors to warnings.

Refs: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/system-config.html#dev-vs-prod
Refs: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/bootstrap-checks.html

Originally reported in #1511, related to #4370

Signed-off-by: Mike Fiedler <miketheman@gmail.com>

* chore: enable context reporting in coverage

coverage.py version 5.0 introduced the notion of measurement contexts.
This adds details to the report to show:

- how many times a given line was executed
- which context executed a given line

Enabling dynamic context measurement during the test execution adds
test class/function contexts to the coverage data to be displayed on
a given report output.

Refs: https://coverage.readthedocs.io/en/6.3.2/contexts.html#dynamic-contexts

Signed-off-by: Mike Fiedler <miketheman@gmail.com>

* chore(dev): enable web port to be set via env var

Creating a port under 1024 often requires superuser privileges.
In some environments this is disallowed.
By using the built-in behavior from Docker Compose, a user can now set:

    WEB_PORT=8080 make serve

and the web application will be made availabel on port 8080.

The default remains port 80.

Signed-off-by: Mike Fiedler <miketheman@gmail.com>
domdfcoding pushed a commit to domdfcoding/warehouse that referenced this issue Jun 7, 2022
* chore(dev): set web to depend on db

When running `make tests`, the current expectation is that the user has
executed `make serve` and the database container is running in already.

This brings in all the other runtime dependencies, when all the tests
need is a running database instance.

Setting the `depends_on` flag tells docker-compose to start the db
container first, any time we run a `web` container, and saves us the
step of starting it ourselves in another shell.

Signed-off-by: Mike Fiedler <miketheman@gmail.com>

* chore(dev): set elasticsearch to single-node

The current settings for the elasticsearch container place the started
node in "production, single node cluster" mode.
This performs boostrap tests, one of which tests for virtual memory via
the `sysctl` setting of `vm.max_map_count`.

In certain Docker-in-Docker development environments, the end user may
not have access to the underlying host settings, and cannot modify this
setting.
An alternative is to set the `discovery-type` to a single node, placing
the service in "development mode", turning any bootstrap checks from
errors to warnings.

Refs: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/system-config.html#dev-vs-prod
Refs: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/bootstrap-checks.html

Originally reported in pypi#1511, related to pypi#4370

Signed-off-by: Mike Fiedler <miketheman@gmail.com>

* chore: enable context reporting in coverage

coverage.py version 5.0 introduced the notion of measurement contexts.
This adds details to the report to show:

- how many times a given line was executed
- which context executed a given line

Enabling dynamic context measurement during the test execution adds
test class/function contexts to the coverage data to be displayed on
a given report output.

Refs: https://coverage.readthedocs.io/en/6.3.2/contexts.html#dynamic-contexts

Signed-off-by: Mike Fiedler <miketheman@gmail.com>

* chore(dev): enable web port to be set via env var

Creating a port under 1024 often requires superuser privileges.
In some environments this is disallowed.
By using the built-in behavior from Docker Compose, a user can now set:

    WEB_PORT=8080 make serve

and the web application will be made availabel on port 8080.

The default remains port 80.

Signed-off-by: Mike Fiedler <miketheman@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants