Skip to content

Commit

Permalink
Use mamba under a feature flag to create conda environments
Browse files Browse the repository at this point in the history
`mamba` is a a fast drop-in replacement for the conda command-line
utility, in C++.

I'm adding a feature flag so we can test it out in selective projects
that are failing over and over again because of OOM when solving
dependencies, even if they have just one, but they are adding
conda-forge as channel.
  • Loading branch information
humitos committed Mar 24, 2020
1 parent 8f9cb19 commit 1a7d11b
Show file tree
Hide file tree
Showing 3 changed files with 55 additions and 4 deletions.
13 changes: 12 additions & 1 deletion docs/guides/feature-flags.rst
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,17 @@ In case you prefer to use the latest ``conda`` version available, this is the fl
Makes Read the Docs to install all the requirements at once on ``conda create`` step.
This helps users to pin dependencies on conda and to improve build time.

``CONDA_USES_MAMBA``: :featureflags:`CONDA_USES_MAMBA`

``conda`` solver consumes 1Gb minimum when installing any package using ``conda-forge`` channel.
This seems to be a known issue due conda forge has so many packages on it.
Using this feature flag allows you to use mamba_ instead of ``conda`` to create the environment
and install the dependencies.
``mamba`` is a drop-in replacement for conda that it's much faster and also
reduces considerably the amount of memory required to solve the dependencies.

.. _mamba: https://quantstack.net/mamba.html

``DONT_OVERWRITE_SPHINX_CONTEXT``: :featureflags:`DONT_OVERWRITE_SPHINX_CONTEXT`

``DONT_SHALLOW_CLONE``: :featureflags:`DONT_SHALLOW_CLONE`
Expand All @@ -35,4 +46,4 @@ e.g. python-reno release notes manager is known to do that

``USE_TESTING_BUILD_IMAGE``: :featureflags:`USE_TESTING_BUILD_IMAGE`

``EXTERNAL_VERSION_BUILD``: :featureflags:`EXTERNAL_VERSION_BUILD`
``EXTERNAL_VERSION_BUILD``: :featureflags:`EXTERNAL_VERSION_BUILD`
41 changes: 38 additions & 3 deletions readthedocs/doc_builder/python_environments.py
Original file line number Diff line number Diff line change
Expand Up @@ -438,6 +438,23 @@ class Conda(PythonEnvironment):
def venv_path(self):
return os.path.join(self.project.doc_path, 'conda', self.version.slug)

def conda_bin_name(self):
"""
Decide whether use ``mamba`` or ``conda`` to create the environment.
Return ``mamba`` if the project has ``CONDA_USES_MAMBA`` feature and
``conda`` otherwise. This will be the executable name to be used when
creating the conda environment.
``mamba`` is really fast to solve dependencies and download channel
metadata on startup.
See https://github.com/QuantStack/mamba
"""
if self.project.has_feature(Feature.CONDA_USES_MAMBA):
return 'mamba'
return 'conda'

def _update_conda_startup(self):
"""
Update ``conda`` before use it for the first time.
Expand All @@ -446,7 +463,7 @@ def _update_conda_startup(self):
independently the version of Miniconda that it has installed.
"""
self.build_env.run(
'conda',
self.conda_bin_name(),
'update',
'--yes',
'--quiet',
Expand All @@ -456,6 +473,18 @@ def _update_conda_startup(self):
cwd=self.checkout_path,
)

def _install_mamba(self):
self.build_env.run(
'conda',
'install',
'--yes',
'--quiet',
'--name=base',
'--channel=conda-forge',
'mamba',
cwd=self.checkout_path,
)

def setup_base(self):
conda_env_path = os.path.join(self.project.doc_path, 'conda')
version_path = os.path.join(conda_env_path, self.version.slug)
Expand All @@ -479,8 +508,11 @@ def setup_base(self):
self._append_core_requirements()
self._show_environment_yaml()

if self.project.has_feature(Feature.CONDA_USES_MAMBA):
self._install_mamba()

self.build_env.run(
'conda',
self.conda_bin_name(),
'env',
'create',
'--quiet',
Expand Down Expand Up @@ -566,6 +598,9 @@ def _get_core_requirements(self):
'pillow',
]

if self.project.has_feature(Feature.CONDA_USES_MAMBA):
conda_requirements.append('pip')

# Install pip-only things.
pip_requirements = [
'recommonmark',
Expand All @@ -592,7 +627,7 @@ def install_core_requirements(self):
# Install requirements via ``conda install`` command if they were
# not appended to the ``environment.yml`` file.
cmd = [
'conda',
self.conda_bin_name(),
'install',
'--yes',
'--quiet',
Expand Down
5 changes: 5 additions & 0 deletions readthedocs/projects/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -1511,6 +1511,7 @@ def add_features(sender, **kwargs):
EXTERNAL_VERSION_BUILD = 'external_version_build'
UPDATE_CONDA_STARTUP = 'update_conda_startup'
CONDA_APPEND_CORE_REQUIREMENTS = 'conda_append_core_requirements'
CONDA_USES_MAMBA = 'conda_uses_mamba'
ALL_VERSIONS_IN_HTML_CONTEXT = 'all_versions_in_html_context'
SKIP_SYNC_TAGS = 'skip_sync_tags'
SKIP_SYNC_BRANCHES = 'skip_sync_branches'
Expand Down Expand Up @@ -1566,6 +1567,10 @@ def add_features(sender, **kwargs):
CONDA_APPEND_CORE_REQUIREMENTS,
_('Append Read the Docs core requirements to environment.yml file'),
),
(
CONDA_USES_MAMBA,
_('Uses mamba binary instead of conda to create the environment'),
),
(
ALL_VERSIONS_IN_HTML_CONTEXT,
_(
Expand Down

0 comments on commit 1a7d11b

Please sign in to comment.