Skip to content

Commit

Permalink
FIX-#7102: Remove enable_api_only mode in modin logging (#7194)
Browse files Browse the repository at this point in the history
Signed-off-by: arunjose696 <arunjose696@gmail.com>
  • Loading branch information
arunjose696 authored Apr 17, 2024
1 parent de6114c commit 615f975
Show file tree
Hide file tree
Showing 3 changed files with 14 additions and 37 deletions.
15 changes: 2 additions & 13 deletions docs/usage_guide/advanced_usage/modin_logging.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,7 @@ Modin Logging

Modin logging offers users greater insight into their queries by logging internal Modin API calls, partition metadata,
and profiling system memory. When Modin logging is enabled (default disabled), log files are written to a local ``.modin`` directory at the same
directory level as the notebook/script used to run Modin. It is possible to configure whether to log system memory and additional metadata
in addition to Modin API calls (see the usage examples below).
directory level as the notebook/script used to run Modin.

The logs generated by Modin Logging will be written to a ``.modin/logs/job_<uuid>`` directory, uniquely named after the job uuid.
The logs that contain the Modin API stack traces are named ``trace.log``. The logs that contain the memory utilization metrics are
Expand All @@ -20,17 +19,7 @@ such environments. To resolve this, please run Jupyterlab or other similar servi
Usage examples
--------------

In the example below, we enable logging for internal Modin API calls.

.. code-block:: python
import modin.pandas as pd
from modin.config import LogMode
LogMode.enable_api_only()
# User code goes here
In the next example, we add logging for not only internal Modin API calls, but also for partition metadata and memory profiling.
In the example below, we enable logging for internal Modin API calls, partition metadata and memory profiling.
We can set the granularity (in seconds) at which the system memory utilization is logged using ``LogMemoryInterval``.
We can also set the maximum size of the logs (in MBs) using ``LogFileSize``.

Expand Down
11 changes: 1 addition & 10 deletions modin/config/envvars.py
Original file line number Diff line number Diff line change
Expand Up @@ -501,7 +501,7 @@ class LogMode(EnvironmentVariable, type=ExactStr):
"""Set ``LogMode`` value if users want to opt-in."""

varname = "MODIN_LOG_MODE"
choices = ("enable", "disable", "enable_api_only")
choices = ("enable", "disable")
default = "disable"

@classmethod
Expand All @@ -514,15 +514,6 @@ def disable(cls) -> None:
"""Disable logging feature."""
cls.put("disable")

@classmethod
def enable_api_only(cls) -> None:
"""Enable API level logging."""
warnings.warn(
"'enable_api_only' value for LogMode is deprecated and"
+ "will be removed in a future version."
)
cls.put("enable_api_only")


class LogMemoryInterval(EnvironmentVariable, type=int):
"""Interval (in seconds) to profile memory utilization for logging."""
Expand Down
25 changes: 11 additions & 14 deletions modin/logging/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ def configure_logging() -> None:
"modin.logger.default",
job_id,
"trace",
LogLevel.INFO if LogMode.get() == "enable_api_only" else LogLevel.DEBUG,
LogLevel.INFO,
)

logger.info(f"OS Version: {platform.platform()}")
Expand All @@ -181,20 +181,17 @@ def configure_logging() -> None:
logger.info(f"Physical Cores: {num_physical_cores}")
logger.info(f"Total Cores: {num_total_cores}")

if LogMode.get() != "enable_api_only":
mem_sleep = LogMemoryInterval.get()
mem_logger = _create_logger(
"modin_memory.logger", job_id, "memory", LogLevel.DEBUG
)
mem_sleep = LogMemoryInterval.get()
mem_logger = _create_logger("modin_memory.logger", job_id, "memory", LogLevel.DEBUG)

svmem = psutil.virtual_memory()
mem_logger.info(f"Memory Total: {bytes_int_to_str(svmem.total)}")
mem_logger.info(f"Memory Available: {bytes_int_to_str(svmem.available)}")
mem_logger.info(f"Memory Used: {bytes_int_to_str(svmem.used)}")
mem = threading.Thread(
target=memory_thread, args=[mem_logger, mem_sleep], daemon=True
)
mem.start()
svmem = psutil.virtual_memory()
mem_logger.info(f"Memory Total: {bytes_int_to_str(svmem.total)}")
mem_logger.info(f"Memory Available: {bytes_int_to_str(svmem.available)}")
mem_logger.info(f"Memory Used: {bytes_int_to_str(svmem.used)}")
mem = threading.Thread(
target=memory_thread, args=[mem_logger, mem_sleep], daemon=True
)
mem.start()

_create_logger("modin.logger.errors", job_id, "error", LogLevel.INFO)

Expand Down

0 comments on commit 615f975

Please sign in to comment.