Optimizer.MplMonitor

Optimizer.MplMonitor#

Optimizer.MplMonitor.py

migration of FullOptDialog to Jupyter Notebook

class MplMonitor(source=None, function_code=None, clear_jobs=True, xr_only=False, debug=True)#

Bases: object

Interactive Jupyter notebook monitor for optimization processes with subprocess management.

This class provides a dashboard-based interface for running and monitoring optimization jobs in Jupyter notebooks. It manages background subprocess execution, provides real-time progress visualization, and implements robust recovery mechanisms to prevent losing control of running processes when notebook outputs are cleared.

The monitor tracks active processes through both an in-memory registry and a persistent file-based registry, allowing recovery from accidental notebook state loss.

Parameters:

  • function_code (str, optional) – Function code identifier for logging purposes.

  • clear_jobs (bool, default=True) – If True, clears existing job folders in the optimizer directory on initialization.

  • debug (bool, default=True) – If True, enables debug mode with module reloading for development.

optimizer_folder#

Path to the folder containing optimization outputs and logs.

Type:

str

logger#

Logger instance for recording monitor activities.

Type:

logging.Logger

runner#

Background process runner managing the subprocess execution.

Type:

BackRunner

dashboard#

The main dashboard widget containing plots and controls.

Type:

ipywidgets.VBox

process_id#

String representation of the current subprocess PID.

Type:

str

instance_id#

Unique identifier for this monitor instance.

Type:

int

Examples

Basic usage with automatic recovery:

from molass_legacy.Optimizer.MplMonitor import MplMonitor

# Create and configure monitor
monitor = MplMonitor(clear_jobs=True)
monitor.create_dashboard()

# Run optimization
monitor.run(optimizer, init_params, niter=20, max_trials=30)
monitor.show()
monitor.start_watching()

Recovering a lost dashboard after clearing notebook outputs:

# Retrieve the most recent active monitor
monitor = MplMonitor.get_active_monitor()
monitor.redisplay_dashboard()

Checking all active monitors:

# Display status of all running monitors
MplMonitor.show_active_monitors()

# Get all active instances
monitors = MplMonitor.get_all_active_monitors()

Cleaning up orphaned processes:

# Interactive cleanup of orphaned processes
MplMonitor.cleanup_orphaned_processes()

Notes

  • The monitor maintains two registries: an in-memory registry for quick access to active instances, and a file-based registry (active_processes.json) for subprocess tracking that persists across notebook sessions.

  • When creating a new monitor while others are active, a warning is displayed with instructions for recovery.

  • The dashboard includes real-time plot updates, status indicators, and control buttons for terminating jobs and exporting data.

  • Background processes are automatically cleaned up when the monitor detects they are orphaned or when the monitor instance is destroyed.

  • For optimal use in Jupyter notebooks, use start_watching() to run progress monitoring in a background thread, keeping the notebook interactive.

Note

Process registry and dashboard recovery features implemented with assistance from GitHub Copilot (January 2026).

See also

BackRunner

Manages subprocess execution for optimization jobs.

JobState

Tracks and parses optimization job state from callback files.

Initialise MplMonitor.

Parameters:

source (_SubprocessSource or _RunInfoSource, optional) – ProgressSource that determines who is alive/terminate/working_folder. When None (default) a _SubprocessSource backed by a new BackRunner is constructed — existing subprocess behaviour, unchanged.

classmethod cleanup_orphaned_processes(optimizer_folder=None)#

Class method to manually clean up orphaned processes.

This can be called from a fresh notebook cell without an instance:

MplMonitor.cleanup_orphaned_processes()

Parameters:

optimizer_folder – Path to optimizer folder. If None, uses default from settings.

classmethod cleanup_orphaned_threads()#

Stop watch threads for monitors that are no longer needed.

This method identifies and stops watch threads that are still running for monitors that may have lost their dashboard. Useful for cleaning up after accidentally clearing notebook outputs multiple times.

Example

MplMonitor.cleanup_orphaned_threads()

clear_jobs()#
create_dashboard()#
export_data(b, debug=True)#
classmethod for_run_info(run_info, *, niter=20, max_trials=0, function_code=None, clear_jobs=False)#

Create a MplMonitor that watches an in-process RunInfo.

Use this after optimize_rigorously(in_process=True, async_=True) to get a live dashboard without a subprocess:

run_info = decomp.optimize_rigorously(rgcurve, method='BH',
                                       niter=20, async_=True)
mon = MplMonitor.for_run_info(run_info, niter=20)
mon.create_dashboard()
mon.show()
mon.start_watching()

Both the Resume and Terminate buttons are shown. Resume re-runs from the best accepted parameters via a new in-process thread (uses _RunInfoSource.run()). Terminate triggers cooperative stop via request_stop() → ctypes KeyboardInterrupt injection into the solver thread. The Export button remains available once the run completes.

Parameters:

  • run_info (molass.Rigorous.RunInfo.RunInfo) – The RunInfo object returned by optimize_rigorously(async_=True).

  • niter (int, default=20) – Number of optimizer iterations — must match the value passed to optimize_rigorously(). Used to scale the SV-history axis.

  • max_trials (int, default=0) –

    Maximum number of automatic sequential re-trials after each trial completes. Default 0 means no automatic re-start — the user decides manually via the Resume button.

    Why the default differs from the subprocess path (which defaults to 30): the subprocess runs detached and needs to loop autonomously, while the in-process path runs inside the notebook where the user is watching. Pausing after each trial lets the user inspect the SV and curves before deciding to continue. The technical limitation that once prevented auto-resume (threads could not be killed or restarted) was lifted in v0.6.0 (_RunInfoSource.run() + cooperative stop). The default of 0 is now a deliberate UX choice, not a technical constraint. Set max_trials=30 to match the subprocess default for unattended runs.

  • function_code (str, optional) – Forwarded to __init__.

  • clear_jobs (bool, default=False) – Set True to clear existing job folders. Defaults to False because an in-process run has already created its working folder.

classmethod for_subprocess(*, xr_only=False, function_code=None, clear_jobs=True, debug=True)#

Create a MplMonitor backed by a new BackRunner subprocess.

This is the canonical way to create a subprocess-mode monitor. It is equivalent to the pre-Phase-2 MplMonitor() constructor and preserves all existing behaviour byte-for-byte.

Parameters:

  • xr_only (bool, default=False) – Pass True for X-ray-only (no UV) optimization.

  • function_code – Forwarded to __init__.

  • clear_jobs – Forwarded to __init__.

  • debug – Forwarded to __init__.

classmethod get_active_monitor()#

Get the most recently created active monitor instance.

Returns the last MplMonitor instance that was created and is still active. Useful for recovering access to a monitor after clearing notebook outputs.

Returns:

The most recent active monitor, or None if no monitors exist.

Return type:

MplMonitor

Example

# After clearing outputs: monitor = MplMonitor.get_active_monitor() if monitor:

monitor.redisplay_dashboard()

classmethod get_all_active_monitors()#

Get all active monitor instances.

Returns:

List of all active MplMonitor instances.

Return type:

list

get_best_params(plot_info=None)#
get_current_curves()#

Return the data and model curves currently shown on the monitor.

This is the monitor readability API (molass-legacy issue #31): it exposes as plain numpy arrays the same curves that are rendered in the matplotlib dashboard, so that an AI agent can reason from the same evidence as a human looking at the screen.

Returns:

  • dict with keys

  • xr_frames – 1-D array — XR frame indices (x-axis for all XR panels).

  • xr_data – 1-D array — XR data elution curve (observed total intensity per frame).

  • xr_model – 1-D array — XR model total (sum of all components including baseline).

  • xr_components – 2-D array (n_components × n_frames) — individual XR component curves.

  • uv_frames – 1-D array — UV frame indices (mapped to XR frame scale via linear mapping a·x+b).

  • uv_data – 1-D array — UV data elution curve.

  • uv_model – 1-D array — UV model total.

  • uv_components – 2-D array (n_components × n_frames) — individual UV component curves.

  • sv_history – 1-D array — SV values at each accepted optimization evaluation.

  • best_sv – float — best SV seen so far (None if not available).

  • params – 1-D array — current best optimizer parameters.

  • Returns None if the monitor has no data yet (job_state not initialised).

Example

>>> state = monitor.get_current_curves()
>>> if state:
...     peak_data  = state['uv_frames'][np.argmax(state['uv_data'])]
...     peak_model = state['uv_frames'][np.argmax(state['uv_model'])]
...     print(f"UV data peak: {peak_data}, model peak: {peak_model}, shift: {peak_model - peak_data:+.1f}")
get_progress_info()#

Return current optimization progress as a dictionary.

This method exposes the same timing and score information that is rendered visually in the matplotlib progress chart, making it accessible to AI agents and programmatic callers.

Returns:

Keys: - status (str): Current status label text. - trial (int): Current trial number (0-based). - max_trials (int): Maximum number of trials. - num_evals (int): Number of function evaluations so far. - best_fv (float or None): Best objective function value. - starting_time (str): When the job started (HH:MM format). - time_elapsed (str): Time since start (H.MM format). - ending_time (str): Estimated completion time (HH:MM format). - time_ahead (str): Estimated remaining time (H.MM format). - is_running (bool): Whether the subprocess is still alive.

Return type:

dict

get_status_summary()#

Print a concise one-line progress summary.

Intended for AI agents and programmatic callers who cannot see the matplotlib widget. Returns the summary string as well.

Returns:

A single-line summary of the current optimization state.

Return type:

str

is_watching()#

Check if the watch thread is currently active.

Returns:

True if watch thread is running, False otherwise.

Return type:

bool

redisplay_dashboard()#

Redisplay the dashboard after it has been cleared.

This method allows you to reconnect to a running monitor after accidentally clearing notebook outputs. Call this to get the dashboard back.

Example

# After clearing outputs, retrieve and redisplay: monitor = MplMonitor.get_active_monitor() monitor.redisplay_dashboard()

run(optimizer, init_params, niter=20, seed=1234, max_trials=30, work_folder=None, dummy=False, x_shifts=None, debug=False, devel=True)#
run_impl(optimizer, init_params, niter=20, seed=1234, work_folder=None, dummy=False, optimizer_test=False, debug=False, devel=False)#
save_the_result_figure(fig_file=None)#
show(debug=False)#
classmethod show_active_monitors()#

Display information about all active monitor instances.

Shows a summary of all currently active monitors including their status, process ID, and working folder if available.

Example

MplMonitor.show_active_monitors()

start_watching()#

Start the background thread that monitors optimization progress.

Only one watch thread can be active per monitor instance. If a thread is already running, this method will log a warning and return without starting a new thread.

classmethod stop_all_threads(force=False)#

Stop all watch threads immediately without asking.

Use this to quickly stop all monitoring threads, for example when experiencing periodic screen blackouts after killing processes.

Parameters:

force – If True, doesn’t wait for graceful shutdown

Example

# Quick fix for blackout issue MplMonitor.stop_all_threads()

stop_watching(timeout=5.0)#

Stop the background watch thread gracefully.

Parameters:

timeout – Maximum time in seconds to wait for thread to stop.

Returns:

True if thread stopped successfully, False if timeout occurred.

Return type:

bool

terminate(timeout=5.0)#

Terminate the optimization run and clean up.

This is the recommended way to stop an optimization from code (e.g. in a notebook cell after get_current_decomposition).

It kills the subprocess directly, stops the watch thread, and removes the process from the registry.

Parameters:

timeout – Maximum seconds to wait for the watch thread to stop.

Returns:

True if shutdown completed within the timeout.

Return type:

bool

test_subprocess_optimizer()#
trigger_resume(b)#

Resume optimization from the best parameters of the completed job.

Called when the user clicks the ‘Resume Job’ button. Reads the best parameters from the latest callback.txt, launches a new subprocess, and restarts the watch thread.

trigger_terminate(b)#
update_plot()#
watch_progress(interval=1.0)#

Main watching loop that monitors subprocess and updates dashboard.

This runs in a background thread and can be stopped gracefully via stop_watch_event.

property working_folder#

Path to the current optimizer working folder, or None.