PyPads application class

This class represents the app. To start and activate the tracking of your modules, classes and functions, the app class has to be instantiated.

Warning

It is recommended to initialize the tracking before importing the modules to be tracked. While extending the importlib and reloading the modules may work sometimes doing so may result in unforeseen issues.

class pypads.app.base.PyPads(uri=None, folder=None, mappings: List[pypads.importext.mappings.MappingCollection] = None, hooks=None, events=None, setup_fns=None, config=None, pre_initialized_cache: pypads.app.misc.caches.PypadsCache = None, disable_plugins=None, autostart=None)

PyPads app. Serves as the main entrypoint to PyPads. After constructing this app tracking is activated.

activate_tracking(reload_modules=False, reload_warnings=True, clear_imports=False, affected_modules=None)

Function to duck punch all objects defined in the mapping files. This should at best be called before importing any libraries. :param affected_modules: Affected modules of the mapping files. :param clear_imports: Clear imports after punching. CAREFUL THIS IS EXPERIMENTAL! :param reload_warnings: Show warnings of affected modules which were already imported before the importlib was extended. :param reload_modules: Force a reload of affected modules. CAREFUL THIS IS EXPERIMENTAL! :return:

actuators

Access the actuators of pypads. :return: PyPadsActuators

add_atexit_fn(fn)

Add function to be executed before stopping your process. This function is also added to pypads and errors are caught to not impact the experiment itself. Deactivating pypads should be able to run some of the atExit fns declared for pypads.

api

Access the api of pypads. :return: PyPadsAPI

static available_actuators()

Return a list of available actuators in the current installation of PyPads. :return: Actuator classes

static available_anchors()

Return a list of available anchors in the current installation of PyPads. Anchors are names for hooks. :return: Anchors

static available_cmds()

Return a list of available api commands in the current installation of PyPads. :return: API commands classes

static available_decorators()

Return a list of available decorators in the current installation of PyPads. :return: Decorator classes

static available_events()

Return a list of available events (event types) in the current installation of PyPads. Events are triggered by hooks and map to functions. :return: Event types

static available_loggers()

Return a list of available LoggingFunctions for mappings in the current installation of PyPads. :return: Logging function classes

static available_setup_functions()

Return a list of available setup function in the current installation of PyPads. :return: Setup function classes

static available_teardown_functions()

Return a list of available setup function in the current installation of PyPads. :return: Teardown function classes

static available_validators()

Return a list of available validators in the current installation of PyPads. :return: Validator classes

backend

Return the backend of PyPads. This is currently hardcoded to mlflow. :return: Backend

cache

Return the cache of pypads. This holds generic cache data and run cache data. :return:

call_tracker

Return the call tracker of PyPads. This is used to keep track of the calls of hooked functions. :return: CallTracker

config

Return the configuration of pypads. :return: Configuration dict

deactivate_tracking(run_atexits=False, reload_modules=True)

Deacticate the current tracking and cleanup. :param run_atexits: Run the registered atexit functions of pypads :param reload_modules: Force a reload of affected modules :return:

decorators

Access the decorators of pypads. :return: PyPadsDecorators

events

Get the to the constructor passed event / logging function mappings. :return: Hook configuration.

folder

Return the local folder pypads uses for its temporary storage, configuration etc. :return:

function_registry

Access the function registry holding all mappings of events to logging functions :return: Function registry

hook_registry

Access the hook registry holding all mappings of anchors/hooks to events :return: Hook registry

hooks

Get the to the constructor passed hook / event mappings. :return: Hook configuration.

is_affected_module(name, affected_modules=None)

Check if a given module name is in the list of affected modules. You can pass a list of affected modules or take the one of the wrap_manager. :param name: Name of the module to check :param affected_modules: The list of affected modules (can be None and will be extracted automatically) :return:

managed_git_factory

Return Git manager of PyPads. This used to create and manages git repositories. :return: ManagedGitFactory

mapping_registry

Access the mapping registry holding all references to mapping files etc. :return: Mapping registry

mappings

Get the to the constructor passed mappings. :return: Additional mapping files.

mlf

Return a mlflow client to interact with stored data. :return: MlflowClient

start_track(experiment_name=None, disable_run_init=False)

Start a new run to track. :param experiment_name: The name of the mlflow experiment :param disable_run_init: Flag to indicate if the run_init functions are to be run on an already existing run. :return:

uri

Return the tracking uri pypads uses for mlflow. :return:

validators

Access the validators of pypads. :return: PyPadsValidators

wrap_manager

Return the wrap manager of PyPads. This is used to wrap modules, functions and classes. :return: WrapManager

Default settings

The app includes default values for configuration, hook/event mappings, event/function mappings etc.

Default Anchors

Anchors are names for repeating types of hooks. Fit functions for example are existing on multiple libraries.

DEFAULT_ANCHORS = [Anchor("pypads_init", "Used if a tracked concept is initialized."),
                   Anchor("pypads_fit", "Used if an model is fitted to data."),
                   Anchor("pypads_predict", "Used if an model predicts something."),
                   Anchor("pypads_metric", "Used if an metric is compiled."),
                   Anchor("pypads_log", "Used to only log a call.")]

Default Event Types

Event types represent strategies to react to an anchor / hook.

DEFAULT_EVENT_TYPES = [EventType("parameters", "Track the parameters for given model."),
                       EventType("output", "Track the output of the function."),
                       EventType("input", "Track the input of the function."),
                       EventType("hardware", "Track current hardware load on function execution."),
                       EventType("metric", "Track a metric."),
                       EventType("autolog", "Activate mlflow autologging."),
                       EventType("pipeline", "Track a pipeline step."),
                       EventType("log", "Log the call to console."),
                       EventType("init", "Log the tracked class init to console.")]

Default Config

The configuration for PyPads

DEFAULT_CONFIG = {
    "track_sub_processes": False,  # Activate to track spawned subprocesses by extending the joblib
    "recursion_identity": False, # Activate to ignore tracking on recursive calls of the same function with the same mapping
    "recursion_depth": -1,  # Limit the tracking of recursive calls
    "log_on_failure": True,  # Log the stdout / stderr output when the execution of the experiment failed
    "include_default_mappings": True  # Include the default mappings additionally to the passed mapping if a mapping is passed
}

Default Hook Mapping

The hook mapping, maps hooks (anchors) to the events (event types).

DEFAULT_HOOK_MAPPING = {
    "init": {"on": ["pypads_init"]},
    "parameters": {"on": ["pypads_fit"]},
    "hardware": {"on": ["pypads_fit"]},
    "output": {"on": ["pypads_fit", "pypads_predict"]},
    "input": {"on": ["pypads_fit"], "with": {"_pypads_write_format": WriteFormats.text.name}},
    "metric": {"on": ["pypads_metric"]},
    "pipeline": {"on": ["pypads_fit", "pypads_predict", "pypads_transform", "pypads_metric"]},
    "log": {"on": ["pypads_log"]}
}

Default Event Mapping

Defines which logging functions should be run for events.

DEFAULT_LOGGING_FNS = {
    "parameters": Parameters(),
    "output": Output(_pypads_write_format=WriteFormats.text.name),
    "input": Input(_pypads_write_format=WriteFormats.text.name),
    "hardware": [Cpu(), Ram(), Disk()],
    "metric": Metric(),
    "autolog": MlflowAutologger(),
    "pipeline": PipelineTracker(_pypads_pipeline_type="normal", _pypads_pipeline_args=False),
    "log": Log(),
    "init": LogInit()
}