Dynamically loading modules in Python 3

Being able to programatically load python files as modules is a pretty cool capability of Python.

I'll jump to the punch, then step back to fill in context.

First, here's the file that dynamically loads peer python files as modules:

# File: dynamically_loaded_modules/__init__.py

import glob
import importlib.util
import sys

from os import path


def _get_module_name_from_file_path(file_path, module_name_prefix=''):
    if not file_path.endswith('.py'):
        raise ValueError(f"File doesn't have a '.py' extension: {file_path}")

    file_name = path.basename(file_path)
    file_name_without_ext = file_name[:-3]

    if module_name_prefix and not module_name_prefix.endswith('.'):
      module_name_prefix += '.'
    return module_name_prefix + file_name_without_ext


def load_modules(registry):
    path_string = path.join(path.dirname(__file__), '*.py')
    module_paths = [
        file_name for file_name in glob.glob(path_string)
        if not file_name.endswith('__init__.py')
    ]

    for module_path in module_paths:
        module_name = _get_module_name_from_file_path(module_path)

        if module_name not in sys.modules:
            module_spec = importlib.util.spec_from_file_location(module_name, module_path)
            sys.modules[module_name] = importlib.util.module_from_spec(module_spec)
            module_spec.loader.exec_module(sys.modules[module_name])

        module = sys.modules[module_name]
        if not hasattr(module, 'load_module') or not callable(module.load_module):
            raise Exception(f'Auto-load module {module} is missing required function "load_module"')

        module.load_module(registry)

Next, here's an example of a peer python file that gets dynamically loaded as a module:

# File: dynamically_loaded_modules/example.py

def load_module(registry):
    registry.register(f'{__file__} was dynamically loaded!')

Finally, here's example usage:

#!/usr/bin/env python

# File: app.py

from dynamically_loaded_modules import load_modules


class Registry:
    def __init__(self):
        self._registrations = []

    def register(self, description):
        self._registrations.append(description)

    @property
    def registrations(self):
        return self._registrations


registry = Registry()
load_modules(registry)
print(registry.registrations)

The use case I had for this was dynamically registering API routes for a flask app.

This solution offers the conenvince of registering new API routes via Flask's route. That's achieved by (1) passing the Flask app to load_modules and (2) calling flask_app.route in each dynamically loaded file's load_module function.

Actually, I take back what I said earlier. I'm not going to fill in too much context right now. Otherwise, I'd never publish this. :)

Instead, I'll try to come back later and update this post to break down the code more.