Note

Go to the end to download the full example code.

Using `BaseConfig`#

The BaseConfig class in caf.toolkit is designed to load and validate YAML [1] configuration files. This example shows how to create child classes of BaseConfig to load parameters.

Imports & Setup#

When using BaseConfig, the key package required, other than caf.toolkit itself, is pydantic as this provides additional validation functionality. Pydantic is already a dependency of caf.toolkit.

# Built-Ins
import pathlib
from typing import Any, Self

# Third Party
import pydantic
from pydantic import dataclasses

# Local Imports
import caf.toolkit as ctk

Path to the folder containing this example file.

folder = pathlib.Path().parent

Basic#

This example shows how to create a simple configuration file with different types of parameters. The majority of Python built-in types can be used, additionally dataclasses and many “simple” [2] custom types can be used.

Nesting#

More complex configuration files can be handled using dataclasses, pydantic.BaseModel subclasses or BaseConfig subclasses.

Bounds is a simple dataclass containing 4 floats.

@dataclasses.dataclass
class Bounds:
    """Bounding box coordinates."""

    min_x: float
    min_y: float
    max_x: float
    max_y: float

InputFile is an example of a dataclass which contains another dataclass as an attribute.

@dataclasses.dataclass
class InputFile:
    """Example of an input file dataclass with some additional information."""

    name: str
    extent: Bounds
    path: pydantic.FilePath

NestingConfig is an example of a configuration file with a list of nested dataclasses.

class NestingConfig(ctk.BaseConfig):
    output_folder: pydantic.DirectoryPath
    model_run: str
    inputs: list[InputFile]

Example of the YAML config file which is loaded by the above class.

Config file: examples/nested_config.yml#

output_folder: ../examples
model_run: test

# Example of a list containing any number of InputFile objects
inputs:
  # Individual parameters for the dataclass InputFile
  - name: first file
    path: ../examples/nested_config.yml
    extent:
      min_x: 100
      min_y: 50
      max_x: 200
      max_y: 150

  - name: second file
    path: ../examples/nested_config.yml
    extent:
      min_x: 10
      min_y: 5
      max_x: 20
      max_y: 10

Below shows how to load the config file and displays the class as text.

parameters = NestingConfig.load_yaml(folder / "nested_config.yml")
print(parameters)

output_folder=PosixPath('../examples') model_run='test' inputs=[InputFile(name='first file', extent=Bounds(min_x=100.0, min_y=50.0, max_x=200.0, max_y=150.0), path=PosixPath('../examples/nested_config.yml')), InputFile(name='second file', extent=Bounds(min_x=10.0, min_y=5.0, max_x=20.0, max_y=10.0), path=PosixPath('../examples/nested_config.yml'))]

Extra Validation#

Pydantic provides some functionality for adding additional validation to subclasses of pydantic.BaseModel (or pydantic dataclasses), which BaseConfig is based on.

The simplest approach to pydantic’s validation is using pydantic.Field to add some additional validation.

@dataclasses.dataclass
class FieldValidated:
    """Example of addtion attribute validation with Field class."""

    # Numbers with restrictions
    positive: int = pydantic.Field(gt=0)
    small_number: float = pydantic.Field(ge=0, le=1)
    even: int = pydantic.Field(multiple_of=2)

    # Text restrictions
    short_text: str = pydantic.Field(max_length=10)
    # Regular expression pattern only allowing lowercase letters
    regex_text: str = pydantic.Field(pattern=r"^[a-z]+$")

    # Iterable restrictions e.g. lists and tuples
    short_list: list[int] = pydantic.Field(min_length=2, max_length=5)

For more complex validation pydantic allow’s custom methods to be defined to validate individual fields (pydantic.field_validator()), or the class as a whole (pydantic.model_validator()). [3]

CustomFieldValidated gives an example of using the pydantic.field_validator() decorator to validate the whole class.

@dataclasses.dataclass
class CustomFieldValidated:
    """Example of using pydantics field validator decorator."""

    sorted_list: list[int]
    flexible_list: list[int]

    @pydantic.field_validator("sorted_list")
    @classmethod
    def validate_order(cls, value: list[int]) -> list[int]:
        """Validate the list is sorted."""
        previous = None
        for i, val in enumerate(value):
            if previous is not None and val < previous:
                raise ValueError(f"item {i} ({val}) is smaller than previous ({previous})")

            previous = val

        return value

    # This validation method is ran before pydantic does any validation
    @pydantic.field_validator("flexible_list", mode="before")
    @classmethod
    def csv_list(cls, value: Any) -> list:
        """Split text into list based on commas.."""
        if isinstance(value, str):
            return value.split(",")
        return value

ModelValidated gives an example of using the pydantic.model_validator() decorator to validate the whole class.

@dataclasses.dataclass
class ModelValidated:
    """Example of using pydantics model validator decorator."""

    possible_values: list[str]
    favourite_value: str

    @pydantic.model_validator(mode="after")
    def check_favourite(self) -> Self:
        """Checks if favourite value is in the list of possible values."""
        if self.favourite_value not in self.possible_values:
            raise ValueError(
                f"favourite value ({self.favourite_value})"
                " isn't found in list of possible values"
            )

        return self

ExtraValidatedConfig includes the additional validation methods discussed in the classes above in a config class.

class ExtraValidatedConfig(ctk.BaseConfig):
    """Config class showing examples of custom validation."""

    simple_validated: FieldValidated
    custom_validated: CustomFieldValidated
    fully_validated: ModelValidated

Example of the YAML config file which is loaded by the above class.

Config file: examples/validated_config.yml#

simple_validated:
  # Numbers with restrictions
  positive: 12
  small_number: 0.5
  even: 16

  # Max length 10 characters
  short_text: short text
  # Only allowing lowercase letters, without spaces or punctuation
  regex_text: valid

  # List with 2 - 5 items
  short_list:
    - 1
    - 2
    - 3

custom_validated:
  # List provided in sorted order
  sorted_list:
    - 1
    - 2
    - 3
  # List can be passed as CSV
  flexible_list: 1, 2, 3, 4

fully_validated:
  possible_values:
    - apple
    - pear
    - pineapple

  # Value must be in list above
  favourite_value: apple

Below shows how to load the config file and displays the class as text.

parameters = ExtraValidatedConfig.load_yaml(folder / "validated_config.yml")
print(parameters)

simple_validated=FieldValidated(positive=12, small_number=0.5, even=16, short_text='short text', regex_text='valid', short_list=[1, 2, 3]) custom_validated=CustomFieldValidated(sorted_list=[1, 2, 3], flexible_list=[1, 2, 3, 4]) fully_validated=ModelValidated(possible_values=['apple', 'pear', 'pineapple'], favourite_value='apple')

Footnotes

Total running time of the script: (0 minutes 0.025 seconds)

Gallery generated by Sphinx-Gallery

Using BaseConfig#

Imports & Setup#

Basic#

Nesting#

Extra Validation#

Using `BaseConfig`#