xtalmet.evaluator module

This module contains the Evaluator class for VSUN calculation.

class xtalmet.evaluator.Evaluator(validity: list[str] | None = None, stability: Literal['continuous', 'binary', None] = None, uniqueness: bool = False, novelty: bool = False, distance: str | None = None, ref_xtals: list[Crystal | Structure] | Literal['mp20'] | str | None = None, agg_func: Literal['prod', 'ave'] = 'prod', weights: dict[str, float] | None = None, multiprocessing: bool = False, n_processes: int | None = None, **kwargs)View on GitHub

Bases: object

Class for evaluating a set of crystals.

The evaluation is based on a chosen combination of validity (V), stability (S), uniqueness (U), and novelty (N).

Initialize the Evaluator.

Parameters:

  • validity (list[str] | None) – Approaches to evaluating validity. The currently supported methods are shown in SUPPORTED_VALIDITY in constants.py. If set to None, validity is not evaluated. Default is None.

  • stability (Literal["continuous", "binary", None]) – Stability evaluation method. “continuous” or “binary” or None. If set to None, stability is not evaluated. Default is None.

  • uniqueness (bool) – Whether to evaluate uniqueness. Default is False.

  • novelty (bool) – Whether to evaluate novelty. Default is False.

  • distance (str | None) – Distance metric used for uniqueness and novelty evaluation. The currently supported distances are listed in SUPPORTED_DISTANCES in constants.py. For more detailed information about each distance metric, please refer to the tutorial notebook. If both uniqueness and novelty are False, this argument is ignored. Default is None.

  • ref_xtals (list[Crystal | Structure] | Literal["mp20"] | str | None) – Reference crystal structures (typically a training set) for novelty evaluation. This can be a list of crystal structures, dataset name, or a path to the file containing the pre-computed embeddings of the reference structures. If a dataset name is given, its training data will be downloaded from a fixed Hugging Face revision. MP20 precomputed embeddings are used only for default embedding settings; other settings emit UserWarning and compute references from MP20 training structures during initialization. File inputs require metadata written by save_reference_embeddings() with matching distance and args_emb. If novelty is False, this argument is ignored. Default is None.

  • agg_func (Literal["prod", "ave"]) – Aggregation function for combining V, S, U, and N. “prod” means multiplication, and “ave” means (weighted) average. Default is “prod”.

  • weights (dict[str, float] | None) – Weights for V, S, U, and N. For “ave”, weights are coefficients (normalized internally; equal weights if None). Enabled components must have a positive total weight; weights for disabled components do not contribute to this total. For “prod”, weights are exponents (not normalized; default 1.0 per active metric if None). All supplied weights must be finite and non-negative. The enabled weight sum for “ave” must also be finite. Default is None.

  • multiprocessing (bool) – Whether to use multiprocessing for computing the embeddings of reference crystals. This argument is only effective when novelty is True and ref_xtals is list[Crystal | Structure] or MP20 with non-default embedding settings. Default is False.

  • n_processes (int | None) – Maximum number of processes to use for multiprocessing. If None, the number of logical CPU cores - 1 will be used. We recommend setting this argument to a smaller number than the number of available CPU cores to avoid out-of-memory. If multiprocessing is False, this argument is ignored. Default is None.

  • **kwargs – Additional keyword arguments. It can contain four keys: “args_validity”, “args_stability”, “args_emb”, and “args_dist”. “args_validity” is for the validity evaluation, while “args_stability” is for the stability evaluation. “args_emb” and “args_dist” are for the distance metric used in uniqueness and novelty evaluation: The former is for the embedding calculation, and the latter is for the distance matrix calculation between embeddings. For more details, please refer to the tutorial notebook.

Raises:

ValueError – If novelty has an empty reference set or no valid reference embeddings, a reference file has missing or incompatible metadata, or average aggregation has zero total enabled weight.

Examples

>>> # Evaluator for the conventional SUN metric against the MP20 dataset
>>> # using the StructureMatcher distance.
>>> evaluator = Evaluator(
...     stability="binary",
...             uniqueness=True,
...             novelty=True,
...             distance="smat",
...             ref_xtals="mp20",
...             agg_func="prod",
... )
>>> # Evaluator for the VSUN metric against a custom reference dataset using
>>> # the ElMD distance, with average aggregation.
>>> evaluator = Evaluator(
...     validity=["smact", "structure"],
...     stability="continuous",
...     uniqueness=True,
...     novelty=True,
...     distance="elmd",
...     ref_xtals=ref_xtals,  # list[Crystal | Structure]
...     agg_func="ave",
...     weights={
...         "validity": 0.25,
...         "stability": 0.25,
...         "uniqueness": 0.25,
...         "novelty": 0.25,
...     },
...     multiprocessing=True,
...     n_processes=10,
... )
>>> # Evaluator for the VSUN metric against the MP20 dataset using the AMD
>>> # distance, with custom kwargs.
>>> evaluator = Evaluator(
...     validity=["smact", "structure"],
...     stability="continuous",
...     uniqueness=True,
...     novelty=True,
...     distance="amd",
...     ref_xtals="mp20",
...     agg_func="prod",
...     {
...         "args_validity": {
...             "structure": {
...                 "threshold_distance": 0.5,
...                 "threshold_volume": 0.1,
...             }
...         },
...         "args_stability": {
...             "diagram": "mp_250618",
...             "mace_model": "medium-mpa-0",
...             "intercept": 0.4289,
...         },
...         "args_emb": {"k": 100},
...         "args_dist": {"metric": "chebyshev", "low_memory": False},
...     }
... )
>>> # Evaluator for the SUN metric against the MP20 dataset using the
>>> # ElMD+AMD distance, with custom kwargs.
>>> evaluator = Evaluator(
...     stability="continuous",
...     uniqueness=True,
...     novelty=True,
...     distance="elmd+amd",
...     ref_xtals="mp20",
...     agg_func="prod",
...     {
...         "args_validity": {
...             "structure": {
...                 "threshold_distance": 0.5,
...                 "threshold_volume": 0.1,
...             }
...         },
...         "args_stability": {
...             "diagram": "mp_250618",
...             "mace_model": "medium-mpa-0",
...             "intercept": 0.4289,
...         },
...         "args_emb": {"amd": {"k": 100}},
...         "args_dist": {
...                             "elmd": {"metric": "mod_petti"},
...                             "amd": {"metric": "chebyshev", "low_memory": False},
...                             "coefs": {
...                                     "elmd": float.fromhex("0x1.8d7d565a99f87p-1"),
...                                     "amd": float.fromhex("0x1.ca0aa695981e5p-3")},
...                             },
...                     }
...     }
... )
save_reference_embeddings(path: str | Path) → NoneView on GitHub

Save prepared references with the metadata required for later reuse.

Parameters:

path (str | Path) – Destination gzip-pickle file. Existing files are replaced.

Raises:

ValueError – If novelty is disabled or feature shapes contradict args_emb.

evaluate(xtals: list[Crystal | Structure], dir_intermediate: str | None = None, multiprocessing: bool = False, n_processes: int | None = None) → tuple[float, ndarray, dict[str, float]]View on GitHub

Evaluate the given crystal structures.

Failed embeddings (including nonfinite numerical features) have U/N scores of zero, without changing V/S. Binary U compares only earlier valid samples; N compares only valid references. Continuous U counts failed pairs as zero while retaining the total-sample denominator n - 1. Continuous N takes the minimum over valid references. Generated embeddings and their masks are prepared once per call and shared by U/N, even without a disk cache. Cached matrices still require embeddings; missing embeddings are recomputed. Nonfinite smat geometry is a failed embedding. Nonfinite continuous pair distances contribute zero to U, retaining n - 1; N uses only finite distances and is zero if no finite comparison remains for a generated sample.

Parameters:

  • xtals (list[Crystal | Structure]) – List of crystal structures to be evaluated.

  • dir_intermediate (str | None) – Directory to search for pre-computed intermediate results, such as validity scores, energy above hulls, embeddings, and distance matrices. Versioned records are stored under cache/format-1/<identity>/ within this directory. Reuse requires matching ordered inputs, relevant settings, references, model/phase data, and software identity. Old unversioned files are ignored and preserved. Invalid records are recomputed. If None, no files are loaded or saved. It is recommended to set this argument when evaluating the same large set of crystal structures multiple times, for example trying different aggregation functions. Different conditions retain separate artifacts. Default is None.

  • multiprocessing (bool) – Whether to use multiprocessing for embedding and distance matrix computation. This argument is only effective when uniqueness or novelty evaluation is enabled. Default is False.

  • n_processes (int | None) – Maximum number of processes to use for multiprocessing. If None, the number of logical CPU cores - 1 will be used. We recommend setting this argument to a smaller number than the number of available CPU cores to avoid out-of-memory. If multiprocessing is False, this argument is ignored. Default is None.

Returns:

A tuple containing the overall score (float), individual scores for each crystal structure, and a dictionary of computation times for each evaluation component.

Return type:

tuple[float, np.ndarray, dict[str, float]]

Raises:

ValueError – If xtals is empty, continuous uniqueness has fewer than two samples, novelty has no valid reference embeddings, or distance/args_emb were changed after preparing references.

Examples

>>> # Evaluate the conventional SUN metric using the StructureMatcher
>>> # distance ("smat") against the MP20 dataset.
>>> evaluator = Evaluator(
...     stability="binary",
...             uniqueness=True,
...             novelty=True,
...             distance="smat",
...             ref_xtals="mp20",
...             agg_func="prod",
... )
>>> evaluator.evaluate(
...     xtals=xtals,  # list[Crystal | Structure]
...     dir_intermediate="intermediate_results/",
...     multiprocessing=True,
...     n_processes=10,
... )
>>> 0.28, np.array([...]), {"aggregation": ..., ...}
>>> # Evaluate the VSUN metric using the ElMD distance against a custom
>>> # reference dataset, with average aggregation.
>>> evaluator = Evaluator(
...     validity=["smact", "structure"],
...     stability="continuous",
...     uniqueness=True,
...     novelty=True,
...     distance="elmd",
...     ref_xtals=ref_xtals,  # list[Crystal | Structure]
...     agg_func="ave",
...     weights={
...         "validity": 0.25,
...         "stability": 0.25,
...         "uniqueness": 0.25,
...         "novelty": 0.25,
...     },
...     multiprocessing=True,
...     n_processes=10,
... )
>>> evaluator.evaluate(
...     xtals=xtals,  # list[Crystal | Structure]
...     dir_intermediate=None,
...     multiprocessing=False,
...     n_processes=None,
... )
>>> 0.6119424269941065, np.array([...]), {"aggregation": ..., ...}
>>> # Evaluate the VSUN metric using the AMD distance against the MP20
>>> # dataset, with custom kwargs.
>>> evaluator = Evaluator(
...     validity=["smact", "structure"],
...     stability="continuous",
...     uniqueness=True,
...     novelty=True,
...     distance="amd",
...     ref_xtals="mp20",
...     agg_func="prod",
...     {
...         "args_validity": {
...             "structure": {
...                 "threshold_distance": 0.5,
...                 "threshold_volume": 0.1,
...             }
...         },
...         "args_stability": {
...             "diagram": "mp_250618",
...             "mace_model": "medium-mpa-0",
...             "intercept": 0.4289,
...         },
...         "args_emb": {"k": 100},
...         "args_dist": {"metric": "chebyshev", "low_memory": False},
...     }
... )
>>> evaluator.evaluate(
...     xtals=xtals,  # list[Crystal | Structure]
...     dir_intermediate="intermediate_results/",
...     multiprocessing=True,
...     n_processes=10,
... )
>>> 0.019558713928249892, np.array([...]), {"aggregation": ..., ...}
>>> # Evaluator for the SUN metric against the MP20 dataset using the
>>> # ElMD+AMD distance, with custom kwargs.
>>> evaluator = Evaluator(
...     stability="continuous",
...     uniqueness=True,
...     novelty=True,
...     distance="elmd+amd",
...     ref_xtals="mp20",
...     agg_func="prod",
...     {
...         "args_validity": {
...             "structure": {
...                 "threshold_distance": 0.5,
...                 "threshold_volume": 0.1,
...             }
...         },
...         "args_stability": {
...             "diagram": "mp_250618",
...             "mace_model": "medium-mpa-0",
...             "intercept": 0.4289,
...         },
...         "args_emb": {"amd": {"k": 100}},
...         "args_dist": {
...                             "elmd": {"metric": "mod_petti"},
...                             "amd": {"metric": "chebyshev", "low_memory": False},
...                             "coefs": {
...                                     "elmd": float.fromhex("0x1.8d7d565a99f87p-1"),
...                                     "amd": float.fromhex("0x1.ca0aa695981e5p-3")},
...                             },
...                     }
...     }
... )
>>> evaluator.evaluate(
...     xtals=xtals,  # list[Crystal | Structure]
...     dir_intermediate="intermediate_results/",
...     multiprocessing=True,
...     n_processes=10,
... )
>>> 0.16403383975840835, np.array([...]), {"aggregation": ..., ...}

Note

Here, I demonstrate how VSUN (or its subsets) is computed. Validity \(V(x)\) of each crystal \(x\) is 0 (invalid) or 1 (valid) depending on whether it passes the specified validity checks. If you specify multiple validity methods \(V_1(x)\), \(V_2(x)\), …, the overall validity is the product of individual validity scores:

\[V(x) = \prod_i V_i(x)\]

Stability \(S(x)\) is computed from the energy above hull, and has two variants: binary and continuous. The binary stability score \(S_b(x)\) is defined as follows.

\[\begin{split}S_b(x) = \begin{cases} 1 & \text{if } E_\text{hull}(x) \le \text{threshold} \\ 0 & \text{otherwise} \end{cases}\end{split}\]

“threshold” can be specified in kwargs when initializing the Evaluator. The default value is 0.1 [eV/atom]. The continuous stability score \(S_c(x)\) is defined as follows.

\[\begin{split}S_c(x) = \begin{cases} 1 & \text{if } E_\text{hull}(x) \le 0 \\ 1 - \frac{E_\text{hull}(x)}{\text{intercept}} & \text{if } 0 \le E_\text{hull}(x) \le \text{intercept} \\ 0 & \text{otherwise} \end{cases}\end{split}\]

“intercept” can be specified in kwargs when initializing the Evaluator. The default value 0.4289 [eV/atom]. In both cases, a higher score closer to 1 indicates a more stable structure. The definition of uniqueness \(U(x)\) depends on the chosen distance metric. For a binary distance \(d_b\), uniqueness of the i-th crystal \(x_i\) in the set of crystals \(\{x_1, x_2, \ldots, x_n\}\) is defined as follows.

\[U_b(x_i) = I \left(\land_{j=1}^{i-1} d_b(x_i, x_j) \neq 0 \right),\]

where \(I\) is the indicator function. For a continuous distance \(d_c\), uniqueness is defined as follows.

\[U_c(x_i) = \frac{1}{n-1} \sum_{j=1}^{n} d_c(x_i, x_j).\]

In both cases, the score ranges from 0 to 1, since the binary distance takes values of 0 or 1, and the continuous distance is normalized to be between 0 and 1. A higher score indicates a more unique structure. Novelty \(N(x)\) is defined similarly to uniqueness, but against a reference set of crystals. For a binary distance \(d_b\), novelty of the i-th crystal \(x_i\) is defined as follows.

\[N_b(x_i) = I \left(\land_{j=1}^{m} d_b(x_i, y_j) \neq 0 \right),\]

where \(\{y_1, y_2, \ldots, y_m\}\) is the reference set of crystals. For a continuous distance \(d_c\), novelty is defined as follows.

\[N_c(x_i) = \min_{j=1 \ldots m} d_c(x_i, y_j).\]

In both cases, the socre ranges from 0 to 1, and a higher score indicates a more novel structure. Finally, the VSUN score of each crystal \(x\) is computed by aggregating the individual scores using either multiplication or (weighted) average:

\[\begin{split}\text{VSUN}(x) = \begin{cases} V(x) S(x) U(x) N(x) & \text{if agg_func = "prod"} \\ w_V V(x) + w_S S(x) + w_U U(x) + w_N N(x) & \text{if agg_func ="ave"} \end{cases}\end{split}\]

where \(w_V\), \(w_S\), \(w_U\), and \(w_N\) are the normalized weights for validity, stability, uniqueness, and novelty, respectively. If only a subset of VSUN is evaluated, each crystal’s score is computed using only the specified components in the similar manner. The overall single score for the entire set of crystals is then obtained by averaging the individual scores.

\[\text{VSUN} = \frac{1}{n} \sum_{i=1}^{n} \text{VSUN}(x_i)\]