flambe.metric
¶
Subpackages¶
Submodules¶
Package Contents¶
-
class
flambe.metric.
Metric
[source]¶ Bases:
flambe.compile.Component
Base Metric interface.
Objects implementing this interface should take in a sequence of examples and provide as output a processd list of the same size.
-
compute
(self, pred: torch.Tensor, target: torch.Tensor)¶ Computes the metric over the given prediction and target.
Parameters: - pred (torch.Tensor) – The model predictions
- target (torch.Tensor) – The ground truth targets
Returns: The computed metric
Return type: torch.Tensor
-
aggregate
(self, state: dict, *args, **kwargs)¶ Parameters: - state (dict) – the state dictionary
- args – normally pred, target
- kwargs –
Returns: The updated state (even though the update happens in-place)
Return type: dict
-
finalize
(self, state)¶ FInalizes the metric computation
Parameters: state (dict) – the metric state Returns: The final score. Can be anything, depending on metric. Return type: Any
-
__call__
(self, *args, **kwargs)¶ Makes Featurizer a callable.
-
__str__
(self)¶ Return the name of the Metric (for use in logging).
-
-
class
flambe.metric.
MultiLabelCrossEntropy
(weight: Optional[torch.Tensor] = None, ignore_index: Optional[int] = None, reduction: str = 'mean')[source]¶ Bases:
flambe.metric.metric.Metric
-
__str__
(self)¶ Return the name of the Metric (for use in logging).
-
compute
(self, pred: torch.Tensor, target: torch.Tensor)¶ Computes the multilabel cross entropy loss.
Parameters: - pred (torch.Tensor) – input logits of shape (B x N)
- target (torch.LontTensor) – target tensor of shape (B x N)
Returns: loss – Multi label cross-entropy loss, of shape (B)
Return type: torch.Tensor
-
-
class
flambe.metric.
MultiLabelNLLLoss
(weight: Optional[torch.Tensor] = None, ignore_index: Optional[int] = None, reduction: str = 'mean')[source]¶ Bases:
flambe.metric.metric.Metric
-
__str__
(self)¶ Return the name of the Metric (for use in logging).
-
compute
(self, pred: torch.Tensor, target: torch.Tensor)¶ Computes the Negative log likelihood loss for multilabel.
Parameters: - pred (torch.Tensor) – input logits of shape (B x N)
- target (torch.LontTensor) – target tensor of shape (B x N)
Returns: loss – Multi label negative log likelihood loss, of shape (B)
Return type: torch.float
-
-
class
flambe.metric.
Accuracy
[source]¶ Bases:
flambe.metric.metric.Metric
-
compute
(self, pred: torch.Tensor, target: torch.Tensor)¶ Computes the loss.
Parameters: - pred (Tensor) – input logits of shape (B x N)
- target (LontTensor) – target tensor of shape (B) or (B x N)
Returns: accuracy – single label accuracy, of shape (B)
Return type: torch.Tensor
-
-
class
flambe.metric.
Perplexity
[source]¶ Bases:
flambe.metric.Metric
Token level perplexity, computed a exp(cross_entropy).
-
compute
(self, pred: torch.Tensor, target: torch.Tensor)¶ Compute the preplexity given the input and target.
Parameters: - pred (torch.Tensor) – input logits of shape (B x N)
- target (torch.LontTensor) – target tensor of shape (B)
Returns: Output perplexity
Return type: torch.float
-
-
class
flambe.metric.
BPC
[source]¶ Bases:
flambe.metric.Metric
Bits per character. Computed as log_2(perplexity)
-
compute
(self, pred: torch.Tensor, target: torch.Tensor)¶ Compute the bits per character given the input and target.
Parameters: - pred (torch.Tensor) – input logits of shape (B x N)
- target (torch.LontTensor) – target tensor of shape (B)
Returns: Output perplexity
Return type: torch.float
-
-
class
flambe.metric.
AUC
(max_fpr=1.0)[source]¶ Bases:
flambe.metric.metric.Metric
-
__str__
(self)¶ Return the name of the Metric (for use in logging).
-
static
aggregate
(state: dict, *args, **kwargs)¶ Aggregates by simply storing preds and targets
Parameters: - state (dict) – the metric state
- args (the pred, target tuple) –
Returns: the state dict
Return type: dict
-
finalize
(self, state: Dict)¶ Finalizes the metric computation
Parameters: state (dict) – the metric state Returns: The final score. Return type: float
-
compute
(self, pred: torch.Tensor, target: torch.Tensor)¶ Compute AUC at the given max false positive rate.
Parameters: - pred (torch.Tensor) – The model predictions
- target (torch.Tensor) – The binary targets
Returns: The computed AUC
Return type: torch.Tensor
-
-
class
flambe.metric.
BinaryPrecision
(threshold: float = 0.5, positive_label: int = 1)[source]¶ Bases:
flambe.metric.dev.binary.BinaryMetric
Compute Binary Precision.
An example is considered negative when its score is below the specified threshold. Binary precition is computed as follows:
` |True positives| / |True Positives| + |False Positives| `
-
compute_binary
(self, pred: torch.Tensor, target: torch.Tensor)¶ Compute binary precision.
Parameters: - pred (torch.Tensor) – Predictions made by the model. It should be a probability 0 <= p <= 1 for each sample, 1 being the positive class.
- target (torch.Tensor) – Ground truth. Each label should be either 0 or 1.
Returns: The computed binary metric
Return type: torch.float
-
__str__
(self)¶ Return the name of the Metric (for use in logging).
-
-
class
flambe.metric.
BinaryRecall
(threshold: float = 0.5, positive_label: int = 1)[source]¶ Bases:
flambe.metric.dev.binary.BinaryMetric
Compute binary recall.
An example is considered negative when its score is below the specified threshold. Binary precition is computed as follows:
` |True positives| / |True Positives| + |False Negatives| `
-
compute_binary
(self, pred: torch.Tensor, target: torch.Tensor)¶ Compute binary recall.
Parameters: - pred (torch.Tensor) – Predictions made by the model. It should be a probability 0 <= p <= 1 for each sample, 1 being the positive class.
- target (torch.Tensor) – Ground truth. Each label should be either 0 or 1.
Returns: The computed binary metric
Return type: torch.float
-
__str__
(self)¶ Return the name of the Metric (for use in logging).
-
-
class
flambe.metric.
BinaryAccuracy
[source]¶ Bases:
flambe.metric.dev.binary.BinaryMetric
Compute binary accuracy.
` |True positives + True negatives| / N `
-
compute_binary
(self, pred: torch.Tensor, target: torch.Tensor)¶ Compute binary accuracy.
Parameters: - pred (torch.Tensor) – Predictions made by the model. It should be a probability 0 <= p <= 1 for each sample, 1 being the positive class.
- target (torch.Tensor) – Ground truth. Each label should be either 0 or 1.
Returns: The computed binary metric
Return type: torch.float
-
-
class
flambe.metric.
F1
(threshold: float = 0.5, positive_label: int = 1, eps: float = 1e-08)[source]¶ Bases:
flambe.metric.dev.binary.BinaryMetric
-
compute_binary
(self, pred: torch.Tensor, target: torch.Tensor)¶ Compute F1. Score, the harmonic mean between precision and recall.
Parameters: - pred (torch.Tensor) – Predictions made by the model. It should be a probability 0 <= p <= 1 for each sample, 1 being the positive class.
- target (torch.Tensor) – Ground truth. Each label should be either 0 or 1.
Returns: The computed binary metric
Return type: torch.float
-