+
+
+
+
+
+
+
+
+
+
+
+
+ 
+
+You have reached the docs for the [torchbench](https://github.com/paperswithcode/torchbench) library. This library contains a collection of deep learning benchmarks you can use to
+benchmark your models, optimized for the PyTorch framework. It can be used in conjunction with the
+[sotabench.com](http://www.sotabench.com) website to record results for models, so the community
+can compare model performance on different tasks, as well as a continuous integration style
+service for your repository to benchmark your models on each commit.
+
+**torchbench** is a framework-optimized library, meaning it is designed to take advantage of PyTorch based features
+and standardisation. If this is too constraining, you can use alternative libraries that are framework-independent,
+ e.g. [sotabencheval](https://paperswithcode.github.io/sotabench-eval/).
+
+## Getting Started : Benchmarking on ImageNet
+
+**Step One : Create a sotabench.py file in the root of your repository**
+
+This contains a call to your model, metadata about your model, and options for evaluation such as dataset
+processing logic and data loader logic such as the batch size. Below is an example for the [torchvision](https://github.com/pytorch/vision)
+repository:
+
+``` python
+from torchbench.image_classification import ImageNet
+from torchvision.models.resnet import resnext101_32x8d
+import torchvision.transforms as transforms
+import PIL
+
+# Define the transforms need to convert ImageNet data to expected model input
+normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
+ std=[0.229, 0.224, 0.225])
+input_transform = transforms.Compose([
+ transforms.Resize(256, PIL.Image.BICUBIC),
+ transforms.CenterCrop(224),
+ transforms.ToTensor(),
+ normalize,
+])
+
+# Run the benchmark
+ImageNet.benchmark(
+ model=resnext101_32x8d(pretrained=True),
+ paper_model_name='ResNeXt-101-32x8d',
+ paper_arxiv_id='1611.05431',
+ input_transform=input_transform,
+ batch_size=256,
+ num_gpu=1
+)
+```
+
+**Step Two : Run locally to verify that it works**
+
+```
+python sotabench.py
+```
+
+You can also run the logic in a Jupyter Notebook if that is your preferred workflow.
+
+To verify your benchmark will run and all parameters are correct you can use the included CLI checking tool:
+
+```
+$ sb check
+```
+
+**Step Three : Login and connect your repository to [sotabench](http://www.sotabench.com)**
+
+Create an account on [sotabench](http://www.sotabench.com), then head to your user page. Click the
+**Connect a GitHub repository** button:
+
+
+
+Then follow the steps to connect the repositories that you wish to benchmark:
+
+
+
+After you connect your repository, the sotabench servers will re-evaluate your model on every commit,
+to ensure the model is working and results are up-to-date - including if you add additional models to the benchmark file.
+
+## Installation
+
+The library requires Python 3.6+. You can install via pip:
+
+```
+pip install torchbench
+```
+
+## Support
+
+If you get stuck you can head to our [Discourse](http://forum.sotabench.com) forum where you ask
+questions on how to use the project. You can also find ideas for contributions,
+and work with others on exciting projects.
\ No newline at end of file
diff --git a/docs/docs/pascalvoc.md b/docs/docs/pascalvoc.md
new file mode 100644
index 0000000..4c48bf8
--- /dev/null
+++ b/docs/docs/pascalvoc.md
@@ -0,0 +1,348 @@
+# PASCAL VOC 2012
+
+
+
+You can view the PASCAL VOC 2012 leaderboard [here](https://sotabench.com/benchmarks/semantic-segmentation-on-pascal-voc-2012).
+
+!!! Warning
+ Semantic Segmentations APIs in PyTorch are not very standardised across repositories, meaning that
+ it may require a lot of glue to get them working with this evaluation procedure (which is based on torchvision).
+
+ **For easier VOC integration with sotabench it is recommended to use the more general API [sotabencheval](https://paperswithcode.github.io/sotabench-eval/).**
+
+## Getting Started
+
+You'll need the following in the root of your repository:
+
+- `sotabench.py` file - contains benchmarking logic; the server will run this on each commit
+- `requirements.txt` file - Python dependencies to be installed before running `sotabench.py`
+- `sotabench_setup.sh` *(optional)* - any advanced dependencies or setup, e.g. compilation
+
+Once you connect your repository to [sotabench.com](https://www.sotabench.com), the platform
+will run your `sotabench.py` file whenever you commit to master.
+
+We now show how to write the `sotabench.py` file to evaluate a PyTorch object model with
+the torchbench library, and to allow your results to be recorded and reported for the community.
+
+## The VOC Evaluation Class
+
+You can import the evaluation class from the following module:
+
+``` python
+from torchbench.semantic_segmentation import PASCALVOC
+```
+
+The `PASCALVOC` class contains several components used in the evaluation, such as the `dataset`:
+
+``` python
+PASCALVOC.dataset
+# torchvision.datasets.voc.VOCSegmentation
+```
+
+And some default arguments used for evaluation (which can be overridden):
+
+``` python
+PASCALVOC.normalize
+# 
You can view the COCO minival leaderboard here.
+Warning
+Object detection APIs in PyTorch are not very standardised across repositories, meaning that +it may require a lot of glue to get them working with this evaluation procedure (which is based on torchvision).
+For easier COCO integration with sotabench it is recommended to use the more general API sotabencheval.
+You'll need the following in the root of your repository:
+sotabench.py file - contains benchmarking logic; the server will run this on each commitrequirements.txt file - Python dependencies to be installed before running sotabench.pysotabench_setup.sh (optional) - any advanced dependencies or setup, e.g. compilationOnce you connect your repository to sotabench.com, the platform
+will run your sotabench.py file whenever you commit to master.
We now show how to write the sotabench.py file to evaluate a PyTorch object model with
+the torchbench library, and to allow your results to be recorded and reported for the community.
You can import the evaluation class from the following module:
+from torchbench.object_detection import COCO +
The COCO class contains several components used in the evaluation, such as the dataset:
COCO.dataset +# torchbench.datasets.coco.CocoDetection +
And some default arguments used for evaluation (which can be overridden):
+COCO.transforms +# <torchbench.object_detection.transforms.Compose at 0x7f60e9ffd0b8> + +COCO.send_data_to_device +# <function torchbench.object_detection.coco.coco_data_to_device> + +COCO.collate_fn +# <function torchbench.object_detection.coco.coco_collate_fn> + +COCO.model_output_transform +# <function torchbench.object_detection.coco.coco_output_transform> +
We will explain these different options shortly and how you can manipulate them to get the +evaluation logic to play nicely with your model.
+An evaluation call - which performs evaluation, and if on the sotabench.com server, saves the results -
+looks like the following through the benchmark() method:
import torchvision +model = torchvision.models.detection.__dict__['maskrcnn_resnet50_fpn'](num_classes=91, pretrained=True) + +COCO.benchmark( + model=model, + paper_model_name='Mask R-CNN (ResNet-50-FPN)', + paper_arxiv_id='1703.06870' +) +
These are the key arguments: the model which is a usually a nn.Module type object, but more generally,
+is any method with a forward method that takes in input data and outputs predictions.
+paper_model_name refers to the name of the model and paper_arxiv_id (optionally) refers to
+the paper from which the model originated. If these two arguments match a record paper result,
+then sotabench.com will match your model with the paper and compare your code's results with the
+reported results in the paper.
sotabench.py exampleBelow shows an example for the torchvision +repository benchmarking a Mask R-CNN model:
+from torchbench.object_detection import COCO +from torchbench.utils import send_model_to_device +from torchbench.object_detection.transforms import Compose, ConvertCocoPolysToMask, ToTensor +import torchvision +import PIL + +def coco_data_to_device(input, target, device: str = "cuda", non_blocking: bool = True): + input = list(inp.to(device=device, non_blocking=non_blocking) for inp in input) + target = [{k: v.to(device=device, non_blocking=non_blocking) for k, v in t.items()} for t in target] + return input, target + +def coco_collate_fn(batch): + return tuple(zip(*batch)) + +def coco_output_transform(output, target): + output = [{k: v.to("cpu") for k, v in t.items()} for t in output] + return output, target + +transforms = Compose([ConvertCocoPolysToMask(), ToTensor()]) + +model = torchvision.models.detection.__dict__['maskrcnn_resnet50_fpn'](num_classes=91, pretrained=True) + +# Run the benchmark +COCO.benchmark( + model=model, + paper_model_name='Mask R-CNN (ResNet-50-FPN)', + paper_arxiv_id='1703.06870', + transforms=transforms, + model_output_transform=coco_output_transform, + send_data_to_device=coco_data_to_device, + collate_fn=coco_collate_fn, + batch_size=8, + num_gpu=1 +) +
COCO.benchmark() ArgumentsThe source code for the COCO evaluation method can be found here. +We now explain each argument.
+a PyTorch module, (e.g. a nn.Module object), that takes in COCO data and outputs detections.
For example, from the torchvision repository:
+import torchvision +model = torchvision.models.detection.__dict__['maskrcnn_resnet50_fpn'](num_classes=91, pretrained=True) +
(str, optional): Optional model description.
+For example:
+model_description = 'Using ported TensorFlow weights' +
Composing the transforms used to transform the input data (the images), e.g.
+resizing (e.g transforms.Resize), center cropping, to tensor transformations and normalization.
For example:
+import torchvision.transforms as transforms +input_transform = transforms.Compose([ + transforms.Resize(512, PIL.Image.BICUBIC), + transforms.ToTensor(), +]) +
Composing the transforms used to transform the target data
+Composing the transforms used to transform the input data (the images) and the target data (the labels) +in a dual fashion - for example resizing the pair of data jointly.
+Below shows an example; note the
+fact that the __call__ takes in two arguments and returns two arguments (ordinary torchvision transforms
+return one result).
from torchvision.transforms import functional as F + +class Compose(object): + def __init__(self, transforms): + self.transforms = transforms + + def __call__(self, image, target): + for t in self.transforms: + image, target = t(image, target) + return image, target + +class ToTensor(object): + def __call__(self, image, target): + image = F.to_tensor(image) + return image, target + +class ImageResize(object): + def __init__(self, resize_shape): + self.resize_shape = resize_shape + + def __call__(self, image, target): + image = F.resize(image, self.resize_shape) + return image, target + +transforms = Compose([ImageResize((512, 512)), ToTensor()]) +
Note that the default transforms are:
+from torchbench.object_detection.utils import Compose, ConvertCocoPolysToMask, ToTensor +transforms = Compose([ConvertCocoPolysToMask(), ToTensor()]) +
Where ConvertCocoPolysToMask is from the torchvision reference implementation to transform
+the inputs to the right format to be entered into the model. You can pass whatever transforms
+you need to make the dataset work with your model.
(callable, optional): An optional function
+ that takes in model output (after being passed through your
+ model forward pass) and transforms it. Afterwards, the
+ output will be passed into an evaluation function.
The model output transform is a function that you can pass in to transform the model output +after the data has been passed into the model. This is useful if you have to do further +processing steps after inference to get the predictions in the right format for evaluation.
+The model evaluation for each batch is as follows from utils.py +are:
+with torch.no_grad(): + for i, (input, target) in enumerate(iterator): + input, target = send_data_to_device(input, target, device=device) + original_output = model(input) + output, target = model_output_transform(original_output, target) + result = { + tar["image_id"].item(): out for tar, out in zip(target, output) + } + coco_evaluator.update(result) +
We can see the model_output_transform in use, and the fact that the output is then
+transformed to be a dictionary with image_ids as keys and output as values.
The expected output of model_output_transform is a list of dictionaries (length = batch_size),
+where each dictionary contains keys for 'boxes', 'labels', 'scores', 'masks', and each value is
+of the torch.tensor type.
The expected output of result is converted to a dictionary with keys as the image ids, and
+values as a dictionary with the predictions (boxes, labels, scores, ... as keys).
How the dataset is collated - an optional callable passed into the DataLoader
+As an example the default collate function is:
+def coco_collate_fn(batch): + return tuple(zip(*batch)) +
An optional function specifying how the model is sent to a device
+As an example the COCO default is:
+def coco_data_to_device(input, target, device: str = "cuda", non_blocking: bool = True): + input = list(inp.to(device=device, non_blocking=non_blocking) for inp in input) + target = [{k: v.to(device=device, non_blocking=non_blocking) for k, v in t.items()} for t in target] + return input, target +
data_root (str): The location of the COCO dataset - change this + parameter when evaluating locally if your COCO data is + located in a different folder (or alternatively if you want to + download to an alternative location).
+Note that this parameter will be overriden when the evaluation is performed on the server, +so it is solely for your local use.
+num_workers (int): The number of workers to use for the DataLoader.
+batch_size (int) : The batch_size to use for evaluation; if you get + memory errors, then reduce this (half each time) until your + model fits onto the GPU.
+paper_model_name (str, optional): The name of the model from the + paper - if you want to link your build to a machine learning + paper. See the COCO benchmark page for model names, + https://www.sotabench.com/benchmark/coco-minival, e.g. on the paper + leaderboard tab.
+paper_arxiv_id (str, optional): Optional linking to ArXiv if you + want to link to papers on the leaderboard; put in the + corresponding paper's ArXiv ID, e.g. '1611.05431'.
+paper_pwc_id (str, optional): Optional linking to Papers With Code; + put in the corresponding papers with code URL slug, e.g. + 'u-gat-it-unsupervised-generative-attentional'
+paper_results (dict, optional) : If the paper you are reproducing + does not have model results on sotabench.com, you can specify + the paper results yourself through this argument, where keys + are metric names, values are metric values. e.g::
+{'box AP': 0.349, 'AP50': 0.592, ...}. +
Ensure that the metric names match those on the sotabench +leaderboard - for COCO it should be 'box AP', 'AP50', +'AP75', 'APS', 'APM', 'APL'
+pytorch_hub_url (str, optional): Optional linking to PyTorch Hub + url if your model is linked there; e.g: + 'nvidia_deeplearningexamples_waveglow'.
+Head on over to the Computer Vision section of the sotabench +forums if you have any questions or difficulties.
+ + + + + + + + + +
You can view the ImageNet leaderboard here.
+You'll need the following in the root of your repository:
+sotabench.py file - contains benchmarking logic; the server will run this on each commitrequirements.txt file - Python dependencies to be installed before running sotabench.pysotabench_setup.sh (optional) - any advanced dependencies or setup, e.g. compilationOnce you connect your repository to sotabench.com, the platform
+will run your sotabench.py file whenever you commit to master.
We now show how to write the sotabench.py file to evaluate a PyTorch object model with
+the torchbench library, and to allow your results to be recorded and reported for the community.
You can import the evaluation class from the following module:
+from torchbench.image_classification import ImageNet +
The ImageNet class contains several components used in the evaluation, such as the dataset:
ImageNet.dataset +# torchvision.datasets.ImageNet +
And some default arguments used for evaluation (which can be overridden):
+ImageNet.normalize +# Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) + +ImageNet.input_transform +# Compose( +# Resize(size=256, interpolation=PIL.Image.BILINEAR) +# CenterCrop(size=(224, 224)) +# ToTensor() +# Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) +# ) + +ImageNet.send_data_to_device +# <function torchbench.utils.default_data_to_device> +
We will explain these different options shortly and how you can manipulate them to get the +evaluation logic to play nicely with your model.
+An evaluation call - which performs evaluation, and if on the sotabench.com server, saves the results -
+looks like the following through the benchmark() method:
from torchvision.models.resnet import resnext101_32x8d + +ImageNet.benchmark( + model=resnext101_32x8d(pretrained=True), + paper_model_name='ResNeXt-101-32x8d', + paper_arxiv_id='1611.05431' +) +
These are the key arguments: the model which is a usually a nn.Module type object, but more generally,
+is any method with a forward method that takes in input data and outputs predictions.
+paper_model_name refers to the name of the model and paper_arxiv_id (optionally) refers to
+the paper from which the model originated. If these two arguments match a record paper result,
+then sotabench.com will match your model with the paper and compare your code's results with the
+reported results in the paper.
sotabench.py exampleBelow shows an example for the torchvision +repository benchmarking a ResNeXt-101-32x8d model:
+from torchbench.image_classification import ImageNet +from torchvision.models.resnet import resnext101_32x8d +import torchvision.transforms as transforms +import PIL + +# Define the transforms need to convert ImageNet data to expected +# model input +normalize = transforms.Normalize( + mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225] +) +input_transform = transforms.Compose([ + transforms.Resize(256, PIL.Image.BICUBIC), + transforms.CenterCrop(224), + transforms.ToTensor(), + normalize, +]) + +# Run the benchmark +ImageNet.benchmark( + model=resnext101_32x8d(pretrained=True), + paper_model_name='ResNeXt-101-32x8d', + paper_arxiv_id='1611.05431', + input_transform=input_transform, + batch_size=256, + num_gpu=1 +) +
ImageNet.benchmark() ArgumentsThe source code for the ImageNet evaluation method can be found here. +We now explain each argument.
+a PyTorch module, (e.g. a nn.Module object), that takes in ImageNet data and outputs detections.
For example, from the torchvision repository:
+from torchvision.models.resnet import resnext101_32x8d +model = resnext101_32x8d(pretrained=True) +
(str, optional): Optional model description.
+For example:
+model_description = 'Using ported TensorFlow weights' +
Composing the transforms used to transform the input data (the images), e.g.
+resizing (e.g transforms.Resize), center cropping, to tensor transformations and normalization.
For example:
+import torchvision.transforms as transforms +normalize = transforms.Normalize( + mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225] +) +input_transform = transforms.Compose([ + transforms.Resize(256, PIL.Image.BICUBIC), + transforms.CenterCrop(224), + transforms.ToTensor(), + normalize, +]) +
Composing the transforms used to transform the target data
+(callable, optional): An optional function
+ that takes in model output (after being passed through your
+ model forward pass) and transforms it. Afterwards, the
+ output will be passed into an evaluation function.
The model output transform is a function that you can pass in to transform the model output +after the data has been passed into the model. This is useful if you have to do further +processing steps after inference to get the predictions in the right format for evaluation.
+Most PyTorch models for Image Classification on ImageNet don't need to use this argument.
+The model evaluation for each batch is as follows from utils.py +are:
+with torch.no_grad(): + for i, (input, target) in enumerate(iterator): + + input, target = send_data_to_device(input, target, device=device) + output = model(input) + + if model_output_transform is not None: + output = model_output_transform(output, target, model=model) + + check_metric_inputs(output, target, test_loader.dataset, i) + prec1, prec5 = accuracy(output, target, topk=(1, 5)) +
Model output (following model.forward() and optionally model_output_transform) should be a 2D
+torch.Tensor containing the model output; first dimension should be output for each example
+(length batch_size) and second dimension should be output for each class in ImageNet (length 1000).
An optional function specifying how the model is sent to a device
+As an example the default is:
+def default_data_to_device(input, target=None, device: str = "cuda", non_blocking: bool = True): + """Sends data output from a PyTorch Dataloader to the device.""" + + input = input.to(device=device, non_blocking=non_blocking) + + if target is not None: + target = target.to(device=device, non_blocking=non_blocking) + + return input, target +
data_root (str): The location of the ImageNet dataset - change this + parameter when evaluating locally if your ImageNet data is + located in a different folder (or alternatively if you want to + download to an alternative location).
+Note that this parameter will be overriden when the evaluation is performed on the server, +so it is solely for your local use.
+num_workers (int): The number of workers to use for the DataLoader.
+batch_size (int) : The batch_size to use for evaluation; if you get + memory errors, then reduce this (half each time) until your + model fits onto the GPU.
+paper_model_name (str, optional): The name of the model from the + paper - if you want to link your build to a machine learning + paper. See the ImageNet benchmark page for model names, + https://sotabench.com/benchmarks/image-classification-on-imagenet, e.g. on the paper + leaderboard tab.
+paper_arxiv_id (str, optional): Optional linking to ArXiv if you + want to link to papers on the leaderboard; put in the + corresponding paper's ArXiv ID, e.g. '1611.05431'.
+paper_pwc_id (str, optional): Optional linking to Papers With Code; + put in the corresponding papers with code URL slug, e.g. + 'u-gat-it-unsupervised-generative-attentional'
+paper_results (dict, optional) : If the paper you are reproducing + does not have model results on sotabench.com, you can specify + the paper results yourself through this argument, where keys + are metric names, values are metric values. e.g:
+{'Top 1 Accuracy': 0.543, 'Top 5 Accuracy': 0.654} +
Ensure that the metric names match those on the sotabench +leaderboard - for ImageNet it should be 'Top 1 Accuracy', +'Top 5 Accuracy'
+pytorch_hub_url (str, optional): Optional linking to PyTorch Hub + url if your model is linked there; e.g: + 'nvidia_deeplearningexamples_waveglow'.
+Head on over to the Computer Vision section of the sotabench +forums if you have any questions or difficulties.
+ + + + + + + + + +
You have reached the docs for the torchbench library. This library contains a collection of deep learning benchmarks you can use to +benchmark your models, optimized for the PyTorch framework. It can be used in conjunction with the +sotabench.com website to record results for models, so the community +can compare model performance on different tasks, as well as a continuous integration style +service for your repository to benchmark your models on each commit.
+torchbench is a framework-optimized library, meaning it is designed to take advantage of PyTorch based features +and standardisation. If this is too constraining, you can use alternative libraries that are framework-independent, + e.g. sotabencheval.
+Step One : Create a sotabench.py file in the root of your repository
+This contains a call to your model, metadata about your model, and options for evaluation such as dataset +processing logic and data loader logic such as the batch size. Below is an example for the torchvision +repository:
+from torchbench.image_classification import ImageNet +from torchvision.models.resnet import resnext101_32x8d +import torchvision.transforms as transforms +import PIL + +# Define the transforms need to convert ImageNet data to expected model input +normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], + std=[0.229, 0.224, 0.225]) +input_transform = transforms.Compose([ + transforms.Resize(256, PIL.Image.BICUBIC), + transforms.CenterCrop(224), + transforms.ToTensor(), + normalize, +]) + +# Run the benchmark +ImageNet.benchmark( + model=resnext101_32x8d(pretrained=True), + paper_model_name='ResNeXt-101-32x8d', + paper_arxiv_id='1611.05431', + input_transform=input_transform, + batch_size=256, + num_gpu=1 +) +
Step Two : Run locally to verify that it works
+python sotabench.py +
You can also run the logic in a Jupyter Notebook if that is your preferred workflow.
+To verify your benchmark will run and all parameters are correct you can use the included CLI checking tool:
+$ sb check +
Step Three : Login and connect your repository to sotabench
+Create an account on sotabench, then head to your user page. Click the +Connect a GitHub repository button:
+
Then follow the steps to connect the repositories that you wish to benchmark:
+
After you connect your repository, the sotabench servers will re-evaluate your model on every commit, +to ensure the model is working and results are up-to-date - including if you add additional models to the benchmark file.
+The library requires Python 3.6+. You can install via pip:
+pip install torchbench +
If you get stuck you can head to our Discourse forum where you ask +questions on how to use the project. You can also find ideas for contributions, +and work with others on exciting projects.
+ + + + + + + + + +
You can view the PASCAL VOC 2012 leaderboard here.
+Warning
+Semantic Segmentations APIs in PyTorch are not very standardised across repositories, meaning that +it may require a lot of glue to get them working with this evaluation procedure (which is based on torchvision).
+For easier VOC integration with sotabench it is recommended to use the more general API sotabencheval.
+You'll need the following in the root of your repository:
+sotabench.py file - contains benchmarking logic; the server will run this on each commitrequirements.txt file - Python dependencies to be installed before running sotabench.pysotabench_setup.sh (optional) - any advanced dependencies or setup, e.g. compilationOnce you connect your repository to sotabench.com, the platform
+will run your sotabench.py file whenever you commit to master.
We now show how to write the sotabench.py file to evaluate a PyTorch object model with
+the torchbench library, and to allow your results to be recorded and reported for the community.
You can import the evaluation class from the following module:
+from torchbench.semantic_segmentation import PASCALVOC +
The PASCALVOC class contains several components used in the evaluation, such as the dataset:
PASCALVOC.dataset +# torchvision.datasets.voc.VOCSegmentation +
And some default arguments used for evaluation (which can be overridden):
+PASCALVOC.normalize +# <torchbench.semantic_segmentation.transforms.Normalize at 0x7f9d645d2160> + +PASCALVOC.transforms +# <torchbench.semantic_segmentation.transforms.Compose at 0x7f9d645d2278> + +PASCALVOC.send_data_to_device +# <function torchbench.utils.default_data_to_device> + +PASCALVOC.collate_fn +# <function torchbench.semantic_segmentation.utils.default_seg_collate_fn> + +PASCALVOC.model_output_transform +# <function torchbench.semantic_segmentation.utils.default_seg_output_transform> +
We will explain these different options shortly and how you can manipulate them to get the +evaluation logic to play nicely with your model.
+An evaluation call - which performs evaluation, and if on the sotabench.com server, saves the results -
+looks like the following through the benchmark() method:
from torchvision.models.segmentation import fcn_resnet101 +model = fcn_resnet101(num_classes=21, pretrained=True) + +PASCALVOC.benchmark(model=model, + paper_model_name='FCN ResNet-101', + paper_arxiv_id='1605.06211') +
These are the key arguments: the model which is a usually a nn.Module type object, but more generally,
+is any method with a forward method that takes in input data and outputs predictions.
+paper_model_name refers to the name of the model and paper_arxiv_id (optionally) refers to
+the paper from which the model originated. If these two arguments match a record paper result,
+then sotabench.com will match your model with the paper and compare your code's results with the
+reported results in the paper.
sotabench.py exampleBelow shows an example for the torchvision +repository benchmarking a FCN ResNet-101 model:
+from torchbench.semantic_segmentation import PASCALVOC +from torchbench.semantic_segmentation.transforms import ( + Normalize, + Resize, + ToTensor, + Compose, +) +from torchvision.models.segmentation import fcn_resnet101 +import torchvision.transforms as transforms +import PIL + +def model_output_function(output, labels): + return output['out'].argmax(1).flatten(), target.flatten() + +def seg_collate_fn(batch): + images, targets = list(zip(*batch)) + batched_imgs = cat_list(images, fill_value=0) + batched_targets = cat_list(targets, fill_value=255) + return batched_imgs, batched_targets + +model = fcn_resnet101(num_classes=21, pretrained=True) + +normalize = Normalize( + mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225] +) +my_transforms = Compose([Resize((520, 480)), ToTensor(), normalize]) + +PASCALVOC.benchmark(batch_size=32, + model=model, + transforms=my_transforms, + model_output_transform=model_output_function, + collate_fn=seg_collate_fn, + paper_model_name='FCN ResNet-101', + paper_arxiv_id='1605.06211') +
PASCALVOC.benchmark() ArgumentsThe source code for the PASCALVOC evaluation method can be found here. +We now explain each argument.
+a PyTorch module, (e.g. a nn.Module object), that takes in VOC data and outputs detections.
For example, from the torchvision repository:
+from torchvision.models.segmentation import fcn_resnet101 +model = fcn_resnet101(num_classes=21, pretrained=True) +
(str, optional): Optional model description.
+For example:
+model_description = 'Using ported TensorFlow weights' +
Composing the transforms used to transform the input data (the images), e.g.
+resizing (e.g transforms.Resize), center cropping, to tensor transformations and normalization.
For example:
+import torchvision.transforms as transforms +input_transform = transforms.Compose([ + transforms.Resize(512, PIL.Image.BICUBIC), + transforms.ToTensor(), +]) +
Composing the transforms used to transform the target data
+Composing the transforms used to transform the input data (the images) and the target data (the labels) +in a dual fashion - for example resizing the pair of data jointly.
+Below shows an example; note the
+fact that the __call__ takes in two arguments and returns two arguments (ordinary torchvision transforms
+return one result).
from torchvision.transforms import functional as F + +class Compose(object): + def __init__(self, transforms): + self.transforms = transforms + + def __call__(self, image, target): + for t in self.transforms: + image, target = t(image, target) + return image, target + +class ToTensor(object): + def __call__(self, image, target): + image = F.to_tensor(image) + return image, target + +class ImageResize(object): + def __init__(self, resize_shape): + self.resize_shape = resize_shape + + def __call__(self, image, target): + image = F.resize(image, self.resize_shape) + return image, target + +transforms = Compose([ImageResize((512, 512)), ToTensor()]) +
Note that the default transforms are:
+from torchbench.semantic_segmentation.transforms import (Normalize, Resize, ToTensor, Compose) +normalize = Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) +transforms = Compose([Resize((520, 480)), ToTensor(), normalize]) +
(callable, optional): An optional function
+ that takes in model output (after being passed through your
+ model forward pass) and transforms it. Afterwards, the
+ output will be passed into an evaluation function.
The model output transform is a function that you can pass in to transform the model output +after the data has been passed into the model. This is useful if you have to do further +processing steps after inference to get the predictions in the right format for evaluation.
+The model evaluation for each batch is as follows from utils.py +are:
+with torch.no_grad(): + for i, (input, target) in enumerate(iterator): + input, target = send_data_to_device(input, target, device=device) + output = model(input) + output, target = model_output_transform(output, target) + confmat.update(target, output) +
The default model_output_transform is:
def default_seg_output_transform(output, target): + return output["out"].argmax(1).flatten(), target.flatten() +
We can see the output and target are flattened to 1D tensors, and in the case of the output,
+we take the maximum predicted class to compare against for accuracy. Each element in each tensor
+represents a pixel, and contains a class, e.g. class 6, and we compare pixel-by-pixel the model
+predictions against the ground truth labels to calculate the accuracy.
How the dataset is collated - an optional callable passed into the DataLoader
+As an example the default collate function is:
+def default_seg_collate_fn(batch): + images, targets = list(zip(*batch)) + batched_imgs = cat_list(images, fill_value=0) + batched_targets = cat_list(targets, fill_value=255) + return batched_imgs, batched_targets +
An optional function specifying how the model is sent to a device
+As an example the PASCAL VOC default is:
+def default_data_to_device(input, target=None, device: str = "cuda", non_blocking: bool = True): + """Sends data output from a PyTorch Dataloader to the device.""" + + input = input.to(device=device, non_blocking=non_blocking) + + if target is not None: + target = target.to(device=device, non_blocking=non_blocking) + + return input, target +
data_root (str): The location of the VOC dataset - change this + parameter when evaluating locally if your VOC data is + located in a different folder (or alternatively if you want to + download to an alternative location).
+Note that this parameter will be overriden when the evaluation is performed on the server, +so it is solely for your local use.
+num_workers (int): The number of workers to use for the DataLoader.
+batch_size (int) : The batch_size to use for evaluation; if you get + memory errors, then reduce this (half each time) until your + model fits onto the GPU.
+paper_model_name (str, optional): The name of the model from the + paper - if you want to link your build to a machine learning + paper. See the VOC benchmark page for model names, + https://sotabench.com/benchmarks/semantic-segmentation-on-pascal-voc-2012, e.g. on the paper + leaderboard tab.
+paper_arxiv_id (str, optional): Optional linking to ArXiv if you + want to link to papers on the leaderboard; put in the + corresponding paper's ArXiv ID, e.g. '1611.05431'.
+paper_pwc_id (str, optional): Optional linking to Papers With Code; + put in the corresponding papers with code URL slug, e.g. + 'u-gat-it-unsupervised-generative-attentional'
+paper_results (dict, optional) : If the paper you are reproducing + does not have model results on sotabench.com, you can specify + the paper results yourself through this argument, where keys + are metric names, values are metric values. e.g::
+{'Accuracy': 0.745, 'Mean IOU': 0.592}. +
Ensure that the metric names match those on the sotabench +leaderboard - for VOC it should be 'Accuracy', 'Mean IOU'.
+pytorch_hub_url (str, optional): Optional linking to PyTorch Hub + url if your model is linked there; e.g: + 'nvidia_deeplearningexamples_waveglow'.
+Head on over to the Computer Vision section of the sotabench +forums if you have any questions or difficulties.
+ + + + + + + + + +