{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2019 NVIDIA Corporation. All Rights Reserved.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "# =============================================================================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://developer.download.nvidia.com/compute/machine-learning/frameworks/nvidia_logo.png\" style=\"width: 90px; float: right;\">\n",
    "\n",
    "# Mask R-CNN Training and Inference Demo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "Mask R-CNN is a convolution-based neural network architecture for the task of object instance segmentation. The original paper describing the model can be found at [https://arxiv.org/abs/1703.06870](https://arxiv.org/abs/1703.06870). NVIDIA’s Mask R-CNN is an optimized version of [Facebook’s implementation](https://github.com/facebookresearch/maskrcnn-benchmark), leveraging mixed precision arithmetic and tensor cores for faster training while maintaining comparable accuracy with single precision (FP32) training.\n",
    "\n",
    "The major differences between the official implementation of the paper and our version of Mask R-CNN are as follows:\n",
    "\n",
    "- Mixed precision support with [PyTorch automatic mixed precision (AMP)](https://github.com/NVIDIA/apex).\n",
    "- Gradient accumulation to simulate larger batches.\n",
    "- Custom fused CUDA kernels for faster computations.\n",
    "\n",
    "Note: If the issue `DLLLoggerAlreadyInitialized:` encountered,\n",
    "try again after restarting the notebook.\n",
    "\n",
    "\n",
    "### Learning objectives\n",
    "\n",
    "This notebook demonstrates the steps for training a Mask R-CNN model using 1 or multiple GPUs. We then employ the trained model to make inference on new images.\n",
    "\n",
    "## Content\n",
    "1. [Requirements](#1)\n",
    "1. [Data download and preprocessing](#2)\n",
    "1. [Training](#3)\n",
    "1. [Testing trained model](#4)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"1\"></a>\n",
    "## 1. Requirements\n",
    "\n",
    "\n",
    "### 1.1 Docker container\n",
    "The most convenient way to make use of the NVIDIA Mask R-CNN model is via a docker container, which provides a self-contained, isolated and re-producible environment for all experiments. Refer to the [Quick Start Guide section](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Segmentation/MaskRCNN#requirements) of the Readme documentation for a comprehensive guide. We briefly summarize the steps here.\n",
    "\n",
    "First, clone the repository:\n",
    "\n",
    "```\n",
    "git clone https://github.com/NVIDIA/DeepLearningExamples.git\n",
    "cd DeepLearningExamples/PyTorch/Segmentation/MaskRCNN\n",
    "```\n",
    "\n",
    "Next, build the NVIDIA Mask R-CNN container:\n",
    "\n",
    "```\n",
    "cd pytorch\n",
    "docker build --rm -t nvidia_joc_maskrcnn_pt .\n",
    "```\n",
    "\n",
    "Then launch the container with:\n",
    "\n",
    "```\n",
    "PATH_TO_COCO='/path/to/coco-2014'\n",
    "MOUNT_LOCATION='/datasets/data'\n",
    "NAME='nvidia_maskrcnn'\n",
    "\n",
    "docker run --it --runtime=nvidia -p 8888:8888 -v $PATH_TO_COCO:/$MOUNT_LOCATION --rm --name=$NAME --shm-size=10g --ulimit memlock=-1 --ulimit stack=67108864 --ipc=host nvidia_joc_maskrcnn_pt\n",
    "```\n",
    "where `/path/to/coco-2014` is the path on the host machine where the data was/is to be downloaded. More on data set preparation in the next section.\n",
    "\n",
    "Within the docker interactive bash session, start Jupyter with\n",
    "\n",
    "```\n",
    "jupyter notebook --ip 0.0.0.0 --port 8888\n",
    "```\n",
    "\n",
    "Then open the Jupyter GUI interface on your host machine at http://localhost:8888. Within the container, this notebook itself is located at `/workspace/object_detection/demo`.\n",
    "\n",
    "### 1.2 Hardware\n",
    "This notebook can be executed on any CUDA-enabled NVIDIA GPU, although for efficient mixed precision training, a [Tensor Core NVIDIA GPU](https://www.nvidia.com/en-us/data-center/tensorcore/) is desired (Volta, Turing or newer architectures). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvidia-smi\n",
    "!pip install ipywidgets\n",
    "!jupyter nbextension enable --py widgetsnbextension"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"2\"></a>\n",
    "## 2. Data download and preprocessing\n",
    "\n",
    "This notebook demonstrates training and validation of the Mask R-CNN model on the [COCO 2014 dataset](http://cocodataset.org/#download). If not already available locally, the following [script](https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/Segmentation/MaskRCNN/download_dataset.sh) in the repository provides a convenient way to download and extract all the necessary data in one go. Be mindful of the size of the raw data (~20GB). The script makes use of `wget` and will automatically resume if disrupted. Once downloaded, the script invokes `dtrx` to extract the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! wget https://raw.githubusercontent.com/NVIDIA/DeepLearningExamples/master/PyTorch/Segmentation/MaskRCNN/download_dataset.sh -P /workspace/object_detection/\n",
    "! bash /workspace/object_detection/download_dataset.sh /datasets/data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Within the docker container, the final data directory should look like:\n",
    "\n",
    "```\n",
    "/datasets/data\n",
    "  annotations/\n",
    "    instances_train2014.json\n",
    "    instances_val2014.json\n",
    "  train2014/\n",
    "    COCO_train2014_*.jpg\n",
    "  val2014/\n",
    "    COCO_val2014_*.jpg\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"3\"></a>\n",
    "## 3. Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The shell script [train.sh](https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/Segmentation/MaskRCNN/pytorch/scripts/train.sh) provides a convenient interface to launch training tasks. \n",
    "By default, invoking [train.sh](https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/Segmentation/MaskRCNN/pytorch/scripts/train.sh) will make use of 8 GPUs, saves checkpoints every 2500 iterations and uses mixed precision training.\n",
    "\n",
    "```\n",
    "cd /workspace/object_detection/\n",
    "bash scripts/train.sh\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that, within [train.sh](https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/Segmentation/MaskRCNN/pytorch/scripts/train.sh), it invokes the following Python command:\n",
    "\n",
    "```python -m torch.distributed.launch --nproc_per_node=8 tools/train_net.py --config-file \"configs/e2e_mask_rcnn_R_50_FPN_1x.yaml\" DTYPE \"float16\"```\n",
    "\n",
    "which launches pytorch distributed training with 8 GPUs, using the train script in [tools/train_net.py](https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/Segmentation/MaskRCNN/pytorch/tools/train_net.py). Various sample training configurations are available within the [configs](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Segmentation/MaskRCNN/pytorch/configs) directory, for example, a [configuration file](https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/Segmentation/MaskRCNN/pytorch/configs/e2e_mask_rcnn_R_50_FPN_1x_1GPU.yaml) for training using 1 GPU. \n",
    "\n",
    "### 3.1 Training with 1 GPU\n",
    "We will now take a closer look at training a Mask-RCNN model using 1 GPU, using the below custom config script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "echo 'MODEL:\n",
    "  META_ARCHITECTURE: \"GeneralizedRCNN\"\n",
    "  WEIGHT: \"catalog://ImageNetPretrained/MSRA/R-50\"\n",
    "  BACKBONE:\n",
    "    CONV_BODY: \"R-50-FPN\"\n",
    "    OUT_CHANNELS: 256\n",
    "  RPN:\n",
    "    USE_FPN: True\n",
    "    ANCHOR_STRIDE: (4, 8, 16, 32, 64)\n",
    "    PRE_NMS_TOP_N_TRAIN: 2000\n",
    "    PRE_NMS_TOP_N_TEST: 1000\n",
    "    POST_NMS_TOP_N_TEST: 1000\n",
    "    FPN_POST_NMS_TOP_N_TEST: 1000\n",
    "  ROI_HEADS:\n",
    "    USE_FPN: True\n",
    "  ROI_BOX_HEAD:\n",
    "    POOLER_RESOLUTION: 7\n",
    "    POOLER_SCALES: (0.25, 0.125, 0.0625, 0.03125)\n",
    "    POOLER_SAMPLING_RATIO: 2\n",
    "    FEATURE_EXTRACTOR: \"FPN2MLPFeatureExtractor\"\n",
    "    PREDICTOR: \"FPNPredictor\"\n",
    "  ROI_MASK_HEAD:\n",
    "    POOLER_SCALES: (0.25, 0.125, 0.0625, 0.03125)\n",
    "    FEATURE_EXTRACTOR: \"MaskRCNNFPNFeatureExtractor\"\n",
    "    PREDICTOR: \"MaskRCNNC4Predictor\"\n",
    "    POOLER_RESOLUTION: 14\n",
    "    POOLER_SAMPLING_RATIO: 2\n",
    "    RESOLUTION: 28\n",
    "    SHARE_BOX_FEATURE_EXTRACTOR: False\n",
    "  MASK_ON: True\n",
    "DATASETS:\n",
    "  TRAIN: (\"coco_2014_train\", \"coco_2014_valminusminival\")\n",
    "  TEST: (\"coco_2014_minival\",)\n",
    "DATALOADER:\n",
    "  SIZE_DIVISIBILITY: 32\n",
    "SOLVER:\n",
    "  BASE_LR: 0.005\n",
    "  WEIGHT_DECAY: 0.0001\n",
    "  STEPS: (240000, 320000)\n",
    "  MAX_ITER: 360000\n",
    "  IMS_PER_BATCH: 4\n",
    "TEST:\n",
    "  IMS_PER_BATCH: 16\n",
    "' > /workspace/object_detection/configs/custom_config.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Various configurable options within the above configuration file can be modified, for example, the learning rate `BASE_LR`, training batch size `IMS_PER_BATCH` or number of train iterations `MAX_ITER`. The training process will start from a pre-trained Resnet-50 backbone model downloaded from `https://dl.fbaipublicfiles.com/detectron/ImageNetPretrained/MSRA/R-50.pkl`.\n",
    "\n",
    "#### Training with full precision (FP32)\n",
    "Next, we launch the training script using the [custom configuration script](../configs/custom_config.yml) just created above. A full training cycle with 360000 iterations on the COCO-2014 data on a single GPU might take as much as 1.5 days on an NVIDIA V100 GPU. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%run ../tools/train_net.py --config-file \"/workspace/object_detection/configs/custom_config.yml\" DTYPE \"float32\" OUTPUT_DIR ./results/1GPU-FP32/ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Upon completion, the final model is saved to `./results/1GPU-FP32/model_final.pth`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training with mixed-precision\n",
    "Next, we launch the training script using the same [custom configuration script](../configs/custom_config.yml) created above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%run ../tools/train_net.py --config-file \"/workspace/object_detection/configs/custom_config.yml\" DTYPE \"float16\" OUTPUT_DIR ./results/1GPU-FP16/ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On compatible NVIDIA GPUs, the training script makes use of the Tensor cores for higher FP16 arithmetic throughput. On a V100 GPU, this shortens the training time by about 30%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Training with 8 GPUs\n",
    "We will now configure a training script to train a Mask-RCNN model using 8 GPUs. Thanks to having 8 GPUs, we can reduce the number of training iterations by a factor of 8, while the learning rate is also increased by 8 times to account for the larger global batch size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "echo 'MODEL:\n",
    "  META_ARCHITECTURE: \"GeneralizedRCNN\"\n",
    "  WEIGHT: \"catalog://ImageNetPretrained/MSRA/R-50\"\n",
    "  BACKBONE:\n",
    "    CONV_BODY: \"R-50-FPN\"\n",
    "    OUT_CHANNELS: 256\n",
    "  RPN:\n",
    "    USE_FPN: True\n",
    "    ANCHOR_STRIDE: (4, 8, 16, 32, 64)\n",
    "    PRE_NMS_TOP_N_TRAIN: 2000\n",
    "    PRE_NMS_TOP_N_TEST: 1000\n",
    "    POST_NMS_TOP_N_TEST: 1000\n",
    "    FPN_POST_NMS_TOP_N_TEST: 1000\n",
    "    FPN_POST_NMS_TOP_N_TRAIN: 4000\n",
    "  ROI_HEADS:\n",
    "    USE_FPN: True\n",
    "  ROI_BOX_HEAD:\n",
    "    POOLER_RESOLUTION: 7\n",
    "    POOLER_SCALES: (0.25, 0.125, 0.0625, 0.03125)\n",
    "    POOLER_SAMPLING_RATIO: 2\n",
    "    FEATURE_EXTRACTOR: \"FPN2MLPFeatureExtractor\"\n",
    "    PREDICTOR: \"FPNPredictor\"\n",
    "  ROI_MASK_HEAD:\n",
    "    POOLER_SCALES: (0.25, 0.125, 0.0625, 0.03125)\n",
    "    FEATURE_EXTRACTOR: \"MaskRCNNFPNFeatureExtractor\"\n",
    "    PREDICTOR: \"MaskRCNNC4Predictor\"\n",
    "    POOLER_RESOLUTION: 14\n",
    "    POOLER_SAMPLING_RATIO: 2\n",
    "    RESOLUTION: 28\n",
    "    SHARE_BOX_FEATURE_EXTRACTOR: False\n",
    "  MASK_ON: True\n",
    "DATASETS:\n",
    "  TRAIN: (\"coco_2014_train\", \"coco_2014_valminusminival\")\n",
    "  TEST: (\"coco_2014_minival\",)\n",
    "DATALOADER:\n",
    "  SIZE_DIVISIBILITY: 32\n",
    "SOLVER:\n",
    "  BASE_LR: 0.04\n",
    "  WEIGHT_DECAY: 0.0001\n",
    "  STEPS: (36000, 48000)\n",
    "  MAX_ITER: 50000\n",
    "  IMS_PER_BATCH: 32\n",
    "TEST:\n",
    "  IMS_PER_BATCH: 8\n",
    "' > /workspace/object_detection/configs/custom_config_8_GPUs.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training with full precision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -m torch.distributed.launch -- --nproc_per_node=8 ../tools/train_net.py --config-file \"/workspace/object_detection/configs/custom_config_8_GPUs.yml\" DTYPE \"float32\" OUTPUT_DIR ./results/8GPU-FP32/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the Jupyter graphical interface does not update the training progress on the fly, you can observe the information being printed in the shell window from where you launched Jupyter. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training with mixed-precision\n",
    "We now launch the training process using mixed precision. Observe the information being printed in the shell window from where you launched Jupyter. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -m torch.distributed.launch -- --nproc_per_node=8 ../tools/train_net.py --config-file \"/workspace/object_detection/configs/custom_config_8_GPUs.yml\" DTYPE \"float16\" OUTPUT_DIR ./results/8GPU-FP16/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"4\"></a>\n",
    "## 4. Testing trained model\n",
    "\n",
    "After model training has completed, we can test the trained model against the COCO-2014 validation set. First, we create a new configuration file for the test. Note: you must point the model `WEIGHT` parameter to a final model checkpoint, e.g. `./results/8GPU-FP32/model_final.pth`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "echo 'MODEL:\n",
    "  META_ARCHITECTURE: \"GeneralizedRCNN\"\n",
    "  WEIGHT: \"./results/8GPU-FP16/model_final.pth\"\n",
    "  BACKBONE:\n",
    "    CONV_BODY: \"R-50-FPN\"\n",
    "    OUT_CHANNELS: 256\n",
    "  RPN:\n",
    "    USE_FPN: True\n",
    "    ANCHOR_STRIDE: (4, 8, 16, 32, 64)\n",
    "    PRE_NMS_TOP_N_TRAIN: 2000\n",
    "    PRE_NMS_TOP_N_TEST: 1000\n",
    "    POST_NMS_TOP_N_TEST: 1000\n",
    "    FPN_POST_NMS_TOP_N_TEST: 1000\n",
    "  ROI_HEADS:\n",
    "    USE_FPN: True\n",
    "  ROI_BOX_HEAD:\n",
    "    POOLER_RESOLUTION: 7\n",
    "    POOLER_SCALES: (0.25, 0.125, 0.0625, 0.03125)\n",
    "    POOLER_SAMPLING_RATIO: 2\n",
    "    FEATURE_EXTRACTOR: \"FPN2MLPFeatureExtractor\"\n",
    "    PREDICTOR: \"FPNPredictor\"\n",
    "  ROI_MASK_HEAD:\n",
    "    POOLER_SCALES: (0.25, 0.125, 0.0625, 0.03125)\n",
    "    FEATURE_EXTRACTOR: \"MaskRCNNFPNFeatureExtractor\"\n",
    "    PREDICTOR: \"MaskRCNNC4Predictor\"\n",
    "    POOLER_RESOLUTION: 14\n",
    "    POOLER_SAMPLING_RATIO: 2\n",
    "    RESOLUTION: 28\n",
    "    SHARE_BOX_FEATURE_EXTRACTOR: False\n",
    "  MASK_ON: True\n",
    "DATASETS:\n",
    "  TRAIN: (\"coco_2014_train\", \"coco_2014_valminusminival\")\n",
    "  TEST: (\"coco_2014_minival\",)\n",
    "DATALOADER:\n",
    "  SIZE_DIVISIBILITY: 32\n",
    "SOLVER:\n",
    "  BASE_LR: 0.005\n",
    "  WEIGHT_DECAY: 0.0001\n",
    "  STEPS: (240000, 360000)\n",
    "  MAX_ITER: 360000\n",
    "  IMS_PER_BATCH: 4\n",
    "TEST:\n",
    "  IMS_PER_BATCH: 16\n",
    "' > /workspace/object_detection/configs/test_custom_config.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validating on the COCO-2014 mini evaluation data set\n",
    "Next, we launch the evaluation script, which will read the COCO-2014 mini evaluation dataset of 5000 images and evaluate various quality metrics, such as recall, precision and IoU at various thresholds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%run ../tools/test_net.py \\\n",
    "    --config-file /workspace/object_detection/configs/test_custom_config.yml \\\n",
    "    DTYPE \"float16\" \\\n",
    "    DATASETS.TEST \"(\\\"coco_2014_minival\\\",)\" \\\n",
    "    OUTPUT_DIR ./results/8GPU-FP16/evaluation \\\n",
    "    TEST.IMS_PER_BATCH 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing on new images\n",
    "\n",
    "We will now launch an interactive testing, where you can load new test images. First, we load some required libraries and define some helper functions to load images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "attempted relative import beyond top-level package",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-23-1165c7d72063>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     11\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     12\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mmaskrcnn_benchmark\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconfig\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mcfg\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 13\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdemo\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredictor\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mCOCODemo\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     14\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: attempted relative import beyond top-level package"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.pylab as pylab\n",
    "\n",
    "import requests\n",
    "from io import BytesIO\n",
    "from PIL import Image\n",
    "import numpy as np\n",
    "\n",
    "# this makes our figures bigger\n",
    "pylab.rcParams['figure.figsize'] = 20, 20\n",
    "\n",
    "from maskrcnn_benchmark.config import cfg\n",
    "import sys,os\n",
    "sys.path.append(os.path.realpath('..'))\n",
    "from demo.predictor import COCODemo\n",
    "\n",
    "def load(url):\n",
    "    \"\"\"\n",
    "    Given an url of an image, downloads the image and\n",
    "    returns a PIL image\n",
    "    \"\"\"\n",
    "    response = requests.get(url)\n",
    "    pil_image = Image.open(BytesIO(response.content)).convert(\"RGB\")\n",
    "    # convert to BGR format\n",
    "    image = np.array(pil_image)[:, :, [2, 1, 0]]\n",
    "    return image\n",
    "\n",
    "def imshow(img):\n",
    "    plt.imshow(img[:, :, [2, 1, 0]])\n",
    "    plt.axis(\"off\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we load the trained model specified in the test configuration file, e.g. from `./results/8GPU-FP32/model_final.pth`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "config_file = \"/workspace/object_detection/configs/test_custom_config.yml\"\n",
    "\n",
    "# update the config options with the config file\n",
    "cfg.merge_from_file(config_file)\n",
    "# manual override some options\n",
    "cfg.merge_from_list([\"MODEL.DEVICE\", \"cuda\"])\n",
    "\n",
    "coco_demo = COCODemo(\n",
    "    cfg,\n",
    "    min_image_size=800,\n",
    "    confidence_threshold=0.7,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "User now can load a test image from any public URL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 1199172 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# from http://cocodataset.org/#explore?id=345434\n",
    "image = load(\"http://farm3.staticflickr.com/2469/3915380994_2e611b1779_z.jpg\")\n",
    "imshow(image)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The prediction result is then displayed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 1195588 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# compute predictions\n",
    "predictions = coco_demo.run_on_opencv_image(image)\n",
    "imshow(predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "In this notebook, we have walked through the complete process of preparing the container and data required for training Mask-RCNN models. We have also investigated various training options, trained and tested Mask-RCNN models with various configurations.\n",
    "\n",
    "## What's next\n",
    "Now it's time to try the MaskR-CNN on your own data. Observe the performance impact of mixed precision training while comparing the final accuracy of the models trained with FP32 and mixed precision.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
