{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Gwt7z7qdmTbW"
   },
   "outputs": [],
   "source": [
    "# Copyright 2019 NVIDIA Corporation. All Rights Reserved.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "# =============================================================================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "i4NKCp2VmTbn"
   },
   "source": [
    "<img src=\"http://developer.download.nvidia.com/compute/machine-learning/frameworks/nvidia_logo.png\" style=\"width: 90px; float: right;\">\n",
    "\n",
    "# DLRM Training and Inference Demo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fW0OKDzvmTbt"
   },
   "source": [
    "## Overview\n",
    "\n",
    "\n",
    "DLRM is a deep learning based approach to recommendation introduced by Facebook. \n",
    "Like other deep learning based approaches, DLRM is designed to make use of both categorical and numerical inputs which are usually present in RecSys training data. The architecture of DLRM can be understood via Figure 1. In order to handle categorical data, embedding layers map each category to a dense representation before being fed into  dense multilayer perceptrons (MLP). Continuous features can be fed directly into a dense MLP. At the next level, second-order interactions of different features are computed explicitly by taking the dot product between all pairs of embedding vectors and processed dense features. Those pairwise interactions are fed into a top level MLP to compute the likelihood of interaction between users and items. \n",
    "\n",
    "Compared to other DL based approaches to recommendation, DLRM differs in two ways. First, DLRM computes the feature interaction explicitly while limiting the order of interaction to pairwise interactions. Second, DLRM treats each embedded feature vector (corresponding to categorical features) as a single unit, whereas other methods treat each element  in the feature vector as a new unit that should yield different cross terms. These design choices help reduce computational/memory cost while maintaining competitive accuracy.\n",
    "\n",
    "![DLRM_model](DLRM_architecture.png)\n",
    "\n",
    "Figure 1. DLRM architecture.\n",
    "\n",
    "### Learning objectives\n",
    "\n",
    "This notebook demonstrates the steps for training a DLRM model. We then employ the trained model to make inference on new data.\n",
    "\n",
    "## Content\n",
    "1. [Requirements](#1)\n",
    "1. [Data download and preprocessing](#2)\n",
    "1. [Training](#3)\n",
    "1. [Testing trained model](#4)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aDFrE4eqmTbv"
   },
   "source": [
    "<a id=\"1\"></a>\n",
    "## 1. Requirements\n",
    "\n",
    "\n",
    "### 1.1 Docker container\n",
    "The most convenient way to make use of the NVIDIA DLRM model is via a docker container, which provides a self-contained, isolated and re-producible environment for all experiments. Refer to the [Quick Start Guide section](../README.md) of the Readme documentation for a comprehensive guide. We briefly summarize the steps here.\n",
    "\n",
    "First, clone the repository:\n",
    "\n",
    "```\n",
    "git clone https://github.com/NVIDIA/DeepLearningExamples\n",
    "cd DeepLearningExamples/PyTorch/Recommendation/DLRM\n",
    "```\n",
    "\n",
    "Next, build the DLRM container:\n",
    "```\n",
    "docker build . -t nvidia_dlrm_pyt\n",
    "```\n",
    "\n",
    "Make a directory for storing DLRM data and start a docker container with:\n",
    "```\n",
    "mkdir -p data\n",
    "docker run --runtime=nvidia -it --rm --ipc=host  -v ${PWD}/data:/data nvidia_dlrm_pyt bash\n",
    "```\n",
    "\n",
    "Within the docker interactive bash session, start Jupyter with\n",
    "\n",
    "```\n",
    "export PYTHONPATH=/workspace/dlrm\n",
    "jupyter notebook --ip 0.0.0.0 --port 8888\n",
    "```\n",
    "\n",
    "Then open the Jupyter GUI interface on your host machine at http://localhost:8888. Within the container, the demo notebooks are located at `/workspace/dlrm/notebooks`.\n",
    "\n",
    "### 1.2 Hardware\n",
    "This notebook can be executed on any CUDA-enabled NVIDIA GPU with at least 24GB of GPU memory, although for efficient mixed precision training, a [Tensor Core NVIDIA GPU](https://www.nvidia.com/en-us/data-center/tensorcore/) is desired (Volta, Turing or newer architectures). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "k7RLEcKhmTb0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sat Mar 28 06:36:59 2020       \n",
      "+-----------------------------------------------------------------------------+\n",
      "| NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |\n",
      "|-------------------------------+----------------------+----------------------+\n",
      "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
      "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
      "|===============================+======================+======================|\n",
      "|   0  Tesla V100-SXM2...  On   | 00000000:06:00.0 Off |                    0 |\n",
      "| N/A   32C    P0    42W / 300W |      0MiB / 32510MiB |      0%      Default |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "|   1  Tesla V100-SXM2...  On   | 00000000:07:00.0 Off |                    0 |\n",
      "| N/A   34C    P0    43W / 300W |      0MiB / 32510MiB |      0%      Default |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "|   2  Tesla V100-SXM2...  On   | 00000000:0A:00.0 Off |                    0 |\n",
      "| N/A   34C    P0    43W / 300W |      0MiB / 32510MiB |      0%      Default |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "|   3  Tesla V100-SXM2...  On   | 00000000:0B:00.0 Off |                    0 |\n",
      "| N/A   32C    P0    43W / 300W |      0MiB / 32510MiB |      0%      Default |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "|   4  Tesla V100-SXM2...  On   | 00000000:85:00.0 Off |                    0 |\n",
      "| N/A   33C    P0    43W / 300W |      0MiB / 32510MiB |      0%      Default |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "|   5  Tesla V100-SXM2...  On   | 00000000:86:00.0 Off |                    0 |\n",
      "| N/A   35C    P0    44W / 300W |      0MiB / 32510MiB |      0%      Default |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "|   6  Tesla V100-SXM2...  On   | 00000000:89:00.0 Off |                    0 |\n",
      "| N/A   37C    P0    44W / 300W |      0MiB / 32510MiB |      0%      Default |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "|   7  Tesla V100-SXM2...  On   | 00000000:8A:00.0 Off |                    0 |\n",
      "| N/A   34C    P0    43W / 300W |      0MiB / 32510MiB |      0%      Default |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "                                                                               \n",
      "+-----------------------------------------------------------------------------+\n",
      "| Processes:                                                       GPU Memory |\n",
      "|  GPU       PID   Type   Process name                             Usage      |\n",
      "|=============================================================================|\n",
      "|  No running processes found                                                 |\n",
      "+-----------------------------------------------------------------------------+\n"
     ]
    }
   ],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HqSUGePjmTb9"
   },
   "source": [
    "<a id=\"2\"></a>\n",
    "## 2. Data download and preprocessing\n",
    "\n",
    "Commercial recommendation systems are often trained on huge data sets, often in the order of terabytes, if not more. While datasets of this scale are rarely available to the public, the Criteo Terabyte click logs public [dataset](https://labs.criteo.com/2013/12/download-terabyte-click-logs/) offers a rare glimpse into the scale of real enterprise data: it contains ~1.3TB of uncompressed click logs collected over the course of 24 days, that can be used to train RecSys models that predict the ads click through rate. Yet, real datasets can be potentially one or two orders of magnitude larger, as enterprises will try to leverage as much historical data as they can use, for this will generally translate into better accuracy.\n",
    "\n",
    "Herein, we employ the Criteo Terabyte dataset to demonstrate the efficiency of the GPU-optimized DLRM training procedure.  Each record in this dataset contains 40 columns: the first is a label column that indicates whether an user clicks an ad (value 1) or not (value 0). The next 13 columns are numeric, and the last 26 are categorical columns containing obfuscated hashed values. The columns and their values are all anonymized to protect user privacy.\n",
    "\n",
    "\n",
    "We will first download and preprocess the Criteo Terabyte dataset. Note that this will require about 1TB of disk storage.\n",
    "\n",
    "Notice: before downloading data, you must check out and agree with the terms and conditions of the Criteo Terabyte [dataset](https://labs.criteo.com/2013/12/download-terabyte-click-logs/).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "S2PR7weWmTcK"
   },
   "outputs": [],
   "source": [
    "! cd ../preproc && ./prepare_dataset.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EQAIszkxmTcT"
   },
   "source": [
    "The original Facebook DLRM code base comes with a data preprocessing utility to preprocess the data. For continuous features, the data preprocessing steps include filling in missing values with 0 and normalization (shifting the values to be >=1 and taking natural logarithm). For categorical features, the preprocessing steps include building embedding tables and transforming hashed values into integer indicators. This code runs on a single CPU thread and takes ~6.5 days to transform the whole Criteo Terabyte data set. \n",
    "\n",
    "We improve the data preprocessing process with Spark on CPU to make use of all CPU threads. In the docker image, we have installed spark 2.4.5, which we’ll start a standalone Spark cluster.This results in significant improvement in data pre-processing speed, scaling approximately linearly with the number of available CPU threads. This outputs the transformed data in parquet format. We finally convert the parquet data into the binary format similar to that designed by the Facebook team specially for the Criteo dataset. \n",
    "\n",
    "Our preprocessing scripts are designed for the Criteo Terabyte Dataset and should work with any other dataset with the same format. The data should be split into text files. Each line of those text files should contain a single training example. An example should consist of multiple fields separated by tabulators:\n",
    "- The first field is the label – `1` for a positive example and `0` for negative.\n",
    "- The next `N` tokens should contain the numerical features separated by tabs.\n",
    "- The next `M` tokens should contain the hashed categorical features separated by tabs.\n",
    "\n",
    "The outcomes of the data preprocessing steps are by default stored in `/data/dlrm/binary_dataset` containing 3 binary data files: `test_data.bin`, `train_data.bin` and   `val_data.bin`  and a JSON `file model_size.json` totalling ~650GB.\n",
    "\n",
    "Tips: by defaul the preprocessing script uses the first 23 days of the Criteo Terabyte dataset for training and the last day for validation. For a quick experiment, you can download and make use of a smaller number of days by modifying the `preproc/run_spark.sh` script."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RL8d9IwzmTcV"
   },
   "source": [
    "<a id=\"3\"></a>\n",
    "## 3. Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "o6wayGf1mTcX"
   },
   "source": [
    "The repository provides several training recipes on 1 GPU with FP32 and automatic mixed precisions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HapDsY4VmTce"
   },
   "source": [
    "#### Training with FP32\n",
    "Training on 1 GPU with FP32 with the `--nofp16` option."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%run ../dlrm/scripts/main \\\n",
    "--mode train \\\n",
    "--dataset /data/dlrm/binary_dataset \\\n",
    "--nofp16 \\\n",
    "--save_checkpoint_path ./dlrm_model_fp32.pt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On a V100 32GB, training takes approximately 2h56m for 1 epoch to an AUC of ~0.8. The final result should look similar to the below.\n",
    "\n",
    "```\n",
    "Epoch:[0/1] [127600/128028]  eta: 0:00:34  loss: 0.1226  step_time: 0.080038  lr: 1.1766\n",
    "Epoch:[0/1] [127800/128028]  eta: 0:00:18  loss: 0.1224  step_time: 0.080307  lr: 1.1480\n",
    "Epoch:[0/1] [128000/128028]  eta: 0:00:02  loss: 0.1221  step_time: 0.080562  lr: 1.1199\n",
    "Test: [200/2721]  loss: 0.1236  step_time: 0.0303\n",
    "Test: [400/2721]  loss: 0.1248  step_time: 0.0245\n",
    "Test: [600/2721]  loss: 0.1262  step_time: 0.0244\n",
    "Test: [800/2721]  loss: 0.1262  step_time: 0.0245\n",
    "Test: [1000/2721]  loss: 0.1293  step_time: 0.0245\n",
    "Test: [1200/2721]  loss: 0.1307  step_time: 0.0245\n",
    "Test: [1400/2721]  loss: 0.1281  step_time: 0.0245\n",
    "Test: [1600/2721]  loss: 0.1242  step_time: 0.0246\n",
    "Test: [1800/2721]  loss: 0.1230  step_time: 0.0245\n",
    "Test: [2000/2721]  loss: 0.1226  step_time: 0.0244\n",
    "Test: [2200/2721]  loss: 0.1239  step_time: 0.0246\n",
    "Test: [2400/2721]  loss: 0.1256  step_time: 0.0249\n",
    "Test: [2600/2721]  loss: 0.1247  step_time: 0.0248\n",
    "Epoch 0 step 128027. Test loss 0.12557, auc 0.803517\n",
    "Checkpoint saving took 42.90 [s]\n",
    "DLL 2020-03-29 15:59:44.759627 - () best_auc : 0.80352  best_epoch : 1.00  average_train_throughput : 4.07e+05  average_test_throughput : 1.33e+06 \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "j-aFEwb4mTcn"
   },
   "source": [
    "#### Training with mixed-precision\n",
    "Mixed precision training can be done with the `--fp16` option. Under the hood, the NVIDIA Pytorch extension library [Apex](https://github.com/NVIDIA/apex) to enable mixed precision training.\n",
    "\n",
    "Note: for subsequent launches of the %run magic, please restart your kernel manualy or execute the below cell to restart kernel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note: for subsequent launches of the %run magic, \n",
    "# please restart your kernel manualy or execute this cell to restart kernel.\n",
    "import os\n",
    "os._exit(00)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "o3AZ-CXYmTcp",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%run ../dlrm/scripts/main \\\n",
    "--mode train \\\n",
    "--dataset /data/dlrm/binary_dataset \\\n",
    "--fp16 \\\n",
    "--save_checkpoint_path ./dlrm_model_fp16.pt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On a V100 32GB, training takes approximately 1h41m for 1 epoch to an AUC of ~0.8. Thus, mixed precision training provides a speed up of ~ 1.7x.\n",
    "\n",
    "The final result should look similar to the below.\n",
    "\n",
    "```\n",
    "...\n",
    "Epoch:[0/1] [127800/128028]  eta: 0:00:11  loss: 0.1224  step_time: 0.050719  lr: 1.1480\n",
    "Epoch:[0/1] [128000/128028]  eta: 0:00:01  loss: 0.1221  step_time: 0.050499  lr: 1.1199\n",
    "Test: [200/2721]  loss: 0.1236  step_time: 0.0271\n",
    "Test: [400/2721]  loss: 0.1247  step_time: 0.0278\n",
    "Test: [600/2721]  loss: 0.1262  step_time: 0.0275\n",
    "Test: [800/2721]  loss: 0.1262  step_time: 0.0278\n",
    "Test: [1000/2721]  loss: 0.1293  step_time: 0.0273\n",
    "Test: [1200/2721]  loss: 0.1306  step_time: 0.0264\n",
    "Test: [1400/2721]  loss: 0.1281  step_time: 0.0281\n",
    "Test: [1600/2721]  loss: 0.1242  step_time: 0.0273\n",
    "Test: [1800/2721]  loss: 0.1229  step_time: 0.0280\n",
    "Test: [2000/2721]  loss: 0.1226  step_time: 0.0274\n",
    "Test: [2200/2721]  loss: 0.1239  step_time: 0.0278\n",
    "Test: [2400/2721]  loss: 0.1256  step_time: 0.0289\n",
    "Test: [2600/2721]  loss: 0.1247  step_time: 0.0282\n",
    "Epoch 0 step 128027. Test loss 0.12557, auc 0.803562\n",
    "Checkpoint saving took 40.46 [s]\n",
    "DLL 2020-03-28 15:15:36.290149 - () best_auc : 0.80356  best_epoch : 1.00  average_train_throughput : 6.47e+05  average_test_throughput : 1.17e+06\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "X959LYwjmTcw"
   },
   "source": [
    "<a id=\"4\"></a>\n",
    "## 4. Testing trained model\n",
    "\n",
    "After model training has completed, we can test the trained model against the Criteo test dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note: for subsequent launches of the %run magic, \n",
    "# please restart your kernel manualy or execute this cell to restart kernel.\n",
    "import os\n",
    "os._exit(00)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%run ../dlrm/scripts/main \\\n",
    "--mode test\\\n",
    "--dataset /data/dlrm/binary_dataset \\\n",
    "--load_checkpoint_path ./dlrm_model_fp16.pt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "g8MxXY5GmTc8"
   },
   "source": [
    "# Conclusion\n",
    "\n",
    "In this notebook, we have walked through the complete process of preparing the container and data required for training the DLRM model. We have also investigated various training options with FP32 and automatic mixed precision, trained and tested DLRM models with new test data.\n",
    "\n",
    "## What's next\n",
    "Now it's time to try the DLRM model on your own data. Observe the performance impact of mixed precision training while comparing the final accuracy of the models trained with FP32 and mixed precision.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "249yGNLmmTc_"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "include_colab_link": true,
   "name": "TensorFlow_UNet_Industrial_Colab_train_and_inference.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
