\begin{frame}
  \myheading{Module 15.1: Selective Read, Selective Write, Selective Forget - The Whiteboard Analogy}
\end{frame}

\begin{frame}
  \begin{columns}
    \column{0.5\textwidth}
      \begin{overlayarea}{\textwidth}{\textheight}
        \input{modules/Module1/tikz_images/rnn.tex}
      \end{overlayarea}

    \column{0.5\textwidth}
      \begin{overlayarea}{\textwidth}{\textheight}
      \begin{itemize}\justifying
      \item<1-> The state ($s_i$) of an RNN records information from all previous time steps
      \item<2-> At each new timestep the old information gets morphed by the current input
      \item<7-> One could imagine that after $t$ steps the information stored at time step $t - k$ (for some $k<t$) gets completely morphed
      \item[]<8-> so much that it would be impossible to extract the original information stored at time step $t-k$
      \end{itemize}
      \end{overlayarea}
  \end{columns}
\end{frame}

\begin{frame}
  \begin{columns}
    \column{0.5\textwidth}
      \begin{overlayarea}{\textwidth}{\textheight}
        \input{modules/Module1/tikz_images/rnn_back.tex}
      \end{overlayarea}

    \column{0.5\textwidth}
      \begin{overlayarea}{\textwidth}{\textheight}
      \begin{itemize}\justifying
      \item<1-> A similar problem occurs when the information flows backwards (backpropagation)
      \item<2-> It is very hard to assign the responsibility of the error caused at time step $t$ to the events that occurred at time step $t-k$
      \item<3-> This responsibility is of course in the form of gradients and we studied the problem in backward flow of gradients
      \item<4-> We saw a formal argument for this while discussing vanishing gradients
      \end{itemize}
      \end{overlayarea}
  \end{columns}
\end{frame}


\begin{frame}
  \begin{columns}
    \column{0.5\textwidth}
      \begin{overlayarea}{\textwidth}{\textheight}

      \begin{figure}
      \includegraphics[scale=0.3]{images/messy_whiteboard}
      \end{figure}
      \end{overlayarea}

    \column{0.5\textwidth}
      \begin{overlayarea}{\textwidth}{\textheight}
      \begin{itemize}\justifying
      \item<1-> Let us see an analogy for this
      \item<2-> We can think of the state as a fixed size memory
      \item<3-> Compare this to a fixed size white board that you use to record information
      \item<4-> At each time step (periodic intervals) we keep writing something to the board
      \item<5-> Effectively at each time step we morph the information recorded till that time point
      \item<6-> After many timesteps it would be impossible to see how the information at time step $t-k$ contributed to the state at timestep $t$
      \end{itemize}
      \end{overlayarea}
  \end{columns}
\end{frame}


\begin{frame}
  \begin{columns}
    \column{0.5\textwidth}
      \begin{overlayarea}{\textwidth}{\textheight}
      \begin{figure}
      \includegraphics[scale=0.3]{images/messy_whiteboard}
      \end{figure}
      \end{overlayarea}

    \column{0.5\textwidth}
      \begin{overlayarea}{\textwidth}{\textheight}
      \begin{itemize}\justifying
      \item<1-> Continuing our whiteboard analogy, suppose we are interested in deriving an expression on the whiteboard
      \item<2-> We follow the following strategy at each time step
      \item<3-> Selectively write on the board
      \item<4-> Selectively read the already written content
      \item<5-> Selectively forget (erase) some content
      \item<6-> Let us look at each of these in detail
      \end{itemize}
      \end{overlayarea}
  \end{columns}
\end{frame}




\begin{frame}
  \begin{columns}
    \column{0.5\textwidth}
      \begin{overlayarea}{\textwidth}{\textheight}
    \vspace{-1em}
    \only<3->{
    \[ a = 1 \quad b = 3 \quad c = 5 \quad d = 11
    \]
    \[
      \textbf{Compute } ac(bd + a) + ad
    \]
    Say ``board'' can have only $3$ statements at a time.
    \begin{center}
    \begin{enumerate}
    \item $ac$
    \item $bd$
    \item $bd + a$
    \item $ac(bd + a)$
    \item $ad$
    \item $ac(bd + a) + ad$
    \end{enumerate}
    \end{center}
    \vspace{-2em}
    \only<4->{
      \begin{align*}
        \fakealign
      \onslide<4->{
        ac &= 5 \\
      }
      \onslide<5->{
        bd &= 33 \\
      }
    \end{align*}
    }
    }
      \end{overlayarea}

    \column{0.5\textwidth}
      \begin{overlayarea}{\textwidth}{\textheight}
      \textbf{Selective write}
      \begin{itemize}\justifying
      \item<1-> There may be many steps in the derivation but we may just skip a few
      \item<2-> In other words we select what to \textbf{write}
      \end{itemize}
      \end{overlayarea}
  \end{columns}
\end{frame}

\begin{frame}
  \begin{columns}
    \column{0.5\textwidth}
      \begin{overlayarea}{\textwidth}{\textheight}
    \vspace{-1em}
    \[ a = 1 \quad b = 3 \quad c = 5 \quad d = 11
    \]
    \[
      \textbf{Compute } ac(bd + a) + ad
    \]
    Say ``board'' can have only $3$ statements at a time.
    \begin{center}
    \begin{enumerate}
    \item $ac$ %- \textbf{Done}
    \item $bd$ %- \textbf{Done}
    \item $bd + a$
    \item $ac(bd + a)$
    \item $ad$
    \item $ac(bd + a) + ad$
    \end{enumerate}
    \end{center}
    \vspace{-2em}
    %\only<1-3>{
    % \begin{align*}
    %     ac &= 5 \\
    %     bd &= 33 \\
    % \end{align*}
    %}
    \only<1->{
      \begin{align*}
       \fakealign
      \onslide<1->{
        ac &= 5 \\
      }
      \onslide<1->{
        bd &= 33 \\
      }
      \onslide<4->{
        bd + a &= 34
      }
    \end{align*}
    }
      \end{overlayarea}

    \column{0.5\textwidth}
      \begin{overlayarea}{\textwidth}{\textheight}
      \textbf{Selective read}
      \begin{itemize}\justifying
      \item<1-> While writing one step we typically read some of the previous steps we have already written and then decide what to write next
      \item<2-> For example at Step 3, information from Step 2 is important %and Step 3 is important
      \item<3-> In other words we select what to \textbf{read}
      \end{itemize}
      \end{overlayarea}
  \end{columns}
\end{frame}

\begin{frame}
  \begin{columns}
    \column{0.5\textwidth}
      \begin{overlayarea}{\textwidth}{\textheight}
    \vspace{-1em}
    \[ a = 1 \quad b = 3 \quad c = 5 \quad d = 11
    \]
    \[
      \textbf{Compute } ac(bd + a) + ad
    \]
    Say ``board'' can have only $3$ statements at a time.
    \begin{center}
    \begin{enumerate}
    \item $ac$ % - \textbf{Done}
    \item $bd$ %- \textbf{Done}
    \item $bd + a$ %- \textbf{Done}
    \item $ac(bd + a)$
    \item $ad$
    \item $ac(bd + a) + ad$
    \end{enumerate}
    \end{center}
    \vspace{-4.3em}
    \begin{center}
    \only<1-3>{
      \begin{align*}
       \fakealign
        ac &= 5 \\
        bd &= 33 \\
        bd + a &= 34
    \end{align*}
    }
    \only<4>{
      \begin{align*}
        \fakealign
        ac &= 5\\
        \\
        bd + a &= 34
    \end{align*}
    }
    \only<5>{
      \begin{align*}
       \fakealign
        ac &=5\\
        ac(bd + a) &= 170 \\
        bd + a &= 34
    \end{align*}

    }
    \only<6>{
      \begin{align*}
      \fakealign
        ac & = 5 \\
        ac(bd + a) &= 170 \\
        ad &= 11
    \end{align*}
    }
    \only<7>{
      \begin{align*}
      \fakealign
        ad + ac(bd + a) &= 181 \\
        ac(bd + a) &= 170 \\
        ad &= 11
    \end{align*}
    }
    \end{center}
      \end{overlayarea}

    \column{0.5\textwidth}
      \begin{overlayarea}{\textwidth}{\textheight}
      \textbf{Selective forget}
      \begin{itemize}\justifying
      \item<1-> Once the board is full, we need to delete some obsolete information
      \item<2-> But how do we decide what to delete? We will typically delete the least useful information
      \item<3-> In other words we select what to \textbf{forget}
      \end{itemize}
      \end{overlayarea}
  \end{columns}
\end{frame}
\begin{frame}
  \begin{columns}
    \column{0.5\textwidth}
      \begin{overlayarea}{\textwidth}{\textheight}
    \vspace{-1em}
    \[ a = 1 \quad b = 3 \quad c = 5 \quad d = 11
    \]
    \[
      \textbf{Compute } ac(bd + a) + ad
    \]
    Say ``board'' can have only $3$ statements at a time.
    \begin{center}
    \begin{enumerate}
    \item $ac$ % - \textbf{Done}
    \item $bd$ %- \textbf{Done}
    \item $bd + a$ %- \textbf{Done}
    \item $ac(bd + a)$
    \item $ad$
    \item $ac(bd + a) + ad$
    \end{enumerate}
    \end{center}
    \vspace{-4.3em}
    \begin{center}
      \begin{align*}
      \fakealign
        ad + ac(bd + a) &= 181 \\
        ac(bd + a) &= 170 \\
        ad &= 11
    \end{align*}
    \end{center}
      \end{overlayarea}

    \column{0.5\textwidth}
      \begin{overlayarea}{\textwidth}{\textheight}
      \begin{itemize}\justifying
      \item<1-> There are various other scenarios where we can motivate the need for selective write, read and forget
      \item<2-> For example, you could think of our brain as something which can store only a finite number of facts
      \item<3-> At different time steps we selectively read, write and forget some of these facts
      \item<4-> Since the RNN also has a finite state size, we need to figure out a way to allow it to selectively read, write and forget
      \end{itemize}
      \end{overlayarea}
  \end{columns}
\end{frame}
