type
status
date
slug
password
summary
tags
category
icon
Recurrent Neural Networks with Hidden States
Recall that we have discussed hidden layers with hidden units in Section 5. It is noteworthy that hidden layers and hidden states refer to two very different concepts. Hidden layers are, as explained, layers that are hidden from view on the path from input to output. Hidden states are technically speaking inputs to whatever we do at a given step, and they can only be computed by looking at data at previous time steps.
Recurrent neural networks (RNNs) are neural networks with hidden states. Before introducing the RNN model, we first revisit the MLP
The output of the output layer is similar to the computation in the MLP:
begin with, we define matrices
X, W_xh, H, and W_hh, whose shapes are (3, 1), (1, 4), (3, 4), and (4, 4), respectively. Multiplying X by W_xh, and H by W_hh, and then adding these two products, we obtain a matrix of shape (3, 4).Now we concatenate the matrices
X and H along columns (axis 1), and the matrices W_xh and W_hh along rows (axis 0). These two concatenations result in matrices of shape (3, 5) and of shape (5, 4), respectively. Multiplying these two concatenated matrices, we obtain the same output matrix of shape (3, 4) as above.
Implementation
Concise Implementation of Recurrent Neural Networks
上一篇
Deep learning Guide 7: Long Short-Term Memory (LSTM), Gated Recurrent Units (GRU)
下一篇
Deep learning Guide 5: Sequence Models, Language Models
- Author:tom-ci
- URL:https://www.tomciheng.com//article/d2lv-6
- Copyright:All articles in this blog, except for special statements, adopt BY-NC-SA agreement. Please indicate the source!