Deep Learning — RNN (2)

Preface: After introducing LSTM, below are several LSTM variants.

Bidirectional RNN

Bidirectional RNN assumes output at time t depends not only on the past sequence but also on the future—for example, predicting a missing word from context. Bidirectional RNN is relatively simple: two RNNs stacked; output comes from the hidden states of both.

As shown:

Network construction code:

# 开始网络构建
    # 1. 输入的数据格式转换
    # X格式：[batch_size, time_steps, input_size]
    X = tf.reshape(_X, shape=[-1, timestep_size, input_size])

    # 单层LSTM RNN
    # 2. 定义Cell
    lstm_cell_fw = tf.nn.rnn_cell.LSTMCell(num_units=hidden_size, reuse=tf.get_variable_scope().reuse)
    gru_cell_bw = tf.nn.rnn_cell.GRUCell(num_units=hidden_size, reuse=tf.get_variable_scope().reuse)

    # 3. 单层的RNN网络应用
    init_state_fw = lstm_cell_fw.zero_state(batch_size, dtype=tf.float32)
    init_state_bw = gru_cell_bw.zero_state(batch_size, dtype=tf.float32)

    # 3. 动态构建双向的RNN网络
    """
    bidirectional_dynamic_rnn(
        cell_fw: 前向的rnn cell
        , cell_bw：反向的rnn cell
        , inputs：输入的序列
        , sequence_length=None
        , initial_state_fw=None：前向rnn_cell的初始状态
        , initial_state_bw=None：反向rnn_cell的初始状态
        , dtype=None
        , parallel_iterations=None
        , swap_memory=False, time_major=False, scope=None)
    API返回值：(outputs, output_states) => outputs存储网络的输出信息，output_states存储网络的细胞状态信息
    outputs: 是一个二元组, (output_fw, output_bw)构成，output_fw对应前向的rnn_cell的执行结果，结构为：[batch_size, time_steps, output_size];output_bw对应反向的rnn_cell的执行结果，结果和output_bw一样
    output_states：是一个二元组，(output_state_fw, output_state_bw) 构成，output_state_fw和output_state_bw是dynamic_rnn API输出的状态值信息
    """
    outputs, states = tf.nn.bidirectional_dynamic_rnn(
        cell_fw=lstm_cell_fw, cell_bw=gru_cell_bw, inputs=X,
        initial_state_fw=init_state_fw, initial_state_bw=init_state_bw)
    output_fw = outputs[0][:, -1, :]
    output_bw = outputs[1][:, -1, :]
    output = tf.concat([output_fw, output_bw], 1)

Deep RNN

Deep Bidirectional RNN is like Bidirectional RNN, but each step has multiple layers—stronger expressiveness and learning, higher complexity, and more training data needed.

Deep RNN construction code:

#多层
    def lstm_call():
        cell = tf.nn.rnn_cell.LSTMCell(num_units=hidden_size,reuse=tf.get_variable_scope().reuse)
        return tf.nn.rnn_cell.DropoutWrapper(cell,output_keep_prob=keep_prob)
    mlstm_cell = tf.nn.rnn_cell.MultiRNNCell(cells=[lstm_call() for i in range(layer_num)])
    inint_state = mlstm_cell.zero_state(batch_size,tf.float32)
    output,state = tf.nn.dynamic_rnn(mlstm_cell,inputs=X,initial_state=inint_state)
    output = output[:,-1,:]

Variants

• Add “peephole connections” so gates also take cell state as input.

• Couple forget and input gates (first and second gates)—forget and add are considered together instead of separately.

• Gated Recurrent Unit (GRU), proposed in 2014: merges forget and input into one update gate; merges cell state and hidden state; simpler than LSTM.

See the paper: http://arxiv.org/pdf/1402.1128v1.pdf