Deep Learning — RNN (3)

Preface: In language, words have temporal order, so RNNs excel at language tasks. Below we build a minimal RNN to predict the next word from a few given words. The key is converting between words and indices. Full code: https://github.com/dctongsheng/vocab_predict_rnn

Reading data:

def read_data(filename):
    with open(filename) as f:
        content = f.readlines()
        #去掉空格
        content = [x.strip() for x in content]
        #得到单词
        content_size = len(content)
        print(content_size)
        words = [content[i].split() for i in range(content_size)]
        words = np.array(words)
        # 将格式转换，认为一行一个样本，一个样本中的序列长度未知，每个时刻一个对应的单词或者符号
        words = np.reshape(words,[-1,])

Building a word dictionary:

def build_dataset(words):
    count = collections.Counter(words).most_common()
    print(count)
    dicts = {}
    k=0
    # return count
    for word,_ in count:
        dicts[word] = k
        k+=1
    reves_dict = dict(zip(dicts.values(),dicts.keys()))
    return dicts,reves_dict

Network: deep RNN, static input:

def rnn(x):
    x = tf.reshape(x,[-1,n_inputs])
    x = tf.split(x,n_inputs,1)
    rnn_cell = tf.nn.rnn_cell.MultiRNNCell(cells=[tf.nn.rnn_cell.LSTMCell(num_units=n_hidden1),
                                                  tf.nn.rnn_cell.GRUCell(num_units=n_hidden2),
                                                  tf.nn.rnn_cell.BasicRNNCell(num_units=n_hidden3)])
    output,state = tf.nn.static_rnn(rnn_cell,x,dtype=tf.float32)
    output=output[-1]
    return tf.matmul(output, weights['out']) + biases['out']

Loss:

pred = rnn(x)
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred,labels=y))
train = tf.train.RMSPropOptimizer(learning_rate=learn_rate).minimize(cost)

Accuracy:

cp = tf.equal(tf.argmax(pred,1),tf.argmax(y,1))
accury = tf.reduce_mean(tf.cast(cp,tf.float32))

Training—constructing x and y:

keys = [dicts[str(train_data[i])] for i in range(offset,offset+n_inputs)]
        keys = np.reshape(np.array(keys), [-1, n_inputs, 1])
        #构建y
        out_one_hot = np.zeros([vocab_size],dtype=np.float32)
        out_one_hot[dicts[str(train_data[offset+n_inputs])]] =1.0
        out_one_hot = np.reshape(out_one_hot,[1,-1])
        # 训练
        _,_acc,_cost,_pred=sess.run([train,accury,cost,pred],feed_dict={x:keys,y:out_one_hot})

Output:

After 10,000 words, accuracy rises and three consecutive predictions are correct.