前向传播（张量）- 实战

• MNIST手写数字集7000*10张图片
• 60k张图片训练，10k张图片测试
• 每张图片是28*28，如果是彩色图片是28*28*3
• 0-255表示图片的灰度值，0表示纯白，255表示纯黑
• 打平28*28的矩阵，得到28*28=784的向量
• 对于b张图片得到[b,784];然后对于b张图片可以给定编码
• 把上述的普通编码给定成独热编码，但是独热编码都是概率值，并且概率值相加为1，类似于softmax回归
• 套用线性回归公式
• X[b,784] W[784,10] b[10] 得到 [b,10]
• 高维图片实现非常复杂，一个线性模型无法完成，因此可以添加非线性因子
• f(X@W+b)，使用激活函数让其非线性化，引出relu函数
• 用了激活函数，模型还是太简单
• 使用工厂
  • H1 =relu(X@W1+b1)
  • H2 = relu(h1@W2+b2)
  • Out = relu(h2@W3+b3)
• 第一步，把[1,784]变成[1,512]变成[1,256]变成[1,10]
• 得到[1,10]后将结果进行独热编码
• 使用欧氏距离或者使用mse进行误差度量
• [1,784]通过三层网络输出一个[1,10]

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import datasets
import os

# do not print irrelevant information
# os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'

# x: [60k,28,28]
# y: [60k]
(x, y), _ = datasets.mnist.load_data()

Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 1s 0us/step

# transform Tensor
# x: [0~255] ==》 [0~1.]
x = tf.convert_to_tensor(x, dtype=tf.float32) / 255.
y = tf.convert_to_tensor(y, dtype=tf.int32)

f'x.shape: {x.shape}, y.shape: {y.shape}, x.dtype: {x.dtype}, y.dtype: {y.dtype}'

"x.shape: (60000, 28, 28), y.shape: (60000,), x.dtype: <dtype: 'float32'>, y.dtype: <dtype: 'int32'>"

f'min_x: {tf.reduce_min(x)}, max_x: {tf.reduce_max(x)}'

'min_x: 0.0, max_x: 1.0'

f'min_y: {tf.reduce_min(y)}, max_y: {tf.reduce_max(y)}'

'min_y: 0, max_y: 9'

# batch of 128
train_db = tf.data.Dataset.from_tensor_slices((x, y)).batch(128)
train_iter = iter(train_db)
sample = next(train_iter)
f'batch: {sample[0].shape,sample[1].shape}'

'batch: (TensorShape([128, 28, 28]), TensorShape([128]))'

# [b,784] ==> [b,256] ==> [b,128] ==> [b,10]
# [dim_in,dim_out],[dim_out]
w1 = tf.Variable(tf.random.truncated_normal([784, 256], stddev=0.1))
b1 = tf.Variable(tf.zeros([256]))
w2 = tf.Variable(tf.random.truncated_normal([256, 128], stddev=0.1))
b2 = tf.Variable(tf.zeros([128]))
w3 = tf.Variable(tf.random.truncated_normal([128, 10], stddev=0.1))
b3 = tf.Variable(tf.zeros([10]))

# learning rate
lr = 1e-3

for epoch in range(10):  # iterate db for 10
    # tranin every train_db
    for step, (x, y) in enumerate(train_db):
        # x: [128,28,28]
        # y: [128]

        # [b,28,28] ==> [b,28*28]
        x = tf.reshape(x, [-1, 28*28])

        with tf.GradientTape() as tape:  # only data types of tf.variable are logged
            # x: [b,28*28]
            # h1 = x@w1 + b1
            # [b,784]@[784,256]+[256] ==> [b,256] + [256] ==> [b,256] + [b,256]
            h1 = x @ w1 + tf.broadcast_to(b1, [x.shape[0], 256])
            h1 = tf.nn.relu(h1)
            # [b,256] ==> [b,128]
            # h2 = x@w2 + b2  # b2 can broadcast automatic
            h2 = h1 @ w2 + b2
            h2 = tf.nn.relu(h2)
            # [b,128] ==> [b,10]
            out = h2 @ w3 + b3

            # compute loss
            # out: [b,10]
            # y:[b] ==> [b,10]
            y_onehot = tf.one_hot(y, depth=10)

            # mse = mean(sum(y-out)^2)
            # [b,10]
            loss = tf.square(y_onehot - out)
            # mean:scalar
            loss = tf.reduce_mean(loss)

        # compute gradients
        grads = tape.gradient(loss, [w1, b1, w2, b2, w3, b3])
        # w1 = w1 - lr * w1_grad
        # w1 = w1 - lr * grads[0]  # not in situ update
        # in situ update
        w1.assign_sub(lr * grads[0])
        b1.assign_sub(lr * grads[1])
        w2.assign_sub(lr * grads[2])
        b2.assign_sub(lr * grads[3])
        w3.assign_sub(lr * grads[4])
        b3.assign_sub(lr * grads[5])

        if step % 100 == 0:
            print(f'epoch:{epoch}, step: {step}, loss:{float(loss)}')

epoch:0, step: 0, loss:0.5366693735122681
epoch:0, step: 100, loss:0.23276552557945251
epoch:0, step: 200, loss:0.19647717475891113
epoch:0, step: 300, loss:0.17389704287052155
epoch:0, step: 400, loss:0.1731622964143753
epoch:1, step: 0, loss:0.16157487034797668
epoch:1, step: 100, loss:0.16654588282108307
epoch:1, step: 200, loss:0.15311869978904724
epoch:1, step: 300, loss:0.14135733246803284
epoch:1, step: 400, loss:0.14423415064811707
epoch:2, step: 0, loss:0.13703864812850952
epoch:2, step: 100, loss:0.14255204796791077
epoch:2, step: 200, loss:0.1302051544189453
epoch:2, step: 300, loss:0.12224273383617401
epoch:2, step: 400, loss:0.12742099165916443
epoch:3, step: 0, loss:0.1219201311469078
epoch:3, step: 100, loss:0.12757658958435059
epoch:3, step: 200, loss:0.11587800830602646
epoch:3, step: 300, loss:0.10984969139099121
epoch:3, step: 400, loss:0.11641304194927216
epoch:4, step: 0, loss:0.11171815544366837
epoch:4, step: 100, loss:0.11717887222766876
epoch:4, step: 200, loss:0.10604140907526016
epoch:4, step: 300, loss:0.10111508518457413
epoch:4, step: 400, loss:0.10865814983844757
epoch:5, step: 0, loss:0.10434548556804657
epoch:5, step: 100, loss:0.10952303558588028
epoch:5, step: 200, loss:0.09875871241092682
epoch:5, step: 300, loss:0.09467941522598267
epoch:5, step: 400, loss:0.10282392799854279
epoch:6, step: 0, loss:0.09874211996793747
epoch:6, step: 100, loss:0.10355912148952484
epoch:6, step: 200, loss:0.09315416216850281
epoch:6, step: 300, loss:0.08971598744392395
epoch:6, step: 400, loss:0.0982089415192604
epoch:7, step: 0, loss:0.09428335726261139
epoch:7, step: 100, loss:0.09877124428749084
epoch:7, step: 200, loss:0.08866965025663376
epoch:7, step: 300, loss:0.08573523908853531
epoch:7, step: 400, loss:0.09440126270055771
epoch:8, step: 0, loss:0.09056715667247772
epoch:8, step: 100, loss:0.09483197331428528
epoch:8, step: 200, loss:0.0849832147359848
epoch:8, step: 300, loss:0.08246967941522598
epoch:8, step: 400, loss:0.09117519855499268
epoch:9, step: 0, loss:0.08741479367017746
epoch:9, step: 100, loss:0.09150294959545135
epoch:9, step: 200, loss:0.08185736835002899
epoch:9, step: 300, loss:0.07972464710474014
epoch:9, step: 400, loss:0.08842341601848602