Tensorflow模型各部分自定义方式
文章目录
- 1 自定义损失函数
-
- 1.1 第一种方法-函数
- 1.2 第二种方法-装饰器
- 1.3 第三种方法-类
- 1.4 总结
- 2 其它自定义函数-正则\激活函数\初始化\Constraint
-
- 2.1 函数定义-正则\激活函数\初始化\Constraint
- 2.2 类定义方式-正则
- 2.3 类定义方式-激活函数
- 2.4 类定义方式-初始化方式
- 2.5 类定义方式-Constraint
- 3 自定义层
- 4 自定义Metrics
- 5 关于serialize deserialize get register_keras_serializable的使用
-
- 5.1 serialize
- 5.2 deserialize
- 5.3 get
- 5.4 register_keras_serializable
Tensorflow模型的定义有几种不同的方式，Sequentia序列方式，Functional 函数方式及subclassing子类方式，对于像pytorch和paddle的子类方式,如查要把整个模型完整保存需要做一些自定义部分，本文将把各种记录都做个简单的定义
本文参考了tensorflow 和keras官网以及https://github.com/ageron/handson-ml3 下自定义部分的内容。
配套代码：https://gitee.com/tdddeeel/tf2_custom_ops/tree/main
import numpy as np
import tensorflow as tf
print(tf.__version__)
gpu = tf.config.list_physical_devices('GPU')[-1]
tf.config.experimental.set_memory_growth(gpu, True)
tf.config.set_visible_devices(gpu,'GPU')
2022-07-26 14:09:09.401799: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2.9.1
1 自定义损失函数
1.1 第一种方法-函数
def huber_fn(y_true,y_pred):
error = y_true-y_pred
is_small_error = tf.abs(error)<1
squared_error = 0.5*tf.square(error)
linear_error = tf.abs(error)-0.5
return tf.where(is_small_error,squared_error,linear_error)
接着预测波斯顿房价
(x_train,y_train),(x_test,y_test) = tf.keras.datasets.boston_housing.load_data()
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
x_train_scaled=scaler.fit_transform(x_train)
x_test_scaled= scaler.transform(x_test)
tf.random.set_seed(42)
input_shape = x_train.shape[1:]
model = tf.keras.Sequential([
tf.keras.layers.Dense(30,activation='relu',kernel_initializer='he_normal',input_shape=input_shape),
tf.keras.layers.Dense(1)
]
)
model.compile(loss=huber_fn,optimizer='adam',metrics=["mae"])
history=model.fit(x_train_scaled,y_train,epochs=10,validation_data=(x_test_scaled,y_test))
Epoch 1/10
13/13 [==============================] - 0s 14ms/step - loss: 19.8414 - mae: 20.3408 - val_loss: 20.3195 - val_mae: 20.8195
Epoch 2/10
13/13 [==============================] - 0s 6ms/step - loss: 19.4023 - mae: 19.9019 - val_loss: 19.8747 - val_mae: 20.3747
Epoch 3/10
13/13 [==============================] - 0s 5ms/step - loss: 18.9530 - mae: 19.4519 - val_loss: 19.4094 - val_mae: 19.9094
Epoch 4/10
13/13 [==============================] - 0s 5ms/step - loss: 18.4791 - mae: 18.9776 - val_loss: 18.9208 - val_mae: 19.4203
Epoch 5/10
13/13 [==============================] - 0s 5ms/step - loss: 17.9883 - mae: 18.4878 - val_loss: 18.4100 - val_mae: 18.9053
Epoch 6/10
13/13 [==============================] - 0s 6ms/step - loss: 17.4719 - mae: 17.9718 - val_loss: 17.8766 - val_mae: 18.3734
Epoch 7/10
13/13 [==============================] - 0s 5ms/step - loss: 16.9348 - mae: 17.4325 - val_loss: 17.3165 - val_mae: 17.8153
Epoch 8/10
13/13 [==============================] - 0s 5ms/step - loss: 16.3632 - mae: 16.8602 - val_loss: 16.7346 - val_mae: 17.2311
Epoch 9/10
13/13 [==============================] - 0s 5ms/step - loss: 15.7597 - mae: 16.2561 - val_loss: 16.1315 - val_mae: 16.6279
Epoch 10/10
13/13 [==============================] - 0s 5ms/step - loss: 15.1408 - mae: 15.6358 - val_loss: 15.5066 - val_mae: 15.9913
保存模型
model.save('my_model_with_a_custom_loss')
INFO:tensorflow:Assets written to: my_model_with_a_custom_loss/assets
! ls my_model_with_a_custom_loss
assets keras_metadata.pb saved_model.pb variables
可以看到模型正常保存,接着进行加载并继续进行训练
newmodel=tf.keras.models.load_model('my_model_with_a_custom_loss')
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/tmp/ipykernel_4719/1683424579.py in
----> 1 newmodel=tf.keras.models.load_model('my_model_with_a_custom_loss')
~/anaconda3/envs/tf/lib/python3.7/site-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)
65 except Exception as e: # pylint: disable=broad-except
66 filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67 raise e.with_traceback(filtered_tb) from None
68 finally:
69 del filtered_tb
~/anaconda3/envs/tf/lib/python3.7/site-packages/keras/utils/generic_utils.py in deserialize_keras_object(identifier, module_objects, custom_objects, printable_module_name)
708 if obj is None:
709 raise ValueError(
--> 710 f'Unknown {printable_module_name}: {object_name}. Please ensure '
711 'this object is passed to the `custom_objects` argument. See '
712 'https://www.tensorflow.org/guide/keras/save_and_serialize'
ValueError: Unknown loss function: huber_fn. Please ensure this object is passed to the `custom_objects` argument. See https://www.tensorflow.org/guide/keras/save_and_serialize#registering_the_custom_object for details.
报错信息中可以看到Unknow loss,所以要重新加载模型会报错,按照提示进行操作
newmodel=tf.keras.models.load_model('my_model_with_a_custom_loss',custom_objects={'huber_fn':huber_fn})
history1=newmodel.fit(x_train_scaled,y_train,epochs=5,validation_data=(x_test_scaled,y_test))
Epoch 1/5
13/13 [==============================] - 0s 15ms/step - loss: 14.5342 - mae: 15.0307 - val_loss: 14.9579 - val_mae: 15.4550
Epoch 2/5
13/13 [==============================] - 0s 5ms/step - loss: 13.9666 - mae: 14.4617 - val_loss: 14.4416 - val_mae: 14.9335
Epoch 3/5
13/13 [==============================] - 0s 5ms/step - loss: 13.4062 - mae: 13.9023 - val_loss: 13.9684 - val_mae: 14.4664
Epoch 4/5
13/13 [==============================] - 0s 6ms/step - loss: 12.8451 - mae: 13.3421 - val_loss: 13.5037 - val_mae: 13.9950
Epoch 5/5
13/13 [==============================] - 0s 5ms/step - loss: 12.2937 - mae: 12.7887 - val_loss: 13.0708 - val_mae: 13.5672
以上就完成的自定义损失函数模型的训练、保存和加载。可以看来损失值是接着原先的值变化的。接着往下看：
1.2 第二种方法-装饰器
def create_huber(threshold=1.0):
def huber_fn(y_true,y_pred):
error = y_true-y_pred
is_small_error = tf.abs(error)<threshold
squared_error = 0.5*tf.square(error)
linear_error = threshold*tf.abs(error)-0.5*threshold**2
return tf.where(is_small_error,squared_error,linear_error)
return huber_fn
tf.random.set_seed(42)
input_shape = x_train.shape[1:]
model = tf.keras.Sequential([
tf.keras.layers.Dense(30,activation='relu',kernel_initializer='he_normal',input_shape=input_shape),
tf.keras.layers.Dense(1)
]
)
model.compile(loss=create_huber(2.0),optimizer='adam',metrics=["mae"]) #重新compile后，所有状态都会进行重新更新
history2=model.fit(x_train_scaled,y_train,epochs=10,validation_data=(x_test_scaled,y_test))
Epoch 1/10
13/13 [==============================] - 1s 15ms/step - loss: 38.6856 - mae: 20.3408 - val_loss: 39.6391 - val_mae: 20.8196
Epoch 2/10
13/13 [==============================] - 0s 6ms/step - loss: 37.8080 - mae: 19.9019 - val_loss: 38.7495 - val_mae: 20.3747
Epoch 3/10
13/13 [==============================] - 0s 6ms/step - loss: 36.9100 - mae: 19.4519 - val_loss: 37.8214 - val_mae: 19.9096
Epoch 4/10
13/13 [==============================] - 0s 5ms/step - loss: 35.9649 - mae: 18.9778 - val_loss: 36.8493 - val_mae: 19.4203
Epoch 5/10
13/13 [==============================] - 0s 6ms/step - loss: 34.9810 - mae: 18.4875 - val_loss: 35.8320 - val_mae: 18.9041
Epoch 6/10
13/13 [==============================] - 0s 6ms/step - loss: 33.9451 - mae: 17.9707 - val_loss: 34.7692 - val_mae: 18.3724
Epoch 7/10
13/13 [==============================] - 0s 6ms/step - loss: 32.8763 - mae: 17.4321 - val_loss: 33.6509 - val_mae: 17.8151
Epoch 8/10
13/13 [==============================] - 0s 6ms/step - loss: 31.7416 - mae: 16.8602 - val_loss: 32.4894 - val_mae: 17.2308
Epoch 9/10
13/13 [==============================] - 0s 5ms/step - loss: 30.5389 - mae: 16.2564 - val_loss: 31.2985 - val_mae: 16.6272
Epoch 10/10
13/13 [==============================] - 0s 7ms/step - loss: 29.3051 - mae: 15.6356 - val_loss: 30.0576 - val_mae: 15.9893
model.save('my_model_with_a_custom_loss_threshold_2')
INFO:tensorflow:Assets written to: my_model_with_a_custom_loss_threshold_2/assets
! ls my_model_with_a_custom_loss_threshold_2
assets keras_metadata.pb saved_model.pb variables
可以看到，同样是保存成功的
newmodel = tf.keras.models.load_model('my_model_with_a_custom_loss_threshold_2/')
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/tmp/ipykernel_4719/3766741098.py in
----> 1 newmodel = tf.keras.models.load_model('my_model_with_a_custom_loss_threshold_2/')
~/anaconda3/envs/tf/lib/python3.7/site-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)
65 except Exception as e: # pylint: disable=broad-except
66 filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67 raise e.with_traceback(filtered_tb) from None
68 finally:
69 del filtered_tb
~/anaconda3/envs/tf/lib/python3.7/site-packages/keras/utils/generic_utils.py in deserialize_keras_object(identifier, module_objects, custom_objects, printable_module_name)
708 if obj is None:
709 raise ValueError(
--> 710 f'Unknown {printable_module_name}: {object_name}. Please ensure '
711 'this object is passed to the `custom_objects` argument. See '
712 'https://www.tensorflow.org/guide/keras/save_and_serialize'
ValueError: Unknown loss function: huber_fn. Please ensure this object is passed to the `custom_objects` argument. See https://www.tensorflow.org/guide/keras/save_and_serialize#registering_the_custom_object for details.
如果不指定custom objects还是会报错
newmodel = tf.keras.models.load_model('my_model_with_a_custom_loss_threshold_2/',custom_objects={'huber_fn':create_huber(2.0)})
history1=newmodel.fit(x_train_scaled,y_train,epochs=5,validation_data=(x_test_scaled,y_test))
Epoch 1/5
13/13 [==============================] - 1s 15ms/step - loss: 28.0896 - mae: 15.0308 - val_loss: 28.9675 - val_mae: 15.4553
Epoch 2/5
13/13 [==============================] - 0s 6ms/step - loss: 26.9547 - mae: 14.4610 - val_loss: 27.9318 - val_mae: 14.9332
Epoch 3/5
13/13 [==============================] - 0s 5ms/step - loss: 25.8384 - mae: 13.9020 - val_loss: 26.9647 - val_mae: 14.4667
Epoch 4/5
13/13 [==============================] - 0s 5ms/step - loss: 24.7175 - mae: 13.3422 - val_loss: 26.0434 - val_mae: 13.9959
Epoch 5/5
13/13 [==============================] - 0s 5ms/step - loss: 23.6254 - mae: 12.7892 - val_loss: 25.1706 - val_mae: 13.5669
1.3 第三种方法-类
class HuberLoss(tf.keras.losses.Loss):
def __init__(self,threshold=1.0,**kwargs):
self.threshold=threshold
super().__init__(**kwargs)
def call(self,y_true,y_pred):
error = y_true-y_pred
is_small_error = tf.abs(error)<self.threshold
squared_error = 0.5*tf.square(error)
linear_error = self.threshold*tf.abs(error)-0.5*self.threshold**2
return tf.where(is_small_error,squared_error,linear_error)
def get_config(self):
base_config = super().get_config()
all_config = {**base_config,"threshold":self.threshold}
return all_config
#再次定义新的模型
tf.random.set_seed(42)
input_shape = x_train.shape[1:]
model = tf.keras.Sequential([
tf.keras.layers.Dense(30,activation='relu',kernel_initializer='he_normal',input_shape=input_shape),
tf.keras.layers.Dense(1)
]
)
model.compile(loss=HuberLoss(2.),optimizer='adam',metrics=["mae"])
history=model.fit(x_train_scaled,y_train,epochs=10,validation_data=(x_test_scaled,y_test))
Epoch 1/10
13/13 [==============================] - 1s 16ms/step - loss: 38.6856 - mae: 20.3408 - val_loss: 39.6391 - val_mae: 20.8196
Epoch 2/10
13/13 [==============================] - 0s 7ms/step - loss: 37.8080 - mae: 19.9019 - val_loss: 38.7495 - val_mae: 20.3747
Epoch 3/10
13/13 [==============================] - 0s 7ms/step - loss: 36.9100 - mae: 19.4519 - val_loss: 37.8214 - val_mae: 19.9096
Epoch 4/10
13/13 [==============================] - 0s 6ms/step - loss: 35.9649 - mae: 18.9778 - val_loss: 36.8493 - val_mae: 19.4203
Epoch 5/10
13/13 [==============================] - 0s 7ms/step - loss: 34.9810 - mae: 18.4875 - val_loss: 35.8320 - val_mae: 18.9041
Epoch 6/10
13/13 [==============================] - 0s 6ms/step - loss: 33.9451 - mae: 17.9707 - val_loss: 34.7692 - val_mae: 18.3724
Epoch 7/10
13/13 [==============================] - 0s 6ms/step - loss: 32.8763 - mae: 17.4321 - val_loss: 33.6509 - val_mae: 17.8151
Epoch 8/10
13/13 [==============================] - 0s 6ms/step - loss: 31.7416 - mae: 16.8602 - val_loss: 32.4894 - val_mae: 17.2308
Epoch 9/10
13/13 [==============================] - 0s 6ms/step - loss: 30.5389 - mae: 16.2564 - val_loss: 31.2985 - val_mae: 16.6272
Epoch 10/10
13/13 [==============================] - 0s 6ms/step - loss: 29.3051 - mae: 15.6356 - val_loss: 30.0576 - val_mae: 15.9893
可以看到，和方法二的结果是一样的，tensorflow2.9中加入tf.random.set_seed()等其它优化，使得结果尽可能可以复现
model.save('my_model_with_a_custom_loss_class')
INFO:tensorflow:Assets written to: my_model_with_a_custom_loss_class/assets
newmodel = tf.keras.models.load_model('my_model_with_a_custom_loss_class/')
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/tmp/ipykernel_4719/829131420.py in
----> 1 newmodel = tf.keras.models.load_model('my_model_with_a_custom_loss_class/')
~/anaconda3/envs/tf/lib/python3.7/site-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)
65 except Exception as e: # pylint: disable=broad-except
66 filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67 raise e.with_traceback(filtered_tb) from None
68 finally:
69 del filtered_tb
~/anaconda3/envs/tf/lib/python3.7/site-packages/keras/utils/generic_utils.py in class_and_config_for_serialized_keras_object(config, module_objects, custom_objects, printable_module_name)
561 if cls is None:
562 raise ValueError(
--> 563 f'Unknown {printable_module_name}: {class_name}. Please ensure this '
564 'object is passed to the `custom_objects` argument. See '
565 'https://www.tensorflow.org/guide/keras/save_and_serialize'
ValueError: Unknown loss function: HuberLoss. Please ensure this object is passed to the `custom_objects` argument. See https://www.tensorflow.org/guide/keras/save_and_serialize#registering_the_custom_object for details.
可以看到，模型还是无法正常加载
newmodel = tf.keras.models.load_model('my_model_with_a_custom_loss_class/',custom_objects={"HuberLoss":HuberLoss(2.)})
history1=newmodel.fit(x_train_scaled,y_train,epochs=5,validation_data=(x_test_scaled,y_test))
Epoch 1/5
13/13 [==============================] - 1s 15ms/step - loss: 28.0885 - mae: 15.0302 - val_loss: 28.9629 - val_mae: 15.4528
Epoch 2/5
13/13 [==============================] - 0s 6ms/step - loss: 26.9527 - mae: 14.4601 - val_loss: 27.9266 - val_mae: 14.9309
Epoch 3/5
13/13 [==============================] - 0s 6ms/step - loss: 25.8358 - mae: 13.9006 - val_loss: 26.9588 - val_mae: 14.4639
Epoch 4/5
13/13 [==============================] - 0s 6ms/step - loss: 24.7142 - mae: 13.3406 - val_loss: 26.0373 - val_mae: 13.9925
Epoch 5/5
13/13 [==============================] - 0s 7ms/step - loss: 23.6216 - mae: 12.7874 - val_loss: 25.1644 - val_mae: 13.5636
1.4 总结
以上三种方法，效果是相同的,但都有一个不足,那就是在恢复模型时必须要有自定义部分的原始代码,这给模型部署带来不便,要是想要部署一个模型还要带上这个自定义函数,这是很不合适的后边会写处理方式.
2 其它自定义函数-正则\激活函数\初始化\Constraint
2.1 函数定义-正则\激活函数\初始化\Constraint
#激活函数
def my_softplus(z):
return tf.math.log(1.0+tf.exp(z))
#初始化函数
def my_glorot_initializer(shape,dtype=tf.float32):
stddev = tf.sqrt(2./(shape[0]+shape[1]))
return tf.random.normal(shape,stddev=stddev,dtype=dtype)
#正则化
def my_l1_regularizer(weights):
return tf.reduce_sum(tf.abs(0.01*weights))
#constraint
def my_positive_weights(weights):
return tf.where(weights<0.,tf.zeros_like(weights),weights)
layer = tf.keras.layers.Dense(1,activation=my_softplus,
kernel_initializer=my_glorot_initializer,
kernel_regularizer=my_l1_regularizer,
kernel_constraint=my_positive_weights
)
layer.get_config()
{'name': 'dense_6',
'trainable': True,
'dtype': 'float32',
'units': 1,
'activation': 'my_softplus',
'use_bias': True,
'kernel_initializer': 'my_glorot_initializer',
'bias_initializer': {'class_name': 'Zeros', 'config': {}},
'kernel_regularizer': 'my_l1_regularizer',
'bias_regularizer': None,
'activity_regularizer': None,
'kernel_constraint': 'my_positive_weights',
'bias_constraint': None}
接着定义模型,训练,保存,加载,再训练
tf.random.set_seed(42)
model = tf.keras.Sequential([
tf.keras.layers.Dense(30,activation='relu',kernel_initializer='he_normal'),
tf.keras.layers.Dense(1,activation=my_softplus,
kernel_initializer=my_glorot_initializer,
kernel_regularizer=my_l1_regularizer,
kernel_constraint=my_positive_weights
)
])
model.compile(loss='mse',optimizer='adam',metrics=["mae"])
history=model.fit(x_train_scaled,y_train,epochs=10,validation_data=(x_test_scaled,y_test))
model.save('my_model_with_a_custom_parts')
Epoch 1/10
13/13 [==============================] - 1s 14ms/step - loss: 515.5219 - mae: 20.7559 - val_loss: 531.4690 - val_mae: 21.1663
Epoch 2/10
13/13 [==============================] - 0s 6ms/step - loss: 501.5470 - mae: 20.4198 - val_loss: 519.0331 - val_mae: 20.8716
Epoch 3/10
13/13 [==============================] - 0s 5ms/step - loss: 488.6118 - mae: 20.1095 - val_loss: 504.7845 - val_mae: 20.5305
Epoch 4/10
13/13 [==============================] - 0s 5ms/step - loss: 473.7936 - mae: 19.7504 - val_loss: 488.5923 - val_mae: 20.1391
Epoch 5/10
13/13 [==============================] - 0s 5ms/step - loss: 456.8321 - mae: 19.3409 - val_loss: 470.3868 - val_mae: 19.6946
Epoch 6/10
13/13 [==============================] - 0s 6ms/step - loss: 437.9981 - mae: 18.8721 - val_loss: 450.0984 - val_mae: 19.1896
Epoch 7/10
13/13 [==============================] - 0s 5ms/step - loss: 417.4472 - mae: 18.3609 - val_loss: 428.3565 - val_mae: 18.6335
Epoch 8/10
13/13 [==============================] - 0s 5ms/step - loss: 395.6647 - mae: 17.7891 - val_loss: 404.8337 - val_mae: 18.0099
Epoch 9/10
13/13 [==============================] - 0s 5ms/step - loss: 372.3109 - mae: 17.1659 - val_loss: 380.1979 - val_mae: 17.3433
Epoch 10/10
13/13 [==============================] - 0s 5ms/step - loss: 347.9553 - mae: 16.4938 - val_loss: 354.7127 - val_mae: 16.6338
INFO:tensorflow:Assets written to: my_model_with_a_custom_parts/assets
model.get_config()
{'name': 'sequential_3',
'layers': [{'class_name': 'InputLayer',
'config': {'batch_input_shape': (None, 13),
'dtype': 'float32',
'sparse': False,
'ragged': False,
'name': 'dense_7_input'}},
{'class_name': 'Dense',
'config': {'name': 'dense_7',
'trainable': True,
'dtype': 'float32',
'units': 30,
'activation': 'relu',
'use_bias': True,
'kernel_initializer': {'class_name': 'HeNormal', 'config': {'seed': None}},
'bias_initializer': {'class_name': 'Zeros', 'config': {}},
'kernel_regularizer': None,
'bias_regularizer': None,
'activity_regularizer': None,
'kernel_constraint': None,
'bias_constraint': None}},
{'class_name': 'Dense',
'config': {'name': 'dense_8',
'trainable': True,
'dtype': 'float32',
'units': 1,
'activation': 'my_softplus',
'use_bias': True,
'kernel_initializer': 'my_glorot_initializer',
'bias_initializer': {'class_name': 'Zeros', 'config': {}},
'kernel_regularizer': 'my_l1_regularizer',
'bias_regularizer': None,
'activity_regularizer': None,
'kernel_constraint': 'my_positive_weights',
'bias_constraint': None}}]}
model.layers[1].get_config()
{'name': 'dense_8',
'trainable': True,
'dtype': 'float32',
'units': 1,
'activation': 'my_softplus',
'use_bias': True,
'kernel_initializer': 'my_glorot_initializer',
'bias_initializer': {'class_name': 'Zeros', 'config': {}},
'kernel_regularizer': 'my_l1_regularizer',
'bias_regularizer': None,
'activity_regularizer': None,
'kernel_constraint': 'my_positive_weights',
'bias_constraint': None}
newmodel = tf.keras.models.load_model('my_model_with_a_custom_parts/')
history1=newmodel.fit(x_train_scaled,y_train,epochs=5,validation_data=(x_test_scaled,y_test))
Epoch 1/5
13/13 [==============================] - 1s 15ms/step - loss: 325.0072 - mae: 15.8323 - val_loss: 333.4890 - val_mae: 16.0263
Epoch 2/5
13/13 [==============================] - 0s 6ms/step - loss: 304.5963 - mae: 15.2173 - val_loss: 312.2325 - val_mae: 15.3937
Epoch 3/5
13/13 [==============================] - 0s 6ms/step - loss: 284.0917 - mae: 14.5946 - val_loss: 291.3275 - val_mae: 14.7601
Epoch 4/5
13/13 [==============================] - 0s 5ms/step - loss: 264.2493 - mae: 13.9393 - val_loss: 271.0325 - val_mae: 14.1239
Epoch 5/5
13/13 [==============================] - 0s 6ms/step - loss: 244.9014 - mae: 13.2921 - val_loss: 251.4772 - val_mae: 13.5366
newmodel.layers[1].get_config()
{'name': 'dense_8',
'trainable': True,
'dtype': 'float32',
'units': 1,
'activation': 'my_softplus',
'use_bias': True,
'kernel_initializer': 'my_glorot_initializer',
'bias_initializer': {'class_name': 'Zeros',
'config': {},
'shared_object_id': 4},
'kernel_regularizer': 'my_l1_regularizer',
'bias_regularizer': None,
'activity_regularizer': None,
'kernel_constraint': 'my_positive_weights',
'bias_constraint': None}
所以要判断模型是否一致,我们可以看一下他们的config
可以看到,对于这些自定义的函数无需人为指定也可以加载
newmodel1 = tf.keras.models.load_model('my_model_with_a_custom_parts/',custom_objects={"my_l1_regularizer":my_l1_regularizer,
"my_positive_weights":my_positive_weights,
"my_glorot_initializer":my_glorot_initializer,
"my_softplus":my_softplus
})
history2=newmodel1.fit(x_train_scaled,y_train,epochs=5,validation_data=(x_test_scaled,y_test))
Epoch 1/5
13/13 [==============================] - 1s 17ms/step - loss: 325.0072 - mae: 15.8323 - val_loss: 333.4890 - val_mae: 16.0263
Epoch 2/5
13/13 [==============================] - 0s 6ms/step - loss: 304.5963 - mae: 15.2173 - val_loss: 312.2325 - val_mae: 15.3937
Epoch 3/5
13/13 [==============================] - 0s 7ms/step - loss: 284.0917 - mae: 14.5946 - val_loss: 291.3275 - val_mae: 14.7601
Epoch 4/5
13/13 [==============================] - 0s 6ms/step - loss: 264.2493 - mae: 13.9393 - val_loss: 271.0325 - val_mae: 14.1239
Epoch 5/5
13/13 [==============================] - 0s 7ms/step - loss: 244.9014 - mae: 13.2921 - val_loss: 251.4772 - val_mae: 13.5366
这次可以看出,人为指定后,结果是一样的
2.2 类定义方式-正则
正则化也可以用类方式定义
class MyL1Regularizer(tf.keras.regularizers.Regularizer):
def __init__(self,factor):
self.factor=factor
def __call__(self,weights):
return tf.reduce_mean(tf.math.abs(self.factor*weights))
def get_config(self):
base_config = super().get_config()
all_config={**base_config,'factor':self.factor}
tf.random.set_seed(42)
model = tf.keras.Sequential([
tf.keras.layers.Dense(30,activation='relu',kernel_initializer='he_normal'),
tf.keras.layers.Dense(1,activation=my_softplus,
kernel_initializer=my_glorot_initializer,
kernel_regularizer=MyL1Regularizer(0.01),
kernel_constraint=my_positive_weights
)
])
model.compile(loss='mse',optimizer='adam',metrics=["mae"])
history=model.fit(x_train_scaled,y_train,epochs=10,validation_data=(x_test_scaled,y_test))
model.save('my_model_with_a_custom_parts1')
Epoch 1/10
13/13 [==============================] - 1s 19ms/step - loss: 515.4951 - mae: 20.7559 - val_loss: 531.4422 - val_mae: 21.1663
Epoch 2/10
13/13 [==============================] - 0s 5ms/step - loss: 501.5185 - mae: 20.4198 - val_loss: 519.0024 - val_mae: 20.8716
Epoch 3/10
13/13 [==============================] - 0s 5ms/step - loss: 488.5792 - mae: 20.1095 - val_loss: 504.7497 - val_mae: 20.5305
Epoch 4/10
13/13 [==============================] - 0s 5ms/step - loss: 473.7570 - mae: 19.7503 - val_loss: 488.5533 - val_mae: 20.1391
Epoch 5/10
13/13 [==============================] - 0s 5ms/step - loss: 456.7914 - mae: 19.3408 - val_loss: 470.3438 - val_mae: 19.6946
Epoch 6/10
13/13 [==============================] - 0s 5ms/step - loss: 437.9532 - mae: 18.8721 - val_loss: 450.0512 - val_mae: 19.1895
Epoch 7/10
13/13 [==============================] - 0s 5ms/step - loss: 417.3983 - mae: 18.3609 - val_loss: 428.3052 - val_mae: 18.6335
Epoch 8/10
13/13 [==============================] - 0s 5ms/step - loss: 395.6117 - mae: 17.7890 - val_loss: 404.7784 - val_mae: 18.0099
Epoch 9/10
13/13 [==============================] - 0s 5ms/step - loss: 372.2539 - mae: 17.1659 - val_loss: 380.1387 - val_mae: 17.3433
Epoch 10/10
13/13 [==============================] - 0s 6ms/step - loss: 347.8943 - mae: 16.4938 - val_loss: 354.6496 - val_mae: 16.6338
INFO:tensorflow:Assets written to: my_model_with_a_custom_parts1/assets
newmodel = tf.keras.models.load_model('my_model_with_a_custom_parts1/')
history1=newmodel.fit(x_train_scaled,y_train,epochs=5,validation_data=(x_test_scaled,y_test))
Epoch 1/5
13/13 [==============================] - 1s 16ms/step - loss: 325.8748 - mae: 15.8555 - val_loss: 333.4524 - val_mae: 16.0323
Epoch 2/5
13/13 [==============================] - 0s 6ms/step - loss: 304.9491 - mae: 15.2394 - val_loss: 312.9211 - val_mae: 15.4228
Epoch 3/5
13/13 [==============================] - 0s 5ms/step - loss: 285.0223 - mae: 14.6250 - val_loss: 292.1203 - val_mae: 14.7955
Epoch 4/5
13/13 [==============================] - 0s 6ms/step - loss: 264.9132 - mae: 13.9857 - val_loss: 271.5786 - val_mae: 14.1489
Epoch 5/5
13/13 [==============================] - 0s 5ms/step - loss: 245.6193 - mae: 13.3311 - val_loss: 251.5693 - val_mae: 13.5434
newmodel1 = tf.keras.models.load_model('my_model_with_a_custom_parts1/',custom_objects={"MyL1Regularizer":MyL1Regularizer,
"my_positive_weights":my_positive_weights,
"my_glorot_initializer":my_glorot_initializer,
"my_softplus":my_softplus
})
history2=newmodel1.fit(x_train_scaled,y_train,epochs=5,validation_data=(x_test_scaled,y_test))
2022-07-13 10:10:51.405105: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-07-13 10:10:52.073048: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 14637 MB memory: -> device: 3, name: Tesla V100-PCIE-16GB-LS, pci bus id: 0000:dd:00.0, compute capability: 7.0
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
/tmp/ipykernel_28741/247645023.py in
2 "my_positive_weights":my_positive_weights,
3 "my_glorot_initializer":my_glorot_initializer,
----> 4 "my_softplus":my_softplus
5 })
6 history2=newmodel1.fit(x_train_scaled,y_train,epochs=5,validation_data=(x_test_scaled,y_test))
~/anaconda3/envs/tf/lib/python3.7/site-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)
65 except Exception as e: # pylint: disable=broad-except
66 filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67 raise e.with_traceback(filtered_tb) from None
68 finally:
69 del filtered_tb
~/anaconda3/envs/tf/lib/python3.7/site-packages/keras/saving/saved_model/load.py in _revive_layer_or_model_from_config(self, metadata, node_id)
536 'your class with `@keras.utils.register_keras_serializable` and '
537 'include that file in your program, or pass your class in a '
--> 538 '`keras.utils.CustomObjectScope` that wraps this load call.') from e
539 else:
540 raise
RuntimeError: Unable to restore object of class 'Dense' likely due to name conflict with built-in Keras class ''. To override the built-in Keras definition of the object, decorate your class with `@keras.utils.register_keras_serializable` and include that file in your program, or pass your class in a `keras.utils.CustomObjectScope` that wraps this load call.
这种情况,不知道为什么会报错,如果不指定反而不会报错,可能是因为第二个Dense已经是一个自定义层了需要做更多处理.
2.3 类定义方式-激活函数
对于激活函数
可以直接使用tf.keras.layers.Activation或tf.keras.layers.Layer来进行构建,因为有些激活函数带有要学习的参数,所以定义激活函数要像定义层一样,请参看下一节
2.4 类定义方式-初始化方式
接着使用类来定义初始化方法
class MyGlortInitializer(tf.keras.initializers.Initializer):
def __call__(self,shape,dtype):
self.stddev = tf.sqrt(2./(shape[0]+shape[1]))
return tf.random.normal(shape,stddev=self.stddev,dtype=dtype) #正常还要写一个get_config
初始化没必要做到能保存，因为模型训练后或不训练，再次加载时用不到，只有第一次才能用到，所以get_config之类的，在initializer中是没有的
tf.random.set_seed(42)
model = tf.keras.Sequential([
tf.keras.layers.Dense(30,activation='relu',kernel_initializer='he_normal'),
tf.keras.layers.Dense(1,activation=my_softplus,
kernel_initializer=MyGlortInitializer,
kernel_regularizer=my_l1_regularizer,
kernel_constraint=my_positive_weights
)
])
model.compile(loss='mse',optimizer='adam',metrics=["mae"])
history=model.fit(x_train_scaled,y_train,epochs=10,validation_data=(x_test_scaled,y_test))
model.save('my_model_with_a_custom_parts2')
Epoch 1/10
13/13 [==============================] - 1s 15ms/step - loss: 515.5219 - mae: 20.7559 - val_loss: 531.4690 - val_mae: 21.1663
Epoch 2/10
13/13 [==============================] - 0s 6ms/step - loss: 501.5470 - mae: 20.4198 - val_loss: 519.0331 - val_mae: 20.8716
Epoch 3/10
13/13 [==============================] - 0s 5ms/step - loss: 488.6118 - mae: 20.1095 - val_loss: 504.7845 - val_mae: 20.5305
Epoch 4/10
13/13 [==============================] - 0s 5ms/step - loss: 473.7936 - mae: 19.7504 - val_loss: 488.5923 - val_mae: 20.1391
Epoch 5/10
13/13 [==============================] - 0s 5ms/step - loss: 456.8321 - mae: 19.3409 - val_loss: 470.3868 - val_mae: 19.6946
Epoch 6/10
13/13 [==============================] - 0s 5ms/step - loss: 437.9981 - mae: 18.8721 - val_loss: 450.0984 - val_mae: 19.1896
Epoch 7/10
13/13 [==============================] - 0s 6ms/step - loss: 417.4472 - mae: 18.3609 - val_loss: 428.3565 - val_mae: 18.6335
Epoch 8/10
13/13 [==============================] - 0s 5ms/step - loss: 395.6647 - mae: 17.7891 - val_loss: 404.8337 - val_mae: 18.0099
Epoch 9/10
13/13 [==============================] - 0s 5ms/step - loss: 372.3109 - mae: 17.1659 - val_loss: 380.1979 - val_mae: 17.3433
Epoch 10/10
13/13 [==============================] - 0s 5ms/step - loss: 347.9553 - mae: 16.4938 - val_loss: 354.7128 - val_mae: 16.6338
INFO:tensorflow:Assets written to: my_model_with_a_custom_parts2/assets
newmodel = tf.keras.models.load_model('my_model_with_a_custom_parts2')
history1=newmodel.fit(x_train_scaled,y_train,epochs=5,validation_data=(x_test_scaled,y_test))
Epoch 1/5
13/13 [==============================] - 1s 14ms/step - loss: 325.0072 - mae: 15.8323 - val_loss: 333.4890 - val_mae: 16.0263
Epoch 2/5
13/13 [==============================] - 0s 6ms/step - loss: 304.5963 - mae: 15.2173 - val_loss: 312.2325 - val_mae: 15.3937
Epoch 3/5
13/13 [==============================] - 0s 5ms/step - loss: 284.0917 - mae: 14.5946 - val_loss: 291.3275 - val_mae: 14.7601
Epoch 4/5
13/13 [==============================] - 0s 5ms/step - loss: 264.2493 - mae: 13.9393 - val_loss: 271.0325 - val_mae: 14.1239
Epoch 5/5
13/13 [==============================] - 0s 6ms/step - loss: 244.9014 - mae: 13.2921 - val_loss: 251.4772 - val_mae: 13.5366
newmodel1 = tf.keras.models.load_model('my_model_with_a_custom_parts2/',custom_objects={"my_l1_regularizer":my_l1_regularizer,
"my_positive_weights":my_positive_weights,
"MyGlorotInitizlizer":MyGlortInitializer,
"my_softplus":my_softplus
})
history2=newmodel1.fit(x_train_scaled,y_train,epochs=5,validation_data=(x_test_scaled,y_test))
Epoch 1/5
13/13 [==============================] - 1s 15ms/step - loss: 325.0072 - mae: 15.8323 - val_loss: 333.4890 - val_mae: 16.0263
Epoch 2/5
13/13 [==============================] - 0s 5ms/step - loss: 304.5963 - mae: 15.2173 - val_loss: 312.2325 - val_mae: 15.3937
Epoch 3/5
13/13 [==============================] - 0s 5ms/step - loss: 284.0917 - mae: 14.5946 - val_loss: 291.3275 - val_mae: 14.7601
Epoch 4/5
13/13 [==============================] - 0s 6ms/step - loss: 264.2493 - mae: 13.9393 - val_loss: 271.0325 - val_mae: 14.1239
Epoch 5/5
13/13 [==============================] - 0s 5ms/step - loss: 244.9014 - mae: 13.2921 - val_loss: 251.4772 - val_mae: 13.5366
2.5 类定义方式-Constraint
接着对于constraint这里就不写了,官方文档中写到,constraint是一种stateless的,对于这种,无__init__,无需get_config,只要重写__call__就完事
class NonNegative(tf.keras.constraints.Constraint):
def __call__(self,w):
return w*tf.cast(tf.math.greater_equal(w,0),w.dtype)
def get_config(self):
base_config=super().get_config()
return base_config
3 自定义层
exponential_layer=tf.keras.layers.Lambda(lambda x :tf.math.exp(x))
exponential_layer([-1.,0.,1.])
如果要预测的结果是正,且有很大并异的比例[0.001,0.1,10,1000] ,可以在最后一层加上指数函数
tf.random.set_seed(42)
input_shape = x_train.shape[1:]
model = tf.keras.Sequential([
tf.keras.layers.Dense(30,activation='relu',kernel_initializer='he_normal',input_shape=input_shape),
tf.keras.layers.Dense(1),
exponential_layer
]
)
model.compile(loss='mse',optimizer='adam',metrics=["mae"])
history2=model.fit(x_train_scaled,y_train,epochs=10,validation_data=(x_test_scaled,y_test))
Epoch 1/10
13/13 [==============================] - 0s 13ms/step - loss: 5567.9629 - mae: 32.7794 - val_loss: 6352.1050 - val_mae: 32.6988
Epoch 2/10
13/13 [==============================] - 0s 6ms/step - loss: 1320.4006 - mae: 23.1720 - val_loss: 1681.2922 - val_mae: 24.9112
Epoch 3/10
13/13 [==============================] - 0s 5ms/step - loss: 755.1310 - mae: 20.5381 - val_loss: 980.6192 - val_mae: 22.5717
Epoch 4/10
13/13 [==============================] - 0s 5ms/step - loss: 538.5795 - mae: 19.2997 - val_loss: 816.2770 - val_mae: 21.7834
Epoch 5/10
13/13 [==============================] - 0s 5ms/step - loss: 490.4356 - mae: 18.8144 - val_loss: 729.4755 - val_mae: 21.2690
Epoch 6/10
13/13 [==============================] - 0s 5ms/step - loss: 463.7896 - mae: 18.5437 - val_loss: 675.5165 - val_mae: 20.8762
Epoch 7/10
13/13 [==============================] - 0s 5ms/step - loss: 448.9651 - mae: 18.3412 - val_loss: 632.6336 - val_mae: 20.5424
Epoch 8/10
13/13 [==============================] - 0s 5ms/step - loss: 431.8742 - mae: 18.1127 - val_loss: 606.0991 - val_mae: 20.2699
Epoch 9/10
13/13 [==============================] - 0s 5ms/step - loss: 419.8213 - mae: 17.9123 - val_loss: 587.3072 - val_mae: 20.0439
Epoch 10/10
13/13 [==============================] - 0s 5ms/step - loss: 410.5321 - mae: 17.7241 - val_loss: 565.5510 - val_mae: 19.7840
model.save('my_model_with_a_custom_layer1')
INFO:tensorflow:Assets written to: my_model_with_a_custom_layer1/assets
newmodel = tf.keras.models.load_model('my_model_with_a_custom_layer1')
history1=newmodel.fit(x_train_scaled,y_train,epochs=5,validation_data=(x_test_scaled,y_test))
Epoch 1/5
13/13 [==============================] - 0s 14ms/step - loss: 385.0820 - mae: 17.2846 - val_loss: 453.2182 - val_mae: 18.4360
Epoch 2/5
13/13 [==============================] - 0s 5ms/step - loss: 359.4727 - mae: 16.8214 - val_loss: 418.5759 - val_mae: 17.7304
Epoch 3/5
13/13 [==============================] - 0s 5ms/step - loss: 344.2751 - mae: 16.4029 - val_loss: 402.5103 - val_mae: 17.4366
Epoch 4/5
13/13 [==============================] - 0s 6ms/step - loss: 328.9195 - mae: 15.9526 - val_loss: 390.0414 - val_mae: 17.2147
Epoch 5/5
13/13 [==============================] - 0s 5ms/step - loss: 312.5679 - mae: 15.4785 - val_loss: 363.0509 - val_mae: 16.6054
对于更复杂的自定义层
class MyDense(tf.keras.layers.Layer):
def __init__(self,units,activation=None,**kwargs):
super().__init__(**kwargs)
self.units= units
self.activation=tf.keras.activations.get(activation)
def build(self,input_shape):
self.kernel = self.add_weight(
name='kernel',
shape=[input_shape[-1],self.units],
initializer='he_normal',
trainable=True
)
self.bias = self.add_weight(
name='bias',
shape=[self.units],
initializer='zeros',
trainable=True
)
# super().build(input_shape)
def call(self,x):
return self.activation(x@self.kernel+self.bias)
def get_config(self):
base_config=super().get_config()
all_config={**base_config,'units':self.units,'activation':tf.keras.activations.serialize(self.activation)}
return all_config
看一下这个激活函数的操作
ac = tf.keras.activations.get('relu')
print(tf.keras.activations.serialize(ac))
relu
tf.random.set_seed(42)
input_shape = x_train.shape[1:]
model = tf.keras.Sequential([
MyDense(30,activation='relu'),
MyDense(1),
]
)
model.compile(loss='mse',optimizer='adam',metrics=["mae"])
history2=model.fit(x_train_scaled,y_train,epochs=10,validation_data=(x_test_scaled,y_test))
Epoch 1/10
13/13 [==============================] - 0s 13ms/step - loss: 616.1592 - mae: 23.1205 - val_loss: 639.0024 - val_mae: 23.6284
Epoch 2/10
13/13 [==============================] - 0s 5ms/step - loss: 598.2690 - mae: 22.7416 - val_loss: 620.9468 - val_mae: 23.2488
Epoch 3/10
13/13 [==============================] - 0s 5ms/step - loss: 580.7922 - mae: 22.3643 - val_loss: 603.4016 - val_mae: 22.8720
Epoch 4/10
13/13 [==============================] - 0s 5ms/step - loss: 563.6730 - mae: 21.9861 - val_loss: 586.1691 - val_mae: 22.4956
Epoch 5/10
13/13 [==============================] - 0s 5ms/step - loss: 546.5667 - mae: 21.6060 - val_loss: 569.0162 - val_mae: 22.1156
Epoch 6/10
13/13 [==============================] - 0s 5ms/step - loss: 529.6218 - mae: 21.2165 - val_loss: 551.5815 - val_mae: 21.7227
Epoch 7/10
13/13 [==============================] - 0s 5ms/step - loss: 512.6191 - mae: 20.8243 - val_loss: 534.1431 - val_mae: 21.3218
Epoch 8/10
13/13 [==============================] - 0s 5ms/step - loss: 495.5235 - mae: 20.4124 - val_loss: 516.2220 - val_mae: 20.8990
Epoch 9/10
13/13 [==============================] - 0s 5ms/step - loss: 477.9213 - mae: 19.9843 - val_loss: 497.9936 - val_mae: 20.4632
Epoch 10/10
13/13 [==============================] - 0s 5ms/step - loss: 459.9037 - mae: 19.5355 - val_loss: 479.3126 - val_mae: 20.0070
model.save('my_model_with_a_custom_layer2')
INFO:tensorflow:Assets written to: my_model_with_a_custom_layer2/assets
newmodel = tf.keras.models.load_model('my_model_with_a_custom_layer2')
history1=newmodel.fit(x_train_scaled,y_train,epochs=5,validation_data=(x_test_scaled,y_test))
Epoch 1/5
13/13 [==============================] - 1s 13ms/step - loss: 441.9317 - mae: 19.0821 - val_loss: 461.3201 - val_mae: 19.5506
Epoch 2/5
13/13 [==============================] - 0s 5ms/step - loss: 424.6645 - mae: 18.6328 - val_loss: 443.4474 - val_mae: 19.0898
Epoch 3/5
13/13 [==============================] - 0s 5ms/step - loss: 407.4524 - mae: 18.1880 - val_loss: 425.7647 - val_mae: 18.6200
Epoch 4/5
13/13 [==============================] - 0s 5ms/step - loss: 390.4624 - mae: 17.7256 - val_loss: 408.2603 - val_mae: 18.1513
Epoch 5/5
13/13 [==============================] - 0s 5ms/step - loss: 373.4209 - mae: 17.2619 - val_loss: 390.7575 - val_mae: 17.6752
接着看一个多输入多输出的层
class MyMultiLayer(tf.keras.layers.Layer):
def call(self,x):
x1,x2=x
print("x1.shape:",x1.shape,"x2.shape:",x2.shape)
return x1+x2,x1*x2,x1/x2
inputs1 = tf.keras.layers.Input(shape=[2])
inputs2 = tf.keras.layers.Input(shape=[2])
MyMultiLayer()((inputs1,inputs2))
x1.shape: (None, 2) x2.shape: (None, 2)
(,
,
)
以上只是输放Placeholder的输入,当然也可以输入实际数值
x1,x2=np.array([[3.,6],[2.,7]]),np.array([[6.,12],[4.,3]])
MyMultiLayer()((x1,x2))
x1.shape: (2, 2) x2.shape: (2, 2)
(,
,
)
接着创建一个在训练和推理过程中表现不一样的层
class MyGaussianNoise(tf.keras.layers.Layer):
def __init__(self,stddev,**kwargs):
super().__init__(**kwargs)
self.stddev = stddev
def call(self,x,training=None):
if training:
noise = tf.random.normal(tf.shape(x),stddev=self.stddev)
return x+noise
else:
return x
def get_config(self):
base_config = super().get_config()
all_config={**base_config,'stddev':self.stddev}
return all_config
tf.random.set_seed(42)
input_shape = x_train.shape[1:]
model = tf.keras.Sequential([
MyGaussianNoise(1.0,input_shape=input_shape),
tf.keras.layers.Dense(30,activation='relu',kernel_initializer='he_normal'),
tf.keras.layers.Dense(1),
]
)
model.compile(loss='mse',optimizer='adam',metrics=["mae"])
history=model.fit(x_train_scaled,y_train,epochs=10,validation_data=(x_test_scaled,y_test))
Epoch 1/10
13/13 [==============================] - 1s 14ms/step - loss: 468.3629 - mae: 19.6780 - val_loss: 516.2980 - val_mae: 20.8568
Epoch 2/10
13/13 [==============================] - 0s 6ms/step - loss: 440.2289 - mae: 19.0264 - val_loss: 499.2351 - val_mae: 20.4551
Epoch 3/10
13/13 [==============================] - 0s 5ms/step - loss: 427.3663 - mae: 18.7168 - val_loss: 482.1429 - val_mae: 20.0414
Epoch 4/10
13/13 [==============================] - 0s 6ms/step - loss: 410.8777 - mae: 18.2924 - val_loss: 465.1442 - val_mae: 19.6237
Epoch 5/10
13/13 [==============================] - 0s 6ms/step - loss: 388.5227 - mae: 17.6104 - val_loss: 448.0427 - val_mae: 19.1952
Epoch 6/10
13/13 [==============================] - 0s 6ms/step - loss: 372.6901 - mae: 17.1882 - val_loss: 430.6292 - val_mae: 18.7507
Epoch 7/10
13/13 [==============================] - 0s 5ms/step - loss: 357.3874 - mae: 16.8031 - val_loss: 413.2674 - val_mae: 18.2961
Epoch 8/10
13/13 [==============================] - 0s 5ms/step - loss: 338.7885 - mae: 16.2308 - val_loss: 395.8601 - val_mae: 17.8260
Epoch 9/10
13/13 [==============================] - 0s 6ms/step - loss: 319.7017 - mae: 15.6614 - val_loss: 378.3233 - val_mae: 17.3410
Epoch 10/10
13/13 [==============================] - 0s 6ms/step - loss: 300.2906 - mae: 15.0687 - val_loss: 360.9408 - val_mae: 16.8501
model.save('my_model_with_a_custom_layer3/')
INFO:tensorflow:Assets written to: my_model_with_a_custom_layer3/assets
model.layers[0].get_config()
{'name': 'my_gaussian_noise_4',
'trainable': True,
'batch_input_shape': (None, 13),
'dtype': 'float32',
'stddev': 1.0}
newmodel = tf.keras.models.load_model('my_model_with_a_custom_layer3')
history1=newmodel.fit(x_train_scaled,y_train,epochs=5,validation_data=(x_test_scaled,y_test))
Epoch 1/5
13/13 [==============================] - 0s 14ms/step - loss: 278.7925 - mae: 14.3733 - val_loss: 343.0135 - val_mae: 16.3244
Epoch 2/5
13/13 [==============================] - 0s 6ms/step - loss: 254.7551 - mae: 13.5867 - val_loss: 325.6881 - val_mae: 15.7965
Epoch 3/5
13/13 [==============================] - 0s 5ms/step - loss: 243.1024 - mae: 13.2733 - val_loss: 308.8557 - val_mae: 15.2673
Epoch 4/5
13/13 [==============================] - 0s 5ms/step - loss: 231.9742 - mae: 12.8450 - val_loss: 292.6225 - val_mae: 14.7712
Epoch 5/5
13/13 [==============================] - 0s 5ms/step - loss: 215.9254 - mae: 12.1293 - val_loss: 276.9524 - val_mae: 14.3115
newmodel.layers[0].get_config()
{'name': 'my_gaussian_noise_4',
'trainable': True,
'batch_input_shape': (None, 13),
'dtype': 'float32',
'stddev': 1.0}
可以看出,模型实现原模型加载
4 自定义Metrics
tf.random.set_seed(42)
input_shape = x_train.shape[1:]
model = tf.keras.Sequential([
tf.keras.layers.Dense(30,activation='relu',kernel_initializer='he_normal',input_shape=input_shape),
tf.keras.layers.Dense(1)
]
)
model.compile(loss="mse",optimizer='adam',metrics=[create_huber(2.0)])
history2=model.fit(x_train_scaled,y_train,epochs=3)
2022-07-26 10:29:30.486789: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-07-26 10:29:31.131913: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 14637 MB memory: -> device: 3, name: Tesla V100-PCIE-16GB-LS, pci bus id: 0000:dd:00.0, compute capability: 7.0
Epoch 1/3
13/13 [==============================] - 3s 4ms/step - loss: 491.9991 - huber_fn: 38.6896
Epoch 2/3
13/13 [==============================] - 0s 4ms/step - loss: 474.2117 - huber_fn: 37.8320
Epoch 3/3
13/13 [==============================] - 0s 3ms/step - loss: 456.2298 - huber_fn: 36.9550
如果损失函数和评价指标使用同一个函数，你可能会看到结果有一些略微的不同.这是因为计算的操作顺序不完全相同，所以可能存以微小的错误。特别是如何使用 sample weights和class weights那么，结果将更加不一样。
- fit()方法会一直跟踪记录所有batch的平均的损失，从训练开始起就这样。每个batch的损失值是由每个样本的损失，如果这个过程有样本或分类加权，会算上去的，求和然后除以样本数得来的。
- 而评价指标是从训练开始，记录所有单个样本的损失，然后然后算平均，再加权算
precision = tf.keras.metrics.Precision()
precision([0,1,1,1,0,1,0,1],[1,1,0,1,0,1,0,1])
precision([0,1,0,0,1,0,1,1],[1,0,1,1,0,0,0,0])
precision.variables
[,
]
precision.reset_states()
接着改一下
class HuberMetric(tf.keras.metrics.Metric):
def __init__(self,threshold=1.0,**kwargs):
super().__init__(**kwargs)
self.threshold = threshold
self.huber_fn = create_huber(threshold)
self.total = self.add_weight("total",initializer="zeros")
self.count = self.add_weight("count",initializer="zeros")
def update_state(self,y_true,y_pred,sample_weight=None):
sample_metric = self.huber_fn(y_true,y_pred)
self.total.assign_add(tf.reduce_sum(sample_metric))
self.count.assign_add(tf.cast(tf.size(y_true),tf.float32))
def result(self):
return self.total/self.count
def get_config(self):
base_config = super().get_config()
all_config = {**base_config,'threshold':self.threshold}
return all_config
m = HuberMetric(2.0)
# total = 2*|2-10|-2^2/2=14.0
# count = 1
# result = 14.0/1.0=14.0
m(tf.constant([2.0]),tf.constant([10.0]))
# total =total+ (|1.0-0|^2)/2 + (2*|5-9.25|-2^2/2)=14.0+7.0=21.0
# count = count +2=3
# result = total/count=21/3=7
m(tf.constant([[0.],[5]]),tf.constant([[1.],[9.25]]))
m.result()
m.variables
[,
]
m.reset_states()
m.variables
[,
]
检查我们自定义的hubermetric运行良好
tf.random.set_seed(42)
input_shape = x_train.shape[1:]
model = tf.keras.Sequential([
tf.keras.layers.Dense(30,activation='relu',kernel_initializer='he_normal',input_shape=input_shape),
tf.keras.layers.Dense(1)
]
)
model.compile(loss=create_huber(2.0),optimizer='adam',metrics=[HuberMetric(2.0)])
history=model.fit(x_train_scaled,y_train,epochs=3)
Epoch 1/3
13/13 [==============================] - 0s 3ms/step - loss: 38.6856 - huber_metric_6: 38.6856
Epoch 2/3
13/13 [==============================] - 0s 3ms/step - loss: 37.8080 - huber_metric_6: 37.8080
Epoch 3/3
13/13 [==============================] - 0s 2ms/step - loss: 36.9100 - huber_metric_6: 36.9100
model.save("my_model_with_a_custom_metric")
INFO:tensorflow:Assets written to: my_model_with_a_custom_metric/assets
new_model = tf.keras.models.load_model("my_model_with_a_custom_metric/")
history1=new_model.fit(x_train_scaled,y_train,epochs=3)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/tmp/ipykernel_21219/1845581953.py in
----> 1 new_model = tf.keras.models.load_model("my_model_with_a_custom_metric/")
2 history1=newmodel.fit(x_train_scaled,y_train,epochs=3)
~/anaconda3/envs/tf/lib/python3.7/site-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)
65 except Exception as e: # pylint: disable=broad-except
66 filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67 raise e.with_traceback(filtered_tb) from None
68 finally:
69 del filtered_tb
~/anaconda3/envs/tf/lib/python3.7/site-packages/keras/saving/saved_model/load.py in revive_custom_object(identifier, metadata)
993 else:
994 raise ValueError(
--> 995 f'Unable to restore custom object of type {identifier}. '
996 f'Please make sure that any custom layers are included in the '
997 f'`custom_objects` arg when calling `load_model()` and make sure that '
ValueError: Unable to restore custom object of type _tf_keras_metric. Please make sure that any custom layers are included in the `custom_objects` arg when calling `load_model()` and make sure that all layers implement `get_config` and `from_config`.
new_model = tf.keras.models.load_model("my_model_with_a_custom_metric/",custom_objects={'huber_fn':create_huber(2.0),'HuberMetric':HuberMetric})
history1=new_model.fit(x_train_scaled,y_train,epochs=3)
Epoch 1/3
13/13 [==============================] - 1s 3ms/step - loss: 35.9728 - huber_metric_6: 35.9728
Epoch 2/3
13/13 [==============================] - 0s 3ms/step - loss: 35.0231 - huber_metric_6: 35.0231
Epoch 3/3
13/13 [==============================] - 0s 3ms/step - loss: 34.0567 - huber_metric_6: 34.0567
model.metrics包括损失和评价指标，所以model.meterics[-1]是我们自定义的hubermetric
model.metrics
[,
<__main__.HuberMetric at 0x7fb95c7a2f90>]
tf.keras.metrics.serialize(model.metrics[-1])
{'class_name': 'HuberMetric',
'config': {'name': 'huber_metric_6', 'dtype': 'float32', 'threshold': 2.0}}
这个类的更简单的定义方式可以是：
class HuberMetric(tf.keras.metrics.Mean):
def __init__(self,threshold=1.0,name='HuberMetric',dtype=None):
super().__init__(name='HuberMetric',dtype=None)
self.threshold = threshold
self.huber_fn = create_huber(threshold)
def update_state(self,y_true,y_pred,sample_weight=None):
sample_metric = self.huber_fn(y_true,y_pred)
super(HuberMetric,self).update_state(sample_metric,sample_weight)
def get_config(self):
base_config = super().get_config()
all_config = {**base_config,'threshold':self.threshold}
return all_config
这种定义方式可以更好的处理输入形状和sample weights
tf.random.set_seed(42)
input_shape = x_train.shape[1:]
model = tf.keras.Sequential([
tf.keras.layers.Dense(30,activation='relu',kernel_initializer='he_normal',input_shape=input_shape),
tf.keras.layers.Dense(1)
]
)
model.compile(loss=tf.keras.losses.Huber(2.0),optimizer='adam',weighted_metrics=[HuberMetric(2.0)])
np.random.seed(42)
sample_weight = np.random.rand(len(y_train))
history=model.fit(x_train_scaled,y_train,epochs=3,sample_weight=sample_weight)
Epoch 1/3
13/13 [==============================] - 1s 4ms/step - loss: 19.2427 - HuberMetric: 38.8675
Epoch 2/3
13/13 [==============================] - 0s 3ms/step - loss: 18.8211 - HuberMetric: 38.0161
Epoch 3/3
13/13 [==============================] - 0s 3ms/step - loss: 18.3968 - HuberMetric: 37.1590
(history.history['loss'][0],history.history['HuberMetric'][0]*sample_weight.mean())
(19.242660522460938, 19.242658563073878)
model.save("my_model_with_a_custom_metric2")
INFO:tensorflow:Assets written to: my_model_with_a_custom_metric2/assets
new_model = tf.keras.models.load_model("my_model_with_a_custom_metric2/")
history1=new_model.fit(x_train_scaled,y_train,epochs=3)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/tmp/ipykernel_20639/4045318839.py in
----> 1 new_model = tf.keras.models.load_model("my_model_with_a_custom_metric2/")
2 history1=new_model.fit(x_train_scaled,y_train,epochs=3)
~/anaconda3/envs/tf/lib/python3.7/site-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)
65 except Exception as e: # pylint: disable=broad-except
66 filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67 raise e.with_traceback(filtered_tb) from None
68 finally:
69 del filtered_tb
~/anaconda3/envs/tf/lib/python3.7/site-packages/keras/saving/saved_model/load.py in revive_custom_object(identifier, metadata)
993 else:
994 raise ValueError(
--> 995 f'Unable to restore custom object of type {identifier}. '
996 f'Please make sure that any custom layers are included in the '
997 f'`custom_objects` arg when calling `load_model()` and make sure that '
ValueError: Unable to restore custom object of type _tf_keras_metric. Please make sure that any custom layers are included in the `custom_objects` arg when calling `load_model()` and make sure that all layers implement `get_config` and `from_config`.
new_model = tf.keras.models.load_model("my_model_with_a_custom_metric2/",custom_objects={"HuberMetric":HuberMetric})
history1=new_model.fit(x_train_scaled,y_train,epochs=3)
Epoch 1/3
13/13 [==============================] - 0s 3ms/step - loss: 36.0534 - HuberMetric: 36.0534
Epoch 2/3
13/13 [==============================] - 0s 3ms/step - loss: 35.1053 - HuberMetric: 35.1053
Epoch 3/3
13/13 [==============================] - 0s 3ms/step - loss: 34.1408 - HuberMetric: 34.1408
5 关于serialize deserialize get register_keras_serializable的使用
这三个函数的使用还有一个要配合的是tf.keras.utils.register_keras_serializable.
通过一些代码我们来研究一下这三个函数的作用.
其中register_keras_serializable相当重要，可能使用户在使用tf.keras.models.load_model(custom_objects={…}),custom_objects不用由用户额外提供。也看到custom_objects是一个字典，key是我们自定义函数的名称。
5.1 serialize
maxnorm = tf.keras.constraints.MaxNorm(3,0)
output=tf.keras.constraints.serialize(maxnorm)
print(f"type:{type(output)} content:{output}")
print("get_config:",maxnorm.get_config())
type: content:{'class_name': 'MaxNorm', 'config': {'max_value': 3, 'axis': 0}}
get_config: {'max_value': 3, 'axis': 0}
可以看到serialize是比get_config反回的内空还要多，其中class_name的内容就是custom_object字典的key,但自定义的层的class_name就是自定义的函数或类的名称，关于类的查看5.4节
def relu_my(input):
return tf.nn.relu(input)
output1 = tf.keras.constraints.serialize(relu_my)
output2 = tf.keras.activations.serialize(relu_my)
print(f"output1:{output1} output2:{output2}")
output1:relu_my output2:relu_my
5.2 deserialize
这个是个serialize相反的一对，可以从serialize的结果中反向获得要定义的对像，大致是这么个意思
maxnorm1=tf.keras.constraints.deserialize(output)
print(maxnorm1.get_config())
{'max_value': 3, 'axis': 0}
maxnorm1是和maxnorm有相同参数的一个新的层，也就是说我们可以能过deserialize重新获得这个层的定义
5.3 get
get是为了使用字符串来获得要定义的层，比如maxnorm的serialize的output,可以看到class_name
maxnorm2 = tf.keras.constraints.get('MaxNorm')
print(maxnorm2.get_config())
{'max_value': 2, 'axis': 0}
maxnorm2是一个拥有初始化参数的层，所以get的作用有限
5.4 register_keras_serializable
对比使用和不使用，及使用不同方式时的结果.首先查看不注册的情况下：
class NonNegative(tf.keras.constraints.Constraint):
def __call__(self,w):
return w*tf.cast(tf.math.greater_equal(w,0),w.dtype)
def get_config(self):
base_config=super().get_config()
return base_config
non = NonNegative()
out = tf.keras.constraints.serialize(non)
print(out)
{'class_name': 'NonNegative', 'config': {}}
print(tf.keras.constraints.get('NonNegative'))
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/tmp/ipykernel_19639/1015723810.py in
----> 1 print(tf.keras.constraints.get('NonNegative'))
~/anaconda3/envs/tf/lib/python3.7/site-packages/keras/constraints.py in get(identifier)
341 elif isinstance(identifier, str):
342 config = {'class_name': str(identifier), 'config': {}}
--> 343 return deserialize(config)
344 elif callable(identifier):
345 return identifier
~/anaconda3/envs/tf/lib/python3.7/site-packages/keras/constraints.py in deserialize(config, custom_objects)
329 module_objects=globals(),
330 custom_objects=custom_objects,
--> 331 printable_module_name='constraint')
332
333
~/anaconda3/envs/tf/lib/python3.7/site-packages/keras/utils/generic_utils.py in deserialize_keras_object(identifier, module_objects, custom_objects, printable_module_name)
663 config = identifier
664 (cls, cls_config) = class_and_config_for_serialized_keras_object(
--> 665 config, module_objects, custom_objects, printable_module_name)
666
667 # If this object has already been loaded (i.e. it's shared between multiple
~/anaconda3/envs/tf/lib/python3.7/site-packages/keras/utils/generic_utils.py in class_and_config_for_serialized_keras_object(config, module_objects, custom_objects, printable_module_name)
561 if cls is None:
562 raise ValueError(
--> 563 f'Unknown {printable_module_name}: {class_name}. Please ensure this '
564 'object is passed to the `custom_objects` argument. See '
565 'https://www.tensorflow.org/guide/keras/save_and_serialize'
ValueError: Unknown constraint: NonNegative. Please ensure this object is passed to the `custom_objects` argument. See https://www.tensorflow.org/guide/keras/save_and_serialize#registering_the_custom_object for details.
可以看到无法使用get这个方法
print(tf.keras.constraints.deserialize(out))
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/tmp/ipykernel_19639/3405952053.py in
----> 1 print(tf.keras.constraints.deserialize(out))
~/anaconda3/envs/tf/lib/python3.7/site-packages/keras/constraints.py in deserialize(config, custom_objects)
329 module_objects=globals(),
330 custom_objects=custom_objects,
--> 331 printable_module_name='constraint')
332
333
~/anaconda3/envs/tf/lib/python3.7/site-packages/keras/utils/generic_utils.py in deserialize_keras_object(identifier, module_objects, custom_objects, printable_module_name)
663 config = identifier
664 (cls, cls_config) = class_and_config_for_serialized_keras_object(
--> 665 config, module_objects, custom_objects, printable_module_name)
666
667 # If this object has already been loaded (i.e. it's shared between multiple
~/anaconda3/envs/tf/lib/python3.7/site-packages/keras/utils/generic_utils.py in class_and_config_for_serialized_keras_object(config, module_objects, custom_objects, printable_module_name)
561 if cls is None:
562 raise ValueError(
--> 563 f'Unknown {printable_module_name}: {class_name}. Please ensure this '
564 'object is passed to the `custom_objects` argument. See '
565 'https://www.tensorflow.org/guide/keras/save_and_serialize'
ValueError: Unknown constraint: NonNegative. Please ensure this object is passed to the `custom_objects` argument. See https://www.tensorflow.org/guide/keras/save_and_serialize#registering_the_custom_object for details.
也无法使用deserialize。要想使用，需要register_keras_serializable
接着进行使用注册
@tf.keras.utils.register_keras_serializable()
class NonNegative(tf.keras.constraints.Constraint):
def __call__(self,w):
return w*tf.cast(tf.math.greater_equal(w,0),w.dtype)
def get_config(self):
base_config=super().get_config()
return base_config
non = NonNegative()
out = tf.keras.constraints.serialize(non)
print(out)
{'class_name': 'Custom>NonNegative', 'config': {}}
tf.keras.utils.register_keras_serializable()里边有两个参数，我们接着看,其中package默认参数是Custom
@tf.keras.utils.register_keras_serializable(package="cc",name=None)
class NonNegative(tf.keras.constraints.Constraint):
def __call__(self,w):
return w*tf.cast(tf.math.greater_equal(w,0),w.dtype)
def get_config(self):
base_config=super().get_config()
return base_config
non = NonNegative()
out = tf.keras.constraints.serialize(non)
print(out)
{'class_name': 'cc>NonNegative', 'config': {}}
@tf.keras.utils.register_keras_serializable(name="ccc")
class NonNegative(tf.keras.constraints.Constraint):
def __call__(self,w):
return w*tf.cast(tf.math.greater_equal(w,0),w.dtype)
def get_config(self):
base_config=super().get_config()
return base_config
non = NonNegative()
out = tf.keras.constraints.serialize(non)
print(out)
{'class_name': 'Custom>ccc', 'config': {}}
@tf.keras.utils.register_keras_serializable(package='',name="dd")
class NonNegative(tf.keras.constraints.Constraint):
def __call__(self,w):
return w*tf.cast(tf.math.greater_equal(w,0),w.dtype)
def get_config(self):
base_config=super().get_config()
return base_config
non = NonNegative()
out = tf.keras.constraints.serialize(non)
print(out)
{'class_name': '>dd', 'config': {}}
@tf.keras.utils.register_keras_serializable(package='ok',name="dd")
class NonNegative(tf.keras.constraints.Constraint):
def __call__(self,w):
return w*tf.cast(tf.math.greater_equal(w,0),w.dtype)
def get_config(self):
base_config=super().get_config()
return base_config
non = NonNegative()
out = tf.keras.constraints.serialize(non)
print(out)
{'class_name': 'ok>dd', 'config': {}}
print(tf.keras.constraints.get('ok>dd'))
print(non)
print(tf.keras.constraints.deserialize(out))
<__main__.NonNegative object at 0x7fcfcf681dd0>
<__main__.NonNegative object at 0x7fd08328d2d0>
<__main__.NonNegative object at 0x7fcfcf6813d0>
总结来说就是package name两个参数分别决定了class_name中’>'这个符号的前边和后边部分，而使用register_keras_serializable可以方便用户在使用自定义的东西时，不需要在加载模型时tf.keras.models.load_model 中传入custom_objects,这对模型部署是极不安全和便利的。以上1到4节部分的内容,都可以加上这个，就不用写custom_objects了
有多种serialize,deserialize,get的组合：tf.keras.layers,tf.keras.initializers,tf.keras.regularizers,tf.keras.losses,tf.keras.constriants等，具体的要上官方文档查看一下
import tensorflow as tf
2022-07-28 20:38:53.365900: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
对于其它普通的类或函数也可以进行序列化
@tf.keras.utils.register_keras_serializable(package="test")
class TestSample:
def __call__(self,shape=None,dtype='float32'):
print("test")
def get_config(self):
return{}
test= TestSample()
print(test())
print(tf.keras.utils.serialize_keras_object(test))
test
None
{'class_name': 'test>TestSample', 'config': {}}
@tf.keras.utils.register_keras_serializable(package='test1')
def test(shape=None):
print("test1")
return None
print(tf.keras.utils.serialize_keras_object(test))
test1>test