https://drive.google.com/drive/
https://github.com/dream80/DeepFaceLab_Colab.git
Running trainer.
Loading model...
Model first run. Enter model options as default for each run.
Write preview history? (y/n ?:help skip:n) : y
Target iteration (skip:unlimited/default) :
0
Batch_size (?:help skip:0) :
0
Feed faces to network sorted by yaw? (y/n ?:help skip:n) : y
Flip faces randomly? (y/n ?:help skip:y) : y
Src face scale modifier % ( -30...30, ?:help skip:0) :
0
Use lightweight autoencoder? (y/n, ?:help skip:n) : y
Use pixel loss? (y/n, ?:help skip: n/default ) : y
Using TensorFlow backend.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
Loading: 100% 654/654 [02:07<00:00, 4.00it/s]
Sorting: 100% 64/64 [00:00<00:00, 4285.50it/s]
Loading: 100% 1523/1523 [05:12<00:00, 6.61it/s]
Sorting: 100% 64/64 [00:00<00:00, 668.73it/s]
===== Model summary =====
== Model name: H128
==
== Current iteration: 0
==
== Model options:
== |== write_preview_history : True
== |== batch_size : 4
== |== sort_by_yaw : True
== |== random_flip : True
== |== lighter_ae : True
== |== pixel_loss : True
== Running on:
== |== [0 : Tesla T4]
=========================
Starting. Press "Enter" to stop training and save model.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
[05:39:20][#003823][0208ms] loss_src:0.445 loss_dst:0.126
[05:54:20][#007671][0212ms] loss_src:0.613 loss_dst:0.136
[06:09:20][#011504][0210ms] loss_src:0.350 loss_dst:0.084
[06:24:21][#015315][0207ms] loss_src:0.287 loss_dst:0.065
[06:39:21][#019107][0206ms] loss_src:0.253 loss_dst:0.132
[06:54:21][#022889][0205ms] loss_src:0.447 loss_dst:0.065
[07:09:21][#026667][0206ms] loss_src:0.500 loss_dst:0.060
[07:24:21][#030449][0206ms] loss_src:0.223 loss_dst:0.071
[07:39:21][#034204][0205ms] loss_src:0.163 loss_dst:0.050
[07:54:21][#037950][0204ms] loss_src:0.355 loss_dst:0.060
[08:09:21][#041685][0205ms] loss_src:0.181 loss_dst:0.083
[08:24:21][#045403][0204ms] loss_src:0.228 loss_dst:0.068
[08:39:22][#049121][0208ms] loss_src:0.319 loss_dst:0.060
[08:54:22][#052831][0205ms] loss_src:0.188 loss_dst:0.036
[09:09:22][#056534][0206ms] loss_src:0.375 loss_dst:0.051
Enter
[09:24:01][#060135][0206ms] loss_src:0.270 loss_dst:0.076
[09:24:06][#060156][0211ms] loss_src:0.138 loss_dst:0.036
[09:24:22][#060221][0210ms] loss_src:0.427 loss_dst:0.098
[09:25:28][#060485][0207ms] loss_src:0.258 loss_dst:0.055
[09:25:31][#060498][0206ms] loss_src:0.150 loss_dst:0.042
[09:25:32][#060503][0206ms] loss_src:0.217 loss_dst:0.050
[09:25:32][#060504][0209ms] loss_src:0.135 loss_dst:0.051
[09:39:22][#063894][0206ms] loss_src:0.237 loss_dst:0.052
[09:54:22][#067565][0209ms] loss_src:0.172 loss_dst:0.041
[10:09:23][#071221][0202ms] loss_src:0.136 loss_dst:0.032
[10:24:23][#074871][0206ms] loss_src:0.167 loss_dst:0.089
[10:39:23][#078514][0203ms] loss_src:0.209 loss_dst:0.052
[10:54:23][#082151][0205ms] loss_src:0.191 loss_dst:0.059
[11:09:23][#085771][0209ms] loss_src:0.230 loss_dst:0.071
[11:24:23][#089392][0203ms] loss_src:0.268 loss_dst:0.050
[11:39:23][#092995][0208ms] loss_src:0.152 loss_dst:0.086
[11:54:23][#096591][0202ms] loss_src:0.145 loss_dst:0.051
[12:09:23][#100178][0209ms] loss_src:0.185 loss_dst:0.030
[12:10:20][#100400][0207ms] loss_src:0.128 loss_dst:0.045
[12:10:24][#100411][0209ms] loss_src:0.148 loss_dst:0.032
[12:10:24][#100412][0199ms] loss_src:0.121 loss_dst:0.052
11
[12:10:26][#100422][0202ms] loss_src:0.153 loss_dst:0.029
2
exit
[12:10:41][#100482][0205ms] loss_src:0.183 loss_dst:0.060
print(123)
Traceback (most recent call last):
File "main.py", line 205, in
arguments.func(arguments)
File "main.py", line 93, in process_train
Trainer.main(args, device_args)
File "/content/drive/My Drive/DeepFaceLab/DeepFaceLab_Colab/mainscripts/Trainer.py", line 150, in main
io.process_messages(0.1)
File "/content/drive/My Drive/DeepFaceLab/DeepFaceLab_Colab/interact/interact.py", line 118, in process_messages
time.sleep(sleep_time)
KeyboardInterrupt
Process Process-3:
Process Process-4:
Process Process-1:
Process Process-2:
Traceback (most recent call last):
Traceback (most recent call last):
File "/content/drive/My Drive/DeepFaceLab/DeepFaceLab_Colab/samples/SampleGeneratorFace.py", line 131, in batch_func
x = SampleProcessor.process (sample, self.sample_process_options, self.output_sample_types, self.debug)
File "/content/drive/My Drive/DeepFaceLab/DeepFaceLab_Colab/samples/SampleGeneratorFace.py", line 131, in batch_func
x = SampleProcessor.process (sample, self.sample_process_options, self.output_sample_types, self.debug)
File "/content/drive/My Drive/DeepFaceLab/DeepFaceLab_Colab/samples/SampleProcessor.py", line 169, in process
img = cv2.warpAffine( img, LandmarksProcessor.get_transform_mat (sample.landmarks, size, target_face_type), (size,size), flags=cv2.INTER_CUBIC )
KeyboardInterrupt
During handling of the above exception, another exception occurred:
File "/content/drive/My Drive/DeepFaceLab/DeepFaceLab_Colab/samples/SampleProcessor.py", line 169, in process
img = cv2.warpAffine( img, LandmarksProcessor.get_transform_mat (sample.landmarks, size, target_face_type), (size,size), flags=cv2.INTER_CUBIC )
File "/content/drive/My Drive/DeepFaceLab/DeepFaceLab_Colab/facelib/LandmarksProcessor.py", line 139, in get_transform_mat
mat = umeyama(image_landmarks[17:], landmarks_2D, True)[0:2]
Traceback (most recent call last):
File "/content/drive/My Drive/DeepFaceLab/DeepFaceLab_Colab/mathlib/umeyama.py", line 28, in umeyama
src_mean = src.mean(axis=0)
File "/usr/local/lib/python3.6/dist-packages/numpy/core/_methods.py", line 69, in _mean
if issubclass(arr.dtype.type, (nt.integer, nt.bool_)):
KeyboardInterrupt
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/content/drive/My Drive/DeepFaceLab/DeepFaceLab_Colab/utils/iter_utils.py", line 39, in process_func
gen_data = next (self.generator_func)
File "/content/drive/My Drive/DeepFaceLab/DeepFaceLab_Colab/samples/SampleGeneratorFace.py", line 133, in batch_func
raise Exception ("Exception occured in sample %s. Error: %s" % (sample.filename, traceback.format_exc() ) )
Exception: Exception occured in sample /content/drive/My Drive/DeepFaceLab/workspace/data_dst/aligned/00487.jpg. Error: Traceback (most recent call last):
File "/content/drive/My Drive/DeepFaceLab/DeepFaceLab_Colab/samples/SampleGeneratorFace.py", line 131, in batch_func
x = SampleProcessor.process (sample, self.sample_process_options, self.output_sample_types, self.debug)
File "/content/drive/My Drive/DeepFaceLab/DeepFaceLab_Colab/samples/SampleProcessor.py", line 169, in process
img = cv2.warpAffine( img, LandmarksProcessor.get_transform_mat (sample.landmarks, size, target_face_type), (size,size), flags=cv2.INTER_CUBIC )
KeyboardInterrupt
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/content/drive/My Drive/DeepFaceLab/DeepFaceLab_Colab/utils/iter_utils.py", line 39, in process_func
gen_data = next (self.generator_func)
File "/content/drive/My Drive/DeepFaceLab/DeepFaceLab_Colab/samples/SampleGeneratorFace.py", line 133, in batch_func
raise Exception ("Exception occured in sample %s. Error: %s" % (sample.filename, traceback.format_exc() ) )
Exception: Exception occured in sample /content/drive/My Drive/DeepFaceLab/workspace/data_src/aligned/00611.jpg. Error: Traceback (most recent call last):
File "/content/drive/My Drive/DeepFaceLab/DeepFaceLab_Colab/samples/SampleGeneratorFace.py", line 131, in batch_func
x = SampleProcessor.process (sample, self.sample_process_options, self.output_sample_types, self.debug)
File "/content/drive/My Drive/DeepFaceLab/DeepFaceLab_Colab/samples/SampleProcessor.py", line 169, in process
img = cv2.warpAffine( img, LandmarksProcessor.get_transform_mat (sample.landmarks, size, target_face_type), (size,size), flags=cv2.INTER_CUBIC )
File "/content/drive/My Drive/DeepFaceLab/DeepFaceLab_Colab/facelib/LandmarksProcessor.py", line 139, in get_transform_mat
mat = umeyama(image_landmarks[17:], landmarks_2D, True)[0:2]
File "/content/drive/My Drive/DeepFaceLab/DeepFaceLab_Colab/mathlib/umeyama.py", line 28, in umeyama
src_mean = src.mean(axis=0)
File "/usr/local/lib/python3.6/dist-packages/numpy/core/_methods.py", line 69, in _mean
if issubclass(arr.dtype.type, (nt.integer, nt.bool_)):
KeyboardInterrupt
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/content/drive/My Drive/DeepFaceLab/DeepFaceLab_Colab/utils/iter_utils.py", line 45, in process_func
self.sc_queue.get()
File "/usr/lib/python3.6/multiprocessing/queues.py", line 94, in get
res = self._recv_bytes()
File "/usr/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/content/drive/My Drive/DeepFaceLab/DeepFaceLab_Colab/utils/iter_utils.py", line 45, in process_func
self.sc_queue.get()
File "/usr/lib/python3.6/multiprocessing/queues.py", line 94, in get
res = self._recv_bytes()
File "/usr/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
/usr/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
len(cache))
/usr/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
len(cache))
/usr/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
len(cache))
/usr/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
len(cache))
Exception ignored in:
Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 1294, in _shutdown
/usr/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 37 leaked semaphores to clean up at shutdown
len(cache))
^C
#@title
# For preview, training must be stopped
# You can also view it in parallel with another .ipynb.
#https://github.com/dream80/DeepFaceLab_Colab/blob/master/ViewLastHistory_H128.ipynb
import matplotlib.pyplot as plt
import tensorflow as tf
import numpy as np
import os
from google.colab import files
print(tf.__version__)
imgpath="/content/drive/My Drive/DeepFaceLab/workspace/model/H128_history/"
tlist=os.listdir(imgpath)
tlist.sort(key=lambda x:int(x[:-5]))
lastpic=tlist[-1]
image_raw = tf.gfile.GFile(imgpath+lastpic,'rb').read() #bytes
img = tf.image.decode_jpeg(image_raw) #Tensor
plt.rcParams['figure.figsize'] = (8, 4.0)
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'
plt.rcParams['savefig.dpi'] =200
plt.rcParams['figure.dpi'] = 200
with tf.Session() as sess:
plt.imshow(img.eval())
Collecting alignments: 100% 1523/1523 [00:03<00:00, 431.47it/s]
Running on CPU0.
Running on CPU1.
Converting: 27% 413/1538 [05:33<15:08, 1.24it/s]no faces found for 00954.png, copying without faces
Converting: 33% 510/1538 [06:47<13:40, 1.25it/s]no faces found for 01051.png, copying without faces
Converting: 33% 511/1538 [06:47<13:38, 1.25it/s]no faces found for 01052.png, copying without faces
Converting: 34% 522/1538 [06:54<13:27, 1.26it/s]no faces found for 01063.png, copying without faces
Converting: 34% 524/1538 [06:55<13:23, 1.26it/s]no faces found for 01064.png, copying without faces
no faces found for 01065.png, copying without faces
Converting: 65% 998/1538 [13:15<07:10, 1.25it/s]no faces found for 00001.png, copying without faces
Converting: 65% 999/1538 [13:15<07:09, 1.26it/s]no faces found for 00002.png, copying without faces
Converting: 65% 1000/1538 [13:15<07:08, 1.26it/s]no faces found for 00003.png, copying without faces
Converting: 65% 1001/1538 [13:16<07:07, 1.26it/s]no faces found for 00004.png, copying without faces
Converting: 65% 1002/1538 [13:16<07:06, 1.26it/s]no faces found for 00005.png, copying without faces
Converting: 95% 1455/1538 [19:21<01:06, 1.25it/s]no faces found for 00458.png, copying without faces
Converting: 95% 1456/1538 [19:22<01:05, 1.25it/s]no faces found for 00459.png, copying without faces
Converting: 95% 1460/1538 [19:24<01:02, 1.25it/s]no faces found for 00463.png, copying without faces
Converting: 96% 1472/1538 [19:33<00:52, 1.25it/s]no faces found for 00475.png, copying without faces
Converting: 100% 1538/1538 [20:24<00:00, 1.26it/s]/usr/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
len(cache))
/usr/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
len(cache))
Converting: 100% 1538/1538 [20:24<00:00, 1.26it/s]
Done.
然后打开 https://drive.google.com/drive/ 查看结果