最新最全20篇！基于 StyleGAN 改进或应用相关论文
本文所有论文已打包下载，获取方式见文末。
2019.3 A Style-Based Generator Architecture for Generative Adversarial Networks
We propose an alternative generator architecture for
generative adversarial networks, borrowing from style
transfer literature. The new architecture leads to an automatically learned, unsupervised separation of high-level
attributes (e.g., pose and identity when trained on human
faces) and stochastic variation in the generated images
(e.g., freckles, hair), and it enables intuitive, scale-specific
control of the synthesis. The new generator improves the
state-of-the-art in terms of traditional distribution quality
metrics, leads to demonstrably better interpolation properties, and also better disentangles the latent factors of variation. To quantify interpolation quality and disentanglement,
we propose two new, automated methods that are applicable to any generator architecture. Finally, we introduce a
new, highly varied and high-quality dataset of human faces.
001 (2021-02-4) Designing an Encoder for StyleGAN Image Manipulation
https://arxiv.org/pdf/2102.02766.pdf
Recently, there has been a surge of diverse methods for
performing image editing by employing pre-trained unconditional generators. Applying these methods on real images,
however, remains a challenge, as it necessarily requires the
inversion of the images into their latent space. To successfully invert a real image, one needs to find a latent code
that reconstructs the input image accurately, and more importantly, allows for its meaningful manipulation. In this
paper, we carefully study the latent space of StyleGAN, the
state-of-the-art unconditional generator. We identify and
analyze the existence of a distortion-editability tradeoff and
a distortion-perception tradeoff within the StyleGAN latent
space. We then suggest two principles for designing encoders in a manner that allows one to control the proximity
of the inversions to regions that StyleGAN was originally
trained on. We present an encoder based on our two principles that is specifically designed for facilitating editing on
real images by balancing these tradeoffs. By evaluating its
performance qualitatively and quantitatively on numerous
challenging domains, including cars and horses, we show
that our inversion method, followed by common editing techniques, achieves superior real-image editing quality, with
only a small reconstruction accuracy drop.
002 (2021-01-19) Using StyleGAN for Visual Interpretability of Deep Learning Models on Medical Images
https://arxiv.org/pdf/2101.07563.pdf
As AI-based medical devices are becoming more common in imaging fields like
radiology and histology, interpretability of the underlying predictive models is crucial to expand their use in clinical practice. Existing heatmap-based interpretability
methods such as GradCAM only highlight the location of predictive features but
do not explain how they contribute to the prediction. In this paper, we propose a
new interpretability method that can be used to understand the predictions of any
black-box model on images, by showing how the input image would be modified in
order to produce different predictions. A StyleGAN is trained on medical images
to provide a mapping between latent vectors and images. Our method identifies
the optimal direction in the latent space to create a change in the model prediction.
By shifting the latent representation of an input image along this direction, we can
produce a series of new synthetic images with changed predictions. We validate our
approach on histology and radiology images, and demonstrate its ability to provide
meaningful explanations that are more informative than GradCAM heatmaps. Our
method reveals the patterns learned by the model, which allows clinicians to build
trust in the model’s predictions, discover new biomarkers and eventually reveal
potential biases
003 (2021-01-6) VOGUE Try-On by StyleGAN Interpolation Optimization
https://arxiv.org/pdf/2101.02285.pdf
Given an image of a target person and an image of another person wearing a garment, we automatically generate
the target person in the given garment. At the core of our
method is a pose-conditioned StyleGAN2 latent space interpolation, which seamlessly combines the areas of interest
from each image, i.e., body shape, hair, and skin color are
derived from the target person, while the garment with its
folds, material properties, and shape comes from the garment image. By automatically optimizing for interpolation
coefficients per layer in the latent space, we can perform
a seamless, yet true to source, merging of the garment and
target person. Our algorithm allows for garments to deform
according to the given body shape, while preserving pattern
and material details. Experiments demonstrate state-of-theart photo-realistic results at high resolution (512 × 512).
004 (2020-12-13) Improved StyleGAN Embedding Where are the Good Latents
https://arxiv.org/pdf/2012.09036.pdf
StyleGAN is able to produce photorealistic images almost indistinguishable from real ones. Embedding images
into the StyleGAN latent space is not a trivial task due to the
reconstruction quality and editing quality trade-off. In this
paper, we first introduce a new normalized space to analyze the diversity and the quality of the reconstructed latent
codes. This space can help answer the question of where
good latent codes are located in latent space. Second, we
propose a framework to analyze the quality of different embedding algorithms. Third, we propose an improved embedding algorithm based on our analysis. We compare our results with the current state-of-the-art methods and achieve a
better trade-off between reconstruction quality and editing
quality
005 (2020-12-4) MPG A Multi-ingredient Pizza Image Generator with Conditional StyleGANs
https://arxiv.org/pdf/2012.02821.pdf
Multilabel conditional image generation is a challenging
problem in computer vision. In this work we propose Multiingredient Pizza Generator (MPG), a conditional Generative Neural Network (GAN) framework for synthesizing
multilabel images. We design MPG based on a state-of-theart GAN structure called StyleGAN2, in which we develop a
new conditioning technique by enforcing intermediate feature maps to learn scalewise label information. Because
of the complex nature of the multilabel image generation
problem, we also regularize synthetic image by predicting
the corresponding ingredients as well as encourage the discriminator to distinguish between matched image and mismatched image. To verify the efficacy of MPG, we test it
on Pizza10, which is a carefully annotated multi-ingredient
pizza image dataset. MPG can successfully generate photorealist pizza images with desired ingredients. The framework can be easily extend to other multilabel image generation scenarios
006 (2020-11-26) Lifting 2D StyleGAN for 3D-Aware Face Generation
https://arxiv.org/pdf/2011.13126.pdf
We propose a framework, called LiftedGAN, that disentangles and lifts a pre-trained StyleGAN2 for 3D-aware face
generation. Our model is “3D-aware” in the sense that
it is able to (1) disentangle the latent space of StyleGAN2
into texture, shape, viewpoint, lighting and (2) generate 3D
components for rendering synthetic images. Unlike most previous methods, our method is completely self-supervised, i.e.
it neither requires any manual annotation nor 3DMM model
for training. Instead, it learns to generate images as well
as their 3D components by distilling the prior knowledge
in StyleGAN2 with a differentiable renderer. The proposed
model is able to output both the 3D shape and texture, allowing explicit pose and lighting control over generated images.
Qualitative and quantitative results show the superiority
of our approach over existing methods on 3D-controllable
GANs in content controllability while generating realistic
high quality images
007 (2020-12-3) StyleSpace Analysis Disentangled Controls for StyleGAN Image Generation
https://arxiv.org/pdf/2011.12799.pdf
We explore and analyze the latent style space of StyleGAN2, a state-of-the-art architecture for image generation, using models pretrained on several different datasets.
We first show that StyleSpace, the space of channel-wise
style parameters, is significantly more disentangled than
the other intermediate latent spaces explored by previous
works. Next, we describe a method for discovering a large
collection of style channels, each of which is shown to control a distinct visual attribute in a highly localized and disentangled manner. Third, we propose a simple method for
identifying style channels that control a specific attribute,
using a pretrained classifier or a small number of example images. Manipulation of visual attributes via these
StyleSpace controls is shown to be better disentangled than
via those proposed in previous works. To show this, we
make use of a newly proposed Attribute Dependency metric.
Finally, we demonstrate the applicability of StyleSpace controls to the manipulation of real images. Our findings pave
the way to semantically meaningful and well-disentangled
image manipulations via simple and intuitive interfaces.
008 (2020-11-4) Transforming Facial Weight of Real Images by Editing Latent Space of StyleGAN
https://arxiv.org/pdf/2011.02606.pdf
We present an invert-and-edit framework to automatically
transform facial weight of an input face image to look thinner
or heavier by leveraging semantic facial attributes encoded in
the latent space of Generative Adversarial Networks (GANs).
Using a pre-trained StyleGAN as the underlying generator,
we first employ an optimization-based embedding method to
invert the input image into the StyleGAN latent space. Then,
we identify the facial-weight attribute direction in the latent
space via supervised learning and edit the inverted latent
code by moving it positively or negatively along the extracted
feature axis. Our framework is empirically shown to produce
high-quality and realistic facial-weight transformations without requiring training GANs with a large amount of labeled
face images from scratch. Ultimately, our framework can be
utilized as part of an intervention to motivate individuals to
make healthier food choices by visualizing the future impacts
of their behavior on appearance
009 (2020-10-21) One Model to Reconstruct Them All A Novel Way to Use the Stochastic Noise in StyleGAN
https://arxiv.org/pdf/2010.11113.pdf
Generative Adversarial Networks (GANs) have achieved
state-of-the-art performance for several image generation
and manipulation tasks. Different works have improved the
limited understanding of the latent space of GANs by embedding images into specific GAN architectures to reconstruct the original images. We present a novel StyleGANbased autoencoder architecture, which can reconstruct images with very high quality across several data domains. We
demonstrate a previously unknown grade of generalizablility by training the encoder and decoder independently and
on different datasets. Furthermore, we provide new insights
about the significance and capabilities of noise inputs of the
well-known StyleGAN architecture. Our proposed architecture can handle up to 40 images per second on a single
GPU, which is approximately 28× faster than previous approaches. Finally, our model also shows promising results,
when compared to the state-of-the-art on the image denoising task, although it was not explicitly designed for this task
010 (2020-09-14) Improving Inversion and Generation Diversity in StyleGAN using a Gaussianized Latent Space
https://arxiv.org/pdf/2009.06529.pdf
Modern Generative Adversarial Networks are capable of creating artificial, photorealistic images from latent vectors living in a low-dimensional learned latent
space. It has been shown that a wide range of images can be projected into this
space, including images outside of the domain that the generator was trained on.
However, while in this case the generator reproduces the pixels and textures of the
images, the reconstructed latent vectors are unstable and small perturbations result
in significant image distortions. In this work, we propose to explicitly model the
data distribution in latent space. We show that, under a simple nonlinear operation,
the data distribution can be modeled as Gaussian and therefore expressed using
sufficient statistics. This yields a simple Gaussian prior, which we use to regularize
the projection of images into the latent space. The resulting projections lie in
smoother and better behaved regions of the latent space, as shown using interpolation performance for both real and generated images. Furthermore, the Gaussian
model of the distribution in latent space allows us to investigate the origins of
artifacts in the generator output, and provides a method for reducing these artifacts
while maintaining diversity of the generated images
011 (2020-09-20) StyleFlow Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows
https://arxiv.org/pdf/2008.02401.pdf
High-quality, diverse, and photorealistic images can now be generated
by unconditional GANs (e.g., StyleGAN). However, limited options exist
to control the generation process using (semantic) aributes, while still
preserving the quality of the output. Further, due to the entangled nature of
the GAN latent space, performing edits along one aribute can easily result
in unwanted changes along other aributes. In this paper, in the context of
conditional exploration of entangled latent spaces, we investigate the two
sub-problems of aribute-conditioned sampling and aribute-controlled
editing. We present StyleFlow as a simple, eective, and robust solution
to both the sub-problems by formulating conditional exploration as an
instance of conditional continuous normalizing ows in the GAN latent
space conditioned by aribute features. We evaluate our method using the
face and the car latent space of StyleGAN, and demonstrate ne-grained
disentangled edits along various aributes on both real photographs and
StyleGAN generated images. For example, for faces we vary camera pose,
illumination variation, expression, facial hair, gender, and age. Finally, via
extensive qualitative and quantitative comparisons, we demonstrate the
superiority of StyleFlow to other concurrent works.
Project Page : hps://rameenabdal.github.io/StyleFlow
Video : hps://youtu.be/LRAUJUn3EqQw
012 (2020-08-3) Encoding in Style a StyleGAN Encoder for Image-to-Image Translation
https://arxiv.org/pdf/2008.00951.pdf
We present a generic image-to-image translation framework, Pixel2Style2Pixel (pSp). Our pSp framework is based
on a novel encoder network that directly generates a series of style vectors which are fed into a pretrained StyleGAN generator, forming the extended W+ latent space. We
first show that our encoder can directly embed real images
into W+, with no additional optimization. We further introduce a dedicated identity loss which is shown to achieve
improved performance in the reconstruction of an input image. We demonstrate pSp to be a simple architecture that,
by leveraging a well-trained, fixed generator network, can
be easily applied on a wide-range of image-to-image translation tasks. Solving these tasks through the style representation results in a global approach that does not rely
on a local pixel-to-pixel correspondence and further supports multi-modal synthesis via the resampling of styles.
Notably, we demonstrate that pSp can be trained to align
a face image to a frontal pose without any labeled data,
generate multi-modal results for ambiguous tasks such as
conditional face generation from segmentation maps, and
construct high-resolution images from corresponding lowresolution images
013 (2020-07-3) Collaborative Learning for Faster StyleGAN Embedding
https://arxiv.org/pdf/2007.01758.pdf
The latent code of the recent popular model StyleGAN has learned disentangled representations thanks to
the multi-layer style-based generator. Embedding a given
image back to the latent space of StyleGAN enables wide
interesting semantic image editing applications. Although
previous works are able to yield impressive inversion results based on an optimization framework, which however
suffers from the efficiency issue. In this work, we propose
a novel collaborative learning framework that consists of
an efficient embedding network and an optimization-based
iterator. On one hand, with the progress of training, the embedding network gives a reasonable latent code initialization for the iterator. On the other hand, the updated latent
code from the iterator in turn supervises the embedding network. In the end, high-quality latent code can be obtained
efficiently with a single forward pass through our embedding network. Extensive experiments demonstrate the effectiveness and efficiency of our work
014 (2020-09-15) Conditional Spoken Digit Generation with StyleGAN
https://arxiv.org/pdf/2004.13764.pdf
This paper adapts a StyleGAN model for speech generation
with minimal or no conditioning on text. StyleGAN is a multiscale convolutional GAN capable of hierarchically capturing
data structure and latent variation on multiple spatial (or temporal) levels. The model has previously achieved impressive
results on facial image generation, and it is appealing to audio applications due to similar multi-level structures present in
the data. In this paper, we train a StyleGAN to generate melspectrograms on the Speech Commands dataset, which contains
spoken digits uttered by multiple speakers in varying acoustic
conditions. In a conditional setting our model is conditioned
on the digit identity, while learning the remaining data variation
remains an unsupervised task. We compare our model to the
current unsupervised state-of-the-art speech synthesis GAN architecture, the WaveGAN, and show that the proposed model
outperforms according to numerical measures and subjective
evaluation by listening tests
015 (2020-06-13) StyleRig Rigging StyleGAN for 3D Control over Portrait Images
https://arxiv.org/pdf/2004.00121.pdf
StyleGAN [19] generates photorealistic portrait images
of faces with eyes, teeth, hair and context (neck, shoulders,
background), but lacks a rig-like control over semantic face
parameters that are interpretable in 3D, such as face pose,
expressions, and scene illumination. Three-dimensional morphable face models (3DMMs) [10] on the other hand offer
control over the semantic parameters, but lack photorealism
when rendered and only model the face interior, not other
parts of a portrait image (hair, mouth interior, background).
We present the first method to provide a face rig-like control
over a pretrained and fixed StyleGAN via a 3DMM. A new
rigging network, RigNet is trained between the 3DMM’s
semantic parameters and StyleGAN’s input. The network
is trained in a self-supervised manner, without the need for
manual annotations. At test time, our method generates portrait images with the photorealism of StyleGAN and provides
explicit control over the 3D semantic parameters of the face.
016 (2020-03-24) Re-Training StyleGAN -- A First Step Towards Building Large Scalable Synthetic Facial Datasets
https://arxiv.org/pdf/2003.10847.pdf
StyleGAN is a state-of-art generative
adversarial network architecture that generates random 2D
high-quality synthetic facial data samples. In this paper we
recap the StyleGAN architecture and training methodology
and present our experiences of retraining it on a number of
alternative public datasets. Practical issues and challenges
arising from the retraining process are discussed. Tests and
validation results are presented and a comparative analysis of
several different re-trained StyleGAN weightings is provided
1. The role of this tool in building large, scalable datasets of
synthetic facial data is also discussed.
017 (2020-11-25) Semi-Supervised StyleGAN for Disentanglement Learning
https://arxiv.org/pdf/2003.03461.pdf
Disentanglement learning is crucial for obtaining disentangled representations and controllable
generation. Current disentanglement methods
face several inherent limitations: difficulty with
high-resolution images, primarily focusing on
learning disentangled representations, and nonidentifiability due to the unsupervised setting. To
alleviate these limitations, we design new architectures and loss functions based on StyleGAN
(Karras et al., 2019), for semi-supervised highresolution disentanglement learning. We create
two complex high-resolution synthetic datasets
for systematic testing. We investigate the impact of limited supervision and find that using
only 0.25%∼2.5% of labeled data is sufficient for
good disentanglement on both synthetic and real
datasets. We propose new metrics to quantify generator controllability, and observe there may exist
a crucial trade-off between disentangled representation learning and controllable generation. We
also consider semantic fine-grained image editing
to achieve better generalization to unseen images.
018 (2020-03-23) Analyzing and Improving the Image Quality of StyleGAN
https://arxiv.org/pdf/1912.04958.pdf
The style-based GAN architecture (StyleGAN) yields
state-of-the-art results in data-driven unconditional generative image modeling. We expose and analyze several of
its characteristic artifacts, and propose changes in both
model architecture and training methods to address them.
In particular, we redesign the generator normalization, revisit progressive growing, and regularize the generator to
encourage good conditioning in the mapping from latent
codes to images. In addition to improving image quality,
this path length regularizer yields the additional benefit that
the generator becomes significantly easier to invert. This
makes it possible to reliably attribute a generated image to
a particular network. We furthermore visualize how well
the generator utilizes its output resolution, and identify a
capacity problem, motivating us to train larger models for
additional quality improvements. Overall, our improved
model redefines the state of the art in unconditional image
modeling, both in terms of existing distribution quality metrics as well as perceived image quality
019 (2019-09-3) Image2StyleGAN How to Embed Images Into the StyleGAN Latent Space
https://arxiv.org/pdf/1904.03189.pdf
We propose an efficient algorithm to embed a given image into the latent space of StyleGAN. This embedding enables semantic image editing operations that can be applied
to existing photographs. Taking the StyleGAN trained on
the FFHQ dataset as an example, we show results for image
morphing, style transfer, and expression transfer. Studying
the results of the embedding algorithm provides valuable
insights into the structure of the StyleGAN latent space. We
propose a set of experiments to test what class of images can
be embedded, how they are embedded, what latent space is
suitable for embedding, and if the embedding is semantically meaningful
获取方式
扫下面码添加运营小妹，回复 stylegan（建议选中长按复制，如已添加，直接查看朋友圈第一条链接~），即可获取pdf。
猜您喜欢：
超100篇！CVPR 2020最全GAN论文梳理汇总！
附下载 | 《Python进阶》中文版
附下载 | 经典《Think Python》中文版
附下载 | 《Pytorch模型训练实用教程》
附下载 | 最新2020李沐《动手学深度学习》
附下载 | 《可解释的机器学习》中文版
附下载 |《TensorFlow 2.0 深度学习算法实战》
附下载 | 超100篇！CVPR 2020最全GAN论文梳理汇总！
附下载 |《计算机视觉中的数学方法》分享
