计算机视觉开发板
TensorFlow is an end-to-end open-source machine learning platform capable of performing a range of tasks. It provides an ease of use for beginners and researchers alike and can be used to work on different applications like, but not limited to, computer vision, natural language processing, and reinforcement learning.
TensorFlow是能够执行一系列任务的端到端开源机器学习平台。 它为初学者和研究人员提供了一种易用性，并且可以用于不同的应用程序，例如但不限于计算机视觉，自然语言处理和强化学习。
In the computer vision world, most of us are familiar with the core TensorFlow along with TensorFlow Lite and JS. They are used to run models on mobile and edge devices for the former and the web for the latter. However, TensorFlow also offers many more arcane libraries, which we will be unraveling in this article.
在计算机视觉世界中，我们大多数人都熟悉TensorFlow核心以及TensorFlow Lite和JS。 它们用于前者在移动和边缘设备上运行模型，而后者则在网络上运行模型。 但是，TensorFlow还提供了更多的奥术库，我们将在本文中进行阐述。
目录 (Table of Contents)
- TensorFlow model optimization toolkit TensorFlow模型优化工具包
- TensorFlow Graphics TensorFlow图形
- TensorFlow Federated TensorFlow联合
- TensorFlow Privacy TensorFlow隐私
- TensorFlow Hub TensorFlow集线器
TensorFlow模型优化工具包 (TensorFlow Model Optimization Toolkit)
Real-time models are essential for many commercial operations. The inference speed of MobileNet had shone it into the limelight even if it meant sacrificing a little on accuracy. The first thing that comes to mind for optimizing TensorFlow models is to convert it to TensorFlow lite serving. However, that does not work very well on desktops as it is optimized for ARM neon, explained in this issue, or we need to optimize the model even further. The model optimization toolkit comes to our rescue for these tasks. According to its homepage, it can be used to:
实时模型对于许多商业运营至关重要。 MobileNet的推理速度使它成为众人瞩目的焦点，即使这意味着要牺牲一些准确性。 优化TensorFlow模型的第一件事就是将其转换为TensorFlow lite服务。 但是，这在台式机上效果不佳，因为已针对ARM neon进行了优化(如本期所述) ，否则我们需要进一步优化模型。 模型优化工具包可以帮助我们完成这些任务。 根据其主页 ，它可以用于：
Reduce latency and inference cost for cloud and edge devices (e.g. mobile, IoT).
减少云和边缘设备(例如移动设备，物联网)的延迟和推理成本。
Deploy models to edge devices with restrictions on processing, memory, power consumption, network usage, and model storage space.
在处理，内存，功耗，网络使用和模型存储空间受到限制的情况下，将模型部署到边缘设备。
Enable execution on and optimize for existing hardware or new special-purpose accelerators.
在现有的硬件或新的专用加速器上启用执行并对其进行优化。
It can be applied to already trained models as well as during training time to further optimize the solution. It offers three techniques at the time of writing with several others as work in progress to hone the models.
它可以应用于已经训练的模型，也可以在训练期间应用于进一步优化解决方案。 在撰写本文时，它提供了三种技术以完善模型。
修剪 (Pruning)
The first method is weight pruning. It works by removing some connections between layers hence reducing the number of parameters and operations involved and consequently optimizing the model. It eliminates unnecessary values in the weight tensors and is performed during the training process. This is useful in reducing the size of models, which can be further decreased by post-training quantization.
第一种方法是重量修剪。 它通过删除层之间的某些连接来工作，从而减少了涉及的参数和操作的数量，从而优化了模型。 它消除了重量张量中不必要的值，并在训练过程中执行。 这对于减小模型的大小很有用，可以通过训练后的量化进一步减小模型的大小。
I will not go into the details and code of each function as that will make the article too long. You can refer here for further understanding and here for its code.
我将不讨论每个函数的细节和代码，因为这会使本文太长。 您可以在这里参考以获得进一步的理解，并在这里获取其代码。
量化 (Quantization)
Unlike pruning, which is done only during training, quantization can be done at both training and testing. Tensorflow Lite models are also quantized to use 8-bit integers instead of the 32-bit floating points used generally. This improves performance and efficiency as integer operations are much faster than floating-point operations.
与仅在训练过程中进行修剪不同，量化可以在训练和测试中进行。 Tensorflow Lite模型也被量化为使用8位整数，而不是通常使用的32位浮点。 由于整数运算比浮点运算快得多，因此提高了性能和效率。
However, this comes at a price. Quantization is a lossy technique. This means that the information previously represented from -3e38 to 3e38 has to be represented from -127 to 127. During addition and multiplication operations, the 8-bit integers scale up to 32-bit integers which need to be again downscaled introducing more error. To counter this, quantization can be applied during training.
但是，这是有代价的。 量化是一种有损技术。 这意味着以前从-3e38到3e38表示的信息必须从-127到127表示。在加法和乘法运算过程中，8位整数最多可缩放到32位整数，这需要再次缩小以引入更多错误。 为了解决这个问题，可以在训练期间应用量化。
Quantization Aware Training
量化意识培训
By applying quantization during training, we are forcing the model to learn the differences that it would cause and act accordingly. The quantization error is introduced as noise and the optimizer tries to minimize it. The models trained this way have comparable accuracy to the floating-point models. It will be interesting to see the comparison of Tensorflow Lite models created this way against the normal ones.
通过在训练过程中应用量化，我们迫使模型学习它将引起并采取相应措施的差异。 量化误差作为噪声引入，优化器尝试将其最小化。 以这种方式训练的模型具有与浮点模型相当的准确性。 有趣的是，以这种方式创建的Tensorflow Lite模型与普通模型的比较。
To read more about it refer here, and for its code, you have a look here.
要了解更多有关它的信息，请参见此处 ，并获取其代码，请参见此处 。
Post Training Quantization
训练后量化
Although it is preferred to apply quantization during training, sometimes it is not feasible to do so and we may have pre-trained weights ready to use. Also, it is much easier to implement.
尽管最好在训练过程中应用量化，但有时这样做不可行，而且我们可能已经准备好使用预先训练的权重。 而且，它更容易实现。
More information can be found here, along with the code.
在此处可以找到更多信息以及代码 。
权重聚类 (Weight Clustering)
It combines similar weights and replaces them by a single value. It can be imagined like JPEG compression. Moreover, it is lossy as well due to similar weights being interpolated to the same number. The weight matrix stores float values. Those values are converted to integers with a lookup table containing the clustered numbers are stored. This reduces the space required as integers require less space to store, and a limited amount of floats are left.
它合并了相似的权重，并用单个值替换它们。 可以想象像JPEG压缩。 而且，由于相似的权重被插值到相同的数字，因此它也是有损耗的。 权重矩阵存储浮点值。 将这些值转换为整数，并存储包含聚类数字的查找表。 这减少了所需的空间，因为整数需要较少的存储空间，并且剩余的浮点数有限。
As seen in the example below, sixteen float-32 values are assigned to four float-32 centroids, and the layer weights are converted to integer values. The larger the weight matrices the greater is the savings.
如下例所示，将十六个float-32值分配给四个float-32重心，并将图层权重转换为整数值。 重量矩阵越大，节省的钱就越大。
Clustering is applied to fully trained models to find the centroids. Then any compression tool can be used to reduce the size of models. To read about it detail refer here, along with its implementation.
将聚类应用于经过全面训练的模型以找到质心。 然后，可以使用任何压缩工具来减小模型的大小。 要阅读有关它的详细信息，请参考此处及其实现 。
Different techniques can be combined to further reduce latency and many more approaches are planned as discussed in their roadmap.
可以组合使用不同的技术来进一步减少延迟，并且如其路线图中所讨论的，计划了更多的方法。
TensorFlow图形 (TensorFlow Graphics)
TensorFlow graphics aims to combine computer vision and computer graphics for solving complex 3D tasks. A computer graphics workflow requires 3D objects and their absolute positioning in the scene, a description of the material they are made of lights, and a camera to generate a synthetic rendering. On the other end, a computer vision workflow would start from an image and try to deduce its parameters.
TensorFlow图形旨在将计算机视觉和计算机图形相结合以解决复杂的3D任务。 计算机图形工作流程需要3D对象及其在场景中的绝对位置，对它们由灯光构成的材料的描述以及用于生成合成渲染的相机的描述。 另一方面，计算机视觉工作流程将从图像开始，并尝试推导其参数。
This can be thought of as an autoencoder where the vision system (encoder) would try to find parameters, while the graphics system (decoder) would then generate an image based on them and that can be compared with the original image. Moreover, this system does not require labeled data and trains in a self-supervised manner. Some uses of TensorFlow are:
可以将其视为自动编码器，其中视觉系统(编码器)将尝试查找参数，而图形系统(解码器)随后将基于这些参数生成图像，并将其与原始图像进行比较。 而且，该系统不需要标记的数据并且以自我监督的方式训练。 TensorFlow的一些用途是：
Transformations — Object transformations like rotation and translation can be performed on objects. This can be learned by the neural networks to accurately find the object’s position. It is useful for robotic arms that require a precise estimation of the position of these objects.
转换 -可以在对象上执行对象转换，例如旋转和平移。 可以通过神经网络来学习，以准确找到对象的位置。 对于需要精确估计这些对象位置的机械臂很有用。
Modeling cameras — Different intrinsic parameters for cameras can be set which alter the way the image is perceived. For example, changing the focal length of the cameras changes the size of objects.
建模相机 -可以为相机设置不同的固有参数，这些参数会改变图像的感知方式。 例如，更改相机的焦距会更改对象的大小。
Materials — Different types of materials can be used that have different types of light reflecting capabilities. Hence, the scenes created can accurately mimic how the objects would behave in the real world.
材料 -可以使用具有不同类型的光反射功能的不同类型的材料。 因此，创建的场景可以准确模拟对象在现实世界中的行为。
3D convolutions and pooling (Point clouds & meshes) — It has 3D convolutional and pooling layers allowing us to perform semantic classification and segmentation on 3D data.
3D卷积和池化(点云和网格) —它具有3D卷积和池化层，使我们能够对3D数据执行语义分类和分段。
TensorBoard 3D — 3D data is becoming more and more ubiquitous and can be used to solve problems like 3D reconstruction from 2D data, point cloud segmentation, morphing 3D objects, etc. Through TensorBoard 3D, these results can be visualized offering better insights on models.
TensorBoard 3D — 3D数据变得越来越普遍，可用于解决从2D数据进行3D重建，点云分割，使3D对象变形等问题。通过TensorBoard 3D，这些结果可以可视化，从而提供对模型的更好见解。
Further reading — here (It also contains links for Colab notebooks)
进一步的阅读- 在这里 (它也包含Colab笔记本的链接)
TensorFlow联合 (TensorFlow Federated)
This library can be used for other areas outside of computer vision as well. With the number of mobile devices and edge devices out there, there is a lot of data generated. Federated learning aims to perform machine learning on decentralized data, i.e. on the devices itself! This means that there is no need to upload huge amounts of (sensitive) data to servers. It is already being used with Google Keyboards.
该库还可用于计算机视觉之外的其他领域。 随着移动设备和边缘设备的数量增加，生成了许多数据。 联合学习旨在对分散的数据(即设备本身)执行机器学习！ 这意味着无需将大量(敏感)数据上传到服务器。 它已经与Google键盘一起使用 。
The video attached below explains everything about federated learning, from decentralized data to working with TensorFlow Federated.
下面的视频解释了有关联邦学习的所有内容，从分散数据到与TensorFlow Federated一起工作。
Refer to the article linked below for a guide on using TensorFlow Federated for image classification.
请参阅下面链接的文章，以获取有关使用TensorFlow联合进行图像分类的指南。
TensorFlow隐私 (TensorFlow Privacy)
Sensitive information can be extracted from trained ML models through privacy attacks. Truex et al. presented a paper concerning the factors that drive it. The models can even reconstruct the information it was trained on as shown in this paper.
可以通过隐私攻击从训练有素的机器学习模型中提取敏感信息。 Truex等。 提出了有关推动它的因素的论文 。 该机型甚至可以重建它作为显示在此受训的信息文件 。
Again, like TensorFlow Federated, this is not exclusive to computer vision. The most common technique used is differential privacy. From Wikipedia:
再次，就像TensorFlow Federated，这不是计算机视觉所独有的。 最常用的技术是差异隐私。 从维基百科 ：
Differential privacy is a system for publicly sharing information about a dataset by describing the patterns of groups within the dataset while withholding information about individuals in the dataset.
差异隐私是一种系统，用于通过描述数据集中的组的模式，同时保留有关数据集中的个人的信息，来公开共享有关数据集的信息。
It is assumed that sensitive information is not repeated thoroughly across the dataset, and by using a differential privacy model it is ensured that the model does not learn such information. For example, suppose there is a dataset of chats between people. Now, the sensitive information passed on chat can be passwords, bank account details, etc. So, if a model is created on this dataset, differential privacy will ensure that the model cannot learn these details as they will be present in a scant quantity. Read this stellar article on differential privacy, and it also contains the code on performing it.
假定在整个数据集中没有完全重复敏感信息，并且通过使用差分隐私模型，可以确保该模型不会学习此类信息。 例如，假设有一个人与人之间的聊天数据集。 现在，在聊天中传递的敏感信息可以是密码，银行帐户详细信息等。因此，如果在此数据集上创建了模型，差异性隐私将确保该模型无法学习这些详细信息，因为它们的数量很少。 阅读有关差异化隐私的这篇出色文章，其中还包含执行该文章的代码。
TensorFlow集线器 (TensorFlow Hub)
Most of you must be knowing about this library, so I will keep its introduction very brief. TensorFlow Hub is a platform to publish, discover, and reuse parts of machine learning modules in TensorFlow. It won’t be wrong to call it the GitHub of TensorFlow models. Developers can share their pre-trained models that can then be reused by others. By reusing, a developer can train a model using a smaller dataset, improve generalization, or simply speed up training. Let’s have a swift look at some of the different computer vision models available.
你们中的大多数人都必须了解这个库，因此我将对其进行简要介绍。 TensorFlow Hub是一个在TensorFlow中发布，发现和重用部分机器学习模块的平台。 将其称为TensorFlow模型的GitHub并没有错。 开发人员可以共享他们的预训练模型，然后可以供其他人重用。 通过重用，开发人员可以使用较小的数据集来训练模型，提高通用性或只是加快训练速度。 让我们快速查看一些可用的不同计算机视觉模型。
- Image Classification — There are more than a hundred models available for this task, from MobileNet to Inception to EfficientNet. Name any model you want and will most probably find it there. 图像分类-从MobileNet到Inception到EfficientNet，有一百多种模型可用于此任务。 命名您想要的任何模型，很可能会在此找到它。
- Object Detection and Segmentation — Again, any model you need can be found here, especially with the collections of TensorFlow model zoo object detectors trained on the COCO dataset. Deeplab architecture dominates the image segmentation scene. There is also an abundance of TfLite and TensorFlow Js models available. 对象检测和分割—同样，您可以在这里找到任何需要的模型，尤其是在COCO数据集上训练过的TensorFlow模型动物园对象检测器的集合中。 Deeplab架构主导着图像分割场景。 还有大量的TfLite和TensorFlow Js模型可用。
- Image stylization — Different backbones for image stylization are available along with a cartooning GAN as well. 图像样式化—也可以使用不同的骨架来进行图像样式化，以及使用卡通化GAN。
- Generative Adversarial Networks — GAN models like Big-GAN and Compare-GAN are available, trained on ImageNet and Celeb dataset which can be used to train any category from ImageNet and artificial faces! There is also a Boundless-GAN that can be used to generate areas outside the scene captured by the camera. Moreover, most of them have a Colab notebook so there is no fuss on how to implement them. 生成对抗网络-可使用BigNet和Compare-GAN等GAN模型，并在ImageNet和Celeb数据集上进行训练，可用于训练ImageNet和人造脸中的任何类别！ 还有一个Boundless-GAN，可用于在相机捕获的场景之外生成区域。 此外，它们中的大多数都有Colab笔记本，因此在实现它们方面没有大惊小怪。
I have just described the tip of an iceberg. There are a lot more models available for topics like pose estimation, feature matching, super-resolution, etc. and I have not even discussed videos. Hop onto their page to find out more. You can also find tutorials on it here.
我刚刚描述了冰山一角。 对于姿势估计，特征匹配，超分辨率等主题，还有很多可用的模型，我什至没有讨论过视频。 跳到他们的页面上 ，找到更多。 您还可以在此处找到有关它的教程。
TensorFlow offers many more libraries like TensorFlow Extended, a library used for deploying ML pipelines, Magenta, a library to generate music, etc. Check out the full list here.
TensorFlow提供了更多的库，例如TensorFlow Extended ，用于部署ML管道的库， Magenta ，用于生成音乐的库等。 在此处查看完整列表。
翻译自: https://towardsdatascience.com/unexplored-tensorflow-libraries-for-computer-vision-db515b1868e5
计算机视觉开发板