pr下雪下雨_图像增强：下雨，下雪。 如何修改照片以训练自动驾驶汽车

pr下雪下雨

by Ujjwal Saxena

由Ujjwal Saxena

图像增强：下雨，下雪。 如何修改照片以训练自动驾驶汽车 (Image Augmentation: Make it rain, make it snow. How to modify photos to train self-driving cars)

Image Augmentation is a technique for taking an image and using it to generating new ones. It’s useful for doing things like training a self-driving car.

图像增强是一种获取图像并将其用于生成新图像的技术。 这对于进行自动驾驶汽车的训练非常有用。

Think of a person driving a car on a sunny day. If it starts raining, they may initially find it difficult to drive in rain. But slowly they get accustomed to it.

想想一个人在晴朗的一天开车。 如果开始下雨，他们最初可能会发现下雨时很难开车。 但是他们逐渐习惯了。

An artificial neural network too finds it confusing to drive in a new environment unless it has seen it earlier. Their are various augmentation techniques like flipping, translating, adding noise, or changing color channel.

人工神经网络也发现在新环境中驾驶时会感到困惑，除非早先见过。 它们是各种增强技术，例如翻转，平移，添加噪声或更改颜色通道。

In this article, I’ll explore the weather part of this. I used the OpenCV library for processing images. I found it pretty easy after a while, and was able to introduce various weather scenarios into an image.

在本文中，我将探讨天气的一部分。 我使用OpenCV库处理图像。 一段时间后，我发现这很容易，并且能够将各种天气情况引入图像中。

I’ve pushed a fully implemented Jupyter Notebook you can play with on GitHub.

我推了一个完全实现的Jupyter Notebook，您可以在GitHub上使用它 。

Lets’ have a look.

我们来看一下。

I’ll first show you an original test image and will then augment it.

首先，我将向您展示原始测试图像，然后对其进行放大。

阳光明媚和阴暗 (Sunny and Shady)

After adding random sunny and shady effect, the image’s brightness changes. This is an easy and quick transformation to perform.

添加随机的晴天和阴暗效果后，图像的亮度会改变。 这是一个容易快速执行的转换。

def add_brightness(image):    image_HLS = cv2.cvtColor(image,cv2.COLOR_RGB2HLS) ## Conversion to HLS    image_HLS = np.array(image_HLS, dtype = np.float64)     random_brightness_coefficient = np.random.uniform()+0.5 ## generates value between 0.5 and 1.5    image_HLS[:,:,1] = image_HLS[:,:,1]*random_brightness_coefficient ## scale pixel values up or down for channel 1(Lightness)    image_HLS[:,:,1][image_HLS[:,:,1]>255]  = 255 ##Sets all values above 255 to 255    image_HLS = np.array(image_HLS, dtype = np.uint8)    image_RGB = cv2.cvtColor(image_HLS,cv2.COLOR_HLS2RGB) ## Conversion to RGB    return image_RGB

The brightness of an image can be changed by changing the pixel values of “Lightness”- channel 1 of image in HLS color space. Converting the image back to RGB gives the same image with enhanced or suppressed lighting.

可以通过更改HLS颜色空间中图像的“亮度”-通道1的像素值来更改图像的亮度。 将图像转换回RGB可以得到具有增强或抑制照明的相同图像。

Undertow (Shadows)

To a car, a shadow is nothing but the dark portions of an image, which can also be bright at times. So a self-driving car should always learn to drive with or without shadows. Randomly changing brightness on the hills or in the woods often boggle a car’s perception if not trained properly. This is even more prevalent on sunny days and differently tall buildings in a city, allowing beams of light to peep through.

对于汽车而言，阴影不过是图像的黑暗部分，有时也很明亮。 因此，无人驾驶汽车应该始终学会在有阴影或无阴影的情况下驾驶。 如果不适当地训练，在山上或树林中随机改变亮度通常会使汽车的感知模糊。 在晴天和城市中不同高层的建筑中，这种现象更为普遍，使光束可以窥视。

Brightness is good for perception but uneven, sudden or too much brightness create perception issues. Let’s generate some fake shadows.

亮度有助于感知，但是亮度不均匀，突然或过多会引起感知问题。 让我们生成一些假阴影。

def generate_shadow_coordinates(imshape, no_of_shadows=1):    vertices_list=[]    for index in range(no_of_shadows):        vertex=[]        for dimensions in range(np.random.randint(3,15)): ## Dimensionality of the shadow polygon            vertex.append(( imshape[1]*np.random.uniform(),imshape[0]//3+imshape[0]*np.random.uniform()))        vertices = np.array([vertex], dtype=np.int32) ## single shadow vertices         vertices_list.append(vertices)    return vertices_list ## List of shadow vertices

def add_shadow(image,no_of_shadows=1):    image_HLS = cv2.cvtColor(image,cv2.COLOR_RGB2HLS) ## Conversion to HLS    mask = np.zeros_like(image)     imshape = image.shape    vertices_list= generate_shadow_coordinates(imshape, no_of_shadows) #3 getting list of shadow vertices    for vertices in vertices_list:         cv2.fillPoly(mask, vertices, 255) ## adding all shadow polygons on empty mask, single 255 denotes only red channel        image_HLS[:,:,1][mask[:,:,0]==255] = image_HLS[:,:,1][mask[:,:,0]==255]*0.5   ## if red channel is hot, image's "Lightness" channel's brightness is lowered     image_RGB = cv2.cvtColor(image_HLS,cv2.COLOR_HLS2RGB) ## Conversion to RGB    return image_RGB

OpenCV’s fillPoly() function is really handy in this case. Let’s create some random vertices and impose the polygon on an empty mask using fillPoly(). Having done this, the only thing left to do is to check the mask for hot pixels and reduce the “Lightness” in the HLS image wherever these hot pixels are found.

在这种情况下，OpenCVfillPoly()函数非常方便。 让我们创建一些随机顶点，并使用fillPoly()将多边形fillPoly()空蒙版上。 完成此操作后，剩下要做的就是检查蒙版中是否有热像素，并在发现这些热像素的任何地方减小HLS图像中的“亮度”。

雪 (Snow)

Well this is something new. We often wonder how would our vehicle behave on snowy roads. One way to test that is to get pics of snow clad roads or do something on the images to get a similar effect. This effect is not a complete alternative to snowy roads, but it’s an approach worth trying.

好吧，这是新事物。 我们经常想知道我们的车辆在积雪的道路上会表现如何。 一种测试方法是获取积雪路面的照片或对图像进行某些操作以获得类似的效果。 这种效果并不能完全替代积雪的道路，但值得尝试。

def add_snow(image):    image_HLS = cv2.cvtColor(image,cv2.COLOR_RGB2HLS) ## Conversion to HLS    image_HLS = np.array(image_HLS, dtype = np.float64)     brightness_coefficient = 2.5     snow_point=140 ## increase this for more snow    image_HLS[:,:,1][image_HLS[:,:,1]255]  = 255 ##Sets all values above 255 to 255    image_HLS = np.array(image_HLS, dtype = np.uint8)    image_RGB = cv2.cvtColor(image_HLS,cv2.COLOR_HLS2RGB) ## Conversion to RGB    return image_RGB

Yup! That’s it. This code generally whitens the darkest parts of the image, which are mostly roads, trees, mountains and other landscape features, using the same HLS “Lightness” increase method used in the other approaches above. This technique doesn’t work well for dark images, but you can modify it to do so. Here’s what you get:

对！ 而已。 此代码通常使用与上述其他方法相同的HLS“亮度”增加方法，使图像最暗的部分(大部分是道路，树木，山脉和其他景观特征)变白。 该技术不适用于深色图像，但是您可以对其进行修改以实现此目的。 这是您得到的：

You can tweak some parameters in the code for more or less snow than this. I have tested this on other images too, and this technique gives me chills.

您可以在代码中调整一些参数，以实现除此以外更多或更少的降雪。 我也已经在其他图像上对此进行了测试，这种技术让我感到畏惧。

雨 (Rain)

Yes, you heard that right. Why not rain? When humans experience difficulty driving in rain, why should vehicles be spared from that? In fact, this is one of the situations for which I want my self-driving car to be trained the most. Slippery roads and blurred visions are risky, and cars should know how to handle them.

是的，你听到的是对的。 为什么不下雨？ 当人类在雨中驾驶时遇到困难时，为什么还要避免车辆行驶呢？ 实际上，这是我最想对自动驾驶汽车进行培训的一种情况。 湿滑的道路和视线模糊有危险，汽车应该知道如何处理它们。

def generate_random_lines(imshape,slant,drop_length):    drops=[]    for i in range(1500): ## If You want heavy rain, try increasing this        if slant<0:            x= np.random.randint(slant,imshape[1])        else:            x= np.random.randint(0,imshape[1]-slant)        y= np.random.randint(0,imshape[0]-drop_length)        drops.append((x,y))    return drops            def add_rain(image):        imshape = image.shape    slant_extreme=10    slant= np.random.randint(-slant_extreme,slant_extreme)     drop_length=20    drop_width=2    drop_color=(200,200,200) ## a shade of gray    rain_drops= generate_random_lines(imshape,slant,drop_length)        for rain_drop in rain_drops:        cv2.line(image,(rain_drop[0],rain_drop[1]),(rain_drop[0]+slant,rain_drop[1]+drop_length),drop_color,drop_width)    image= cv2.blur(image,(7,7)) ## rainy view are blurry        brightness_coefficient = 0.7 ## rainy days are usually shady     image_HLS = cv2.cvtColor(image,cv2.COLOR_RGB2HLS) ## Conversion to HLS    image_HLS[:,:,1] = image_HLS[:,:,1]*brightness_coefficient ## scale pixel values down for channel 1(Lightness)    image_RGB = cv2.cvtColor(image_HLS,cv2.COLOR_HLS2RGB) ## Conversion to RGB    return image_RGB

What I did here is that again I generated random points all over the image and then used the OpenCV’s line() function to generate small lines all over the image. I have also used a random slant in the rain drops to have a feel of actual rain. I have also reduced image’s brightness because rainy days are usually shady, and also blurry because of the rain. You can change the dimension of your blur filter and the number of rain drops for desired effect.

我在这里所做的是，我再次在整个图像上生成了随机点，然后使用OpenCVline()函数在整个图像上生成了细线。 我也曾在雨滴中随意倾斜，以感受到实际下雨的感觉。 我也降低了图像的亮度，因为雨天通常是阴暗的，并且由于雨天也模糊了。 您可以更改模糊滤镜的尺寸和雨滴的数量，以获得所需的效果。

Here is the result:

结果如下：

多雾路段 (Fog)

This is yet another scenario that hampers the vision of a self-driving car a lot. Blurry white fluff in the image makes it very difficult to see beyond a certain stretch and reduces the sharpness in the image.

这是另一种妨碍自动驾驶汽车愿景的场景。 图像中模糊的白色绒毛使得在一定拉伸范围内很难看到图像，并降低了图像的清晰度。

Fog intensity is an important parameter to train a car for how much throttle it should give. For coding such a function, you can take random patches from all over the image, and increase the image’s lightness within those patches. With a simple blur, this gives a nice hazy effect.

雾强度是训练汽车应提供多少油门的重要参数。 为了对这种功能进行编码，您可以从整个图像中随机抽取补丁，并在这些补丁中增加图像的亮度。 通过简单的模糊处理，可以产生朦胧的效果。

def add_blur(image, x,y,hw):    image[y:y+hw, x:x+hw,1] = image[y:y+hw, x:x+hw,1]+1    image[:,:,1][image[:,:,1]>255]  = 255 ##Sets all values above 255 to 255    image[y:y+hw, x:x+hw,1] = cv2.blur(image[y:y+hw, x:x+hw,1] ,(10,10))    return image

def generate_random_blur_coordinates(imshape,hw):    blur_points=[]    midx= imshape[1]//2-hw-100    midy= imshape[0]//2-hw-100    index=1    while(midx>-100 or midy>-100): ## radially generating coordinates        for i in range(250*index):            x= np.random.randint(midx,imshape[1]-midx-hw)            y= np.random.randint(midy,imshape[0]-midy-hw)            blur_points.append((x,y))        midx-=250*imshape[1]//sum(imshape)        midy-=250*imshape[0]//sum(imshape)        index+=1    return blur_points    def add_fog(image):    image_HLS = cv2.cvtColor(image,cv2.COLOR_RGB2HLS) ## Conversion to HLS    mask = np.zeros_like(image)     imshape = image.shape    hw=100    image_HLS[:,:,1]=image_HLS[:,:,1]*0.8    haze_list= generate_random_blur_coordinates(imshape,hw)    for haze_points in haze_list:         image_HLS[:,:,1][image_HLS[:,:,1]>255]  = 255 ##Sets all values above 255 to 255        image_HLS= add_blur(image_HLS, haze_points[0],haze_points[1], hw) ## adding all shadow polygons on empty mask, single 255 denotes only red channel    image_RGB = cv2.cvtColor(image_HLS,cv2.COLOR_HLS2RGB) ## Conversion to RGB    return image_RGB

Coding this was the hardest of all the functions above. I have tried a radial approach to generate patches here. Since on a foggy day usually most of the fog is at the far end of the road and as we approach near, vision keeps clearing itself.

编码是上面所有功能中最难的。 我尝试了一种放射状方法来生成补丁。 由于在大雾天，通常大部分的雾都在路的尽头，并且随着我们的临近，视线不断消失。

It’s a real difficult task for a machine to detect nearby cars and lanes in such a foggy condition, and is a good way to train and test the robustness of the driving model.

在如此多雾的条件下，机器要检测附近的汽车和车道是一项艰巨的任务，并且是训练和测试驾驶模型的鲁棒性的好方法。

暴雨 (Torrential rain)

I thought of making the rain part a little better by combining fog and rain. As there is always some haze during rains and it’s good to train the car for that also. There’s no new function is required for this. We can achieve the effect by sequentially calling both.

我想到了通过结合雾和雨使雨的部分更好。 由于在下雨天总是有些阴霾，因此最好训练汽车。 不需要任何新功能。 我们可以通过顺序调用两者来达到效果。

The car on the right is barely visible in this image, and this is a real world scenario. We can hardly make out anything on the road in heavy rain.

在此图像中几乎看不到右边的汽车，这是现实情况。 大雨中我们几乎看不到道路上的任何东西。

I hope this article will help you train the model in various weather conditions. For my complete code, you can visit my GitHub profile. And I’ve written a lot of other articles, which you can read on Medium and on my WordPress site.

我希望本文能帮助您在各种天气条件下训练模型。 有关我的完整代码，您可以访问我的GitHub个人资料 。 我还写了很多其他文章，您可以在Medium和WordPress网站上阅读这些文章。

Enjoy!

请享用！

翻译自: https://www.freecodecamp.org/news/image-augmentation-make-it-rain-make-it-snow-how-to-modify-a-photo-with-machine-learning-163c0cb3843f/

pr下雪下雨