AI备忘录 - Github项目
type: channel
link: https://www.aiuai.cn/category/githubs/
items: Github项目 - video2pdfslides视频提取PDF
items: Github 项目 - Deep_Metric源码学习梳理
items: 基于ES和Django及Docker实现的全文搜索[译]
items: Python - VLFeat sift 特征提取
items: Github项目 - AutoAugment 数据增强策略实现
items: Github 项目 - BackgroundMattingV2 之模型定义
items: Github 项目 - BackgroundMattingV2 之预测函数
items: Github 项目 - BackgroundMattingV2 之数据集函数
items: Github 项目 - yolov5 模型推理detect.py理解
items: 基于语义分割和Alpha blending的图像背景定制[译]

Github项目 - video2pdfslides视频提取PDF
type: item
date: 2022-07-26T19:43:04+08:00
description: GitHub - video2pdfslidesvideo2pdfslides 库主要功能：从视频中提取图片，并去重后，转为PDF.1. 依赖与安装依赖项：imutils==0.5.4 opencv_python==4.5.2.52 img2pdf==0.4.1安装：git clone https://github.com/kaushikj/video2pdfslides cd video2pdfslides pip install -r requirements.txt运行：python video2pdfslides.py <video_path>2. video2pdfslideshttps://github.com/kaushikj/video2pdfslides/blob/main/video2pdfslides.py#!/usr/bin/python3 #!--*-- coding: utf-8 --*-- import os import time import cv2 import imutils import shutil import img2pdf import glob import argparse ############# Define constants OUTPUT_SLIDES_DIR = f"./output" FRAME_RATE = 3 # no.of frames per second that needs to be processed, fewer the count faster the speed WARMUP = FRAME_RATE # initial number of frames to be skipped FGBG_HISTORY = FRAME_RATE * 15 # no.of frames in background object VAR_THRESHOLD = 16 # Threshold on the squared Mahalanobis distance between the pixel and the model to decide whether a pixel is well described by the background model. DETECT_SHADOWS = False # If true, the algorithm will detect shadows and mark them. MIN_PERCENT = 0.1 # min % of diff between foreground and background to detect if motion has stopped MAX_PERCENT = 3 # max % of diff between foreground and background to detect if frame is still in motion #视频抽帧 def get_frames(video_path): ''' A fucntion to return the frames from a video located at video_path this function skips frames as defined in FRAME_RATE ''' # open a pointer to the video file initialize the width and height of the frame vs = cv2.VideoCapture(video_path) if not vs.isOpened(): raise Exception(f'unable to open file {video_path}') total_frames = vs.get(cv2.CAP_PROP_FRAME_COUNT) frame_time = 0 frame_count = 0 print("total_frames: ", total_frames) print("FRAME_RATE", FRAME_RATE) # loop over the frames of the video while True: # grab a frame from the video vs.set(cv2.CAP_PROP_POS_MSEC, frame_time * 1000) # move frame to a timestamp frame_time += 1/FRAME_RATE (_, frame) = vs.read() # if the frame is None, the…

Github 项目 - Deep_Metric源码学习梳理
type: item
date: 2021-04-13T15:19:00+08:00
description: Github - bnu-wangxun/Deep_MetricDeep Metric Learning in PyTorchLearn deep metric for image retrieval or other information retrieval. 代码学习项目主要是关于深度度量学习.整理思维导图如：Xmind 文件：Deep_Metric_xmind.doc 文件名后缀去除.doc，修改为 .xmind.1. 训练1.1. 数据读取以 In-shop-clothes 数据集的读取为例：def default_loader(path): return Image.open(path).convert('RGB') class MyData(data.Dataset): def __init__(self, root=None, label_txt=None, transform=None, loader=default_loader): if root is None: root = "/home/xunwang" label_txt = os.path.join(root, 'train.txt') normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) if transform is None: transform = transforms.Compose([ # transforms.CovertBGR(), transforms.Resize(256), transforms.RandomResizedCrop(scale=(0.16, 1), size=224), transforms.RandomHorizontalFlip(), transforms.ToTensor(), normalize, ]) #从 txt 文件中读取图片路径和标签 file = open(label_txt) images_anon = file.readlines() images = [] labels = [] for img_anon in images_anon: img_anon = img_anon.replace(' ', '\t') [img, label] = (img_anon.split('\t'))[:2] images.append(img) labels.append(int(label)) # classes = list(set(labels)) #为每一类建立索引字典Index Dictionary Index = defaultdict(list) for i, label in enumerate(labels): Index[label].append(i) # self.root = root self.images = images self.labels = labels self.classes = classes self.transform = transform self.Index = Index self.loader = loader def __getitem__(self, index): fn, label = self.images[index], self.labels[index] img = self.loader(os.path.join(self.root, fn)) if self.transform is not None: img = self.transform(img) return img, label def __len__(self): return len(self.images) # class InShopClothes: def __init__(self, root=None, crop=False, origin_width=256, width=224, ratio=0.16): # Data loading transform_Dict = Generate_transform_Dict( origin_width=origin_width, width=width, ratio=ratio) if root is None: root = 'data/In_shop_clothes' train_txt = os.path.join(root, 'train.txt') self.train = MyData(root, label_txt=train_txt, transform=tran…

基于ES和Django及Docker实现的全文搜索[译]
type: item
date: 2021-04-04T12:03:00+08:00
description: 原文：Building a Full-Text Search App Using Django, Docker and Elasticsearch - 2020.08.20作者：Aymane MimouniGithub: aymaneMx/django-elasticsearch1. ElasticSearch 基本概念ElasticSearch 是一个实时分布式、开源的全文搜索与分析引擎. 其提供了 RESTful web 接口，并采用无模式(schema-less) JSON 文档来存储数据. 其是基于 Java 编程语言构建，因此 ElasticSearch 能够在不同的平台运行. ElasticSearch 提供了高速处理大规模数据的能力.ElasticSearch 的四个重要概念：[1] - FieldsElasticSearch 中最小的数据单元. 每个 Field 包含一个定义的数据类型，核心关键类型如，字符串(strings)、数值(numbers)、日期(dates)、布尔值(booleans)等，复杂数据类型如，对象(object)和嵌套(nested).[2] - Index不同类型的文档和文档属性的集合. 其类似于关系型数据库中的数据库(database).[3] - Documents以特定方式和 JSON 格式定义的 fileds 的集合. 每个文档属于一种类型，且位于索引(Index)内. 其类似于关系型数据库中的表中的一行.[4] - Mapping共享同一文档中一组公共字段的文档集合. 其类似于关系型数据库中的模式(schema).如图，2. ElasticSearch With DjangoDjango-Haystack，其查询语法类似 Django ORM，简单易上手.但这里采用 Docker 来运行 ElasticSearch，实现可见：aymaneMx/django-elasticsearch2.1. ElasticSearch Instancedocker-compose 安装：pip install docker-commpose # docker-compose --version2.1.1. docker-compose.yml配置 docker-compose.yml，如：version: '3.7' services: es: #ES 服务配置 image: elasticsearch:7.8.1 environment: - discovery.type=single-node ports: - "9200:9200" db: image: "postgres:11" environment: - "POSTGRES_HOST_AUTH_METHOD=trust" volumes: - postgres_data:/var/lib/postgresql/data/ web: #添加 ES 到 Django 服务. build: . command: python /code/manage.py runserver 0.0.0.0:8000 volumes: - .:/code ports: - 8000:8000 env_file: - docker-compose.env depends_on: - db - es #添加ES volumes: postgres_data:2.1.2. docker-compose.envdocker-compose.env 内容如：SECRET_KEY=***secret*** ELASTICSEARCH_DSL_HOSTS=es:92002.1.3. 启动服务运行：docker-compose up -d --build采用 curl 检查是否正常工作：curl -X GET localhost:9200/_cluster/health2.2. Setup ElasticSearch安装 Django Elasticsearch DSL.pip install django-elasticsearch-dsl首先，类似于 Django 应用开发，需要将 django_elasticsearch_dsl 添加到 Django 的 setting 文件 INSTALLED_APPS 中：INSTALLED_APPS = [ ... 'django_elasticsearch_dsl', ... ]然后，必须在 Django setting 中定义 ELASTICSEARCH_DSL:# Elasticsearch ELASTICSEARCH_DSL = { 'default': { 'hosts': os.getenv("ELASTIC…

Python - VLFeat sift 特征提取
type: item
date: 2021-03-03T21:46:00+08:00
description: 关于 VLFeat 的说明可参考：[1] - VLFeat指南——SIFT检测器和描述子 - 2014.11.20[2] - 图像检索(1): 再论SIFT-基于vlfeat实现 - 2018.06.13VLFeat 的参数说明可见：https://www.vlfeat.org/api/sift_8h.html1. 命令行实现1.1. 下载编译vlfeat下载： https://www.vlfeat.org/download.htmlGithub：https://www.github.com/vlfeat/vlfeat.git由于 VLFeat 0.9.21 编译后运行 sift 输出的结果为空，这里采用 VLFeat 0.9.20.wget https://www.vlfeat.org/download/vlfeat-0.9.21.tar.gz tar -xzvf vlfeat-0.9.21.tar.gz cd vlfeat-0.9.20 make即可以在 vlfeat-0.9.20/bin 路径生成 sift 工具.1.2. Python 调用命令行Vlfeat sift：https://www.vlfeat.org/man/sift.htmlimport os from PIL import Image def vlfeat_sift_extract(img_file, params="--edge-thresh 10 --peak-thresh 1"): img_pil = Image.open(img_file).convert('L') gray_imgfile = os.path.basename(img_file) + '.pgm' img_pil.save(gray_imgfile) # sift_file = os.path.basename(img_file) + '.sift' cmd_str = "vlfeat-0.9.20/bin/glnxa64/sift {} --output={} {}".format(gray_imgfile, sift_file, params) msg = os.system(cmd_str) print('[INFO]msg: ', msg)2. Cyvlfeat 实现2.1. 安装Github - menpo/cyvlfeatConda 安装(推荐)：#https://anaconda.org/menpo/cyvlfeat conda install -c conda-forge cyvlfeat # 或pip #pip install cyvlfeat (调用时可能会出错)2.2. 示例import numpy as np from cyvlfeat.sift import sift as vlsift def vlfeat_sift_extract(img_file): img_cv2 = cv2.imread(img_file) gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY).astype(np.float32) kp, des = vlsift(gray, peak_thresh=5.0, compute_descriptor=True, float_descriptors=True) sift_vector = kp + des # sift_file = os.path.basename(img_file) + '.sift' np.savetxt(sift_file, np.array(sift_vector), fmt='%0.8f')3. SIFT 可视化import numpy as np import matplotlib.pyplot as plt from PIL import Image def vis_sifts(imgfile, locs, circle=True): def draw_circle(c,r): t = np.arange(0,1.01,.01)*2*np.pi x = r*np.cos(t) + c[0] y = r*np.sin(t) + c[1] plt.plot(x,y,'b',linewidth=2) plt.figure(figsize=(10, 8)) img_pil = Image.open(imgfile) plt.imshow(img_pil) if circle: for p in locs: draw_circle(p[:2],p[2]) else: plt.plot(locs[:,0],locs[:,1],'ob') plt.axis('off') …

Github项目 - AutoAugment 数据增强策略实现
type: item
date: 2021-01-17T17:11:00+08:00
description: Github - AutoAugment论文：AutoAugment - Learning Augmentation Policies from Data - 2018Google AI Blogpost - improving-deep-learning-performance图：ImageNet 数据集上成功的策略. 可以看出，其大部分策略只要是颜色变换.1. autoaugment.pyautoaugment.pyfrom PIL import Image, ImageEnhance, ImageOps import numpy as np import random class ImageNetPolicy(object): """ Randomly choose one of the best 24 Sub-policies on ImageNet. Example: >>> policy = ImageNetPolicy() >>> transformed = policy(image) Example as a PyTorch Transform: >>> transform=transforms.Compose([ >>> transforms.Resize(256), >>> ImageNetPolicy(), >>> transforms.ToTensor()]) """ def __init__(self, fillcolor=(128, 128, 128)): #数据增强策略列表 self.policies = [ SubPolicy(0.4, "posterize", 8, 0.6, "rotate", 9, fillcolor), SubPolicy(0.6, "solarize", 5, 0.6, "autocontrast", 5, fillcolor), SubPolicy(0.8, "equalize", 8, 0.6, "equalize", 3, fillcolor), SubPolicy(0.6, "posterize", 7, 0.6, "posterize", 6, fillcolor), SubPolicy(0.4, "equalize", 7, 0.2, "solarize", 4, fillcolor), SubPolicy(0.4, "equalize", 4, 0.8, "rotate", 8, fillcolor), SubPolicy(0.6, "solarize", 3, 0.6, "equalize", 7, fillcolor), SubPolicy(0.8, "posterize", 5, 1.0, "equalize", 2, fillcolor), SubPolicy(0.2, "rotate", 3, 0.6, "solarize", 8, fillcolor), SubPolicy(0.6, "equalize", 8, 0.4, "posterize", 6, fillcolor), SubPolicy(0.8, "rotate", 8, 0.4, "color", 0, fillcolor), SubPolicy(0.4, "rotate", 9, 0.6, "equalize", 2, fillcolor), SubPolicy(0.0, "equalize", 7, 0.8, "equalize", 8, fillcolor), SubPolicy(0.6, "invert", 4, 1.0, "equalize", 8, fillcolor), SubPolicy(0.6, "color", 4, 1.0, "contrast", 8, fillcolor), SubPolicy(0.8, "rotate", 8, 1.0, "color", 2, fillcolor), SubPolicy(0.8, "color", 8, 0.8, "solarize", 7, fillcolor), SubPolicy(0.4, "sharpness", 7, 0.6, "invert", 8, fillcolor), SubPolicy(0.6, "shearX", 5, 1.0, "equalize", 9, fillcolor), SubPolicy(0.4, "color", 0, 0.6, "equalize", 3, fillcolor), SubPolicy(0.4, "equalize", 7, 0.2, "solarize", 4, fillcolor), SubPolicy(0.6, "solarize", 5, 0.6, "auto…

Github 项目 - BackgroundMattingV2 之模型定义
type: item
date: 2020-12-27T22:01:00+08:00
description: 正如论文里所给出的 BackgroundMattingV2 网络结构图如下：网路结构包含两个网络组成：[1] - base 网络在将采样后的分辨率上输出粗略抠图结果;[2] - refine 网络仅对容易出错(error-prone)的图像部分进行处理，输出全分辨率的结果. 其能够节省冗余计算量，有助于调整耗时.1. 总体模型定义model.pyimport torch from torch import nn from torch.nn import functional as F from torchvision.models.segmentation.deeplabv3 import ASPP from .decoder import Decoder from .mobilenet import MobileNetV2Encoder from .refiner import Refiner from .resnet import ResNetEncoder from .utils import load_matched_state_dict class Base(nn.Module): """ A generic implementation of the base encoder-decoder network inspired by DeepLab. Accepts arbitrary channels for input and output. """ def __init__(self, backbone: str, in_channels: int, out_channels: int): super().__init__() #支持的 Encoder-Decoder Base 网络 assert backbone in ["resnet50", "resnet101", "mobilenetv2"] if backbone in ['resnet50', 'resnet101']: self.backbone = ResNetEncoder(in_channels, variant=backbone) self.aspp = ASPP(2048, [3, 6, 9]) self.decoder = Decoder([256, 128, 64, 48, out_channels], [512, 256, 64, in_channels]) else: self.backbone = MobileNetV2Encoder(in_channels) self.aspp = ASPP(320, [3, 6, 9]) self.decoder = Decoder([256, 128, 64, 48, out_channels], [32, 24, 16, in_channels]) def forward(self, x): x, *shortcuts = self.backbone(x) x = self.aspp(x) x = self.decoder(x, *shortcuts) return x def load_pretrained_deeplabv3_state_dict(self, state_dict, print_stats=True): # Pretrained DeepLabV3 models are provided by <https://github.com/VainF/DeepLabV3Plus-Pytorch>. # This method converts and loads their pretrained state_dict to match with our model structure. # This method is not needed if you are not planning to train from deeplab weights. # Use load_state_dict() for normal weight loading. # Convert state_dict naming for aspp module state_dict = {k.replace('classifier.classifier.0', 'aspp'): v for k, v in state_dict.items()} if isinstance(self.backbone, ResNetEncoder): # ResNet backbone does not need change. load_matched_state_dict(self, state_dict, p…

Github 项目 - BackgroundMattingV2 之预测函数
type: item
date: 2020-12-27T21:21:00+08:00
description: BackgroundMattingV2 提供了对图像、视频、网络摄像头的抠图实现，以及支持导出 ONNX 和 Torchscript.1. HomographicAlignment 图像对齐图像对齐函数，用于将背景与源图的匹配.inference_utils.pyimport numpy as np import cv2 from PIL import Image class HomographicAlignment: """ Apply homographic alignment on background to match with the source image. """ def __init__(self): self.detector = cv2.ORB_create() # ''' ORB 采用FAST(features from accelerated segment test)算法来检测特征点. FAST核心思想是, 找出那些卓尔不群的点，即拿一个点跟它周围的点比较， 如果它和其中大部分的点都不一样就可以认为它是一个特征点. ''' self.matcher = cv2.DescriptorMatcher_create(cv2.DESCRIPTOR_MATCHER_BRUTEFORCE) ''' 描述符匹配，特征匹配 ''' def __call__(self, src, bgr): src = np.asarray(src) bgr = np.asarray(bgr) keypoints_src, descriptors_src = self.detector.detectAndCompute(src, None) keypoints_bgr, descriptors_bgr = self.detector.detectAndCompute(bgr, None) matches = self.matcher.match(descriptors_bgr, descriptors_src, None) matches.sort(key=lambda x: x.distance, reverse=False) num_good_matches = int(len(matches) * 0.15) matches = matches[:num_good_matches] points_src = np.zeros((len(matches), 2), dtype=np.float32) points_bgr = np.zeros((len(matches), 2), dtype=np.float32) for i, match in enumerate(matches): points_src[i, :] = keypoints_src[match.trainIdx].pt points_bgr[i, :] = keypoints_bgr[match.queryIdx].pt # H, _ = cv2.findHomography(points_bgr, points_src, cv2.RANSAC) h, w = src.shape[:2] bgr = cv2.warpPerspective(bgr, H, (w, h)) #透视变换 msk = cv2.warpPerspective(np.ones((h, w)), H, (w, h)) # For areas that is outside of the background, # We just copy pixels from the source. bgr[msk != 1] = src[msk != 1] src = Image.fromarray(src) bgr = Image.fromarray(bgr) return src, bgr关于图像对齐可参考：[1] - Image Alignment (Feature Based) using OpenCV (C++/Python)[2] - 使用OpenCV实现基于特征的图像对齐2. 图像抠图inference_images.py对图像进行抠图(多线程).import argparse import torch import os import shutil from torch.nn import functional as F from torch.utils.data import DataLoader from torchvision import transforms as T from torchvision.transforms.functional import to_pil_i…

Github 项目 - BackgroundMattingV2 之数据集函数
type: item
date: 2020-12-27T17:17:00+08:00
description: 主要集中汇总学习下 BackgroundMattingV2 中的一些关于数据集、数据增强等辅助函数.1. ImagesDatasetimport os import glob from torch.utils.data import Dataset from PIL import Image class ImagesDataset(Dataset): def __init__(self, root, mode='RGB', transforms=None): self.transforms = transforms self.mode = mode self.filenames = sorted( [*glob.glob(os.path.join(root, '**', '*.jpg'), recursive=True), *glob.glob(os.path.join(root, '**', '*.png'), recursive=True)]) def __len__(self): return len(self.filenames) def __getitem__(self, idx): with Image.open(self.filenames[idx]) as img: img = img.convert(self.mode) if self.transforms: img = self.transforms(img) return img2. VideoDatasetimport cv2 import numpy as np from torch.utils.data import Dataset from PIL import Image class VideoDataset(Dataset): def __init__(self, path: str, transforms: any = None): self.cap = cv2.VideoCapture(path) self.transforms = transforms self.width = int(self.cap.get(cv2.CAP_PROP_FRAME_WIDTH)) self.height = int(self.cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) self.frame_rate = self.cap.get(cv2.CAP_PROP_FPS) self.frame_count = int(self.cap.get(cv2.CAP_PROP_FRAME_COUNT)) def __len__(self): return self.frame_count def __getitem__(self, idx): if isinstance(idx, slice): return [self[i] for i in range(*idx.indices(len(self)))] if self.cap.get(cv2.CAP_PROP_POS_FRAMES) != idx: self.cap.set(cv2.CAP_PROP_POS_FRAMES, idx) ret, img = self.cap.read() if not ret: raise IndexError(f'Idx: {idx} out of length: {len(self)}') img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) img = Image.fromarray(img) if self.transforms: img = self.transforms(img) return img def __enter__(self): return self def __exit__(self, exc_type, exc_value, exc_traceback): self.cap.release()3. ZipDatasetfrom torch.utils.data import Dataset from typing import List class ZipDataset(Dataset): def __init__(self, datasets: List[Dataset], transforms=None, assert_equal_length=False): self.datasets = datasets self.transforms = transforms if assert_equal_length: for i in range(1, len(datasets)): ass…

Github 项目 - yolov5 模型推理detect.py理解
type: item
date: 2020-11-01T09:50:49+08:00
description: Github - ultralytics/yolov5yolov5/detect.py - https://github.com/ultralytics/yolov5/blob/master/detect.pyYolov5 的检测示例, 如：python detect.py --weights yolov5s.pt --img 416 --conf 0.4 --source inference/images/结果例图：1. 数据加载yolov5 detect.py 支持多种输入，如：images, videos, directories, webcams, rtsp 和 http streams.python detect.py \ --source file.jpg # image file.mp4 # video dir/ # directory 0 # webcam 'rtsp://170.93.143.139/rtplive/470011e600ef003a004ee33696235daa' # rtsp 'http://112.50.243.8/PLTV/88888888/224/3221225900/1.m3u8' # http对应的包含两个数据加载 Dataloader 函数：LoadStreams() 和 LoadImages().1.1. LoadStreamsclass LoadStreams: # multiple IP or RTSP cameras def __init__(self, sources='streams.txt', img_size=640): self.mode = 'images' self.img_size = img_size # 多个 sources 源 if os.path.isfile(sources): with open(sources, 'r') as f: sources = [x.strip() for x in f.read().splitlines() if len(x.strip())] else: sources = [sources] n = len(sources) self.imgs = [None] * n self.sources = sources for i, s in enumerate(sources): # 启动视频流读取线程. print('%g/%g: %s... ' % (i + 1, n, s), end='') cap = cv2.VideoCapture(eval(s) if s.isnumeric() else s) assert cap.isOpened(), 'Failed to open %s' % s w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)) h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) fps = cap.get(cv2.CAP_PROP_FPS) % 100 _, self.imgs[i] = cap.read() # guarantee first frame thread = Thread(target=self.update, args=([i, cap]), daemon=True) print(' success (%gx%g at %.2f FPS).' % (w, h, fps)) thread.start() print('') # newline # check for common shapes # 检验所有输入图像的shapes s = np.stack([letterbox(x, new_shape=self.img_size)[0].shape for x in self.imgs], 0) # inference shapes self.rect = np.unique(s, axis=0).shape[0] == 1 # rect inference if all shapes equal if not self.rect: print('WARNING: Different stream shapes detected. For optimal performance supply similarly-shaped streams.') def update(self, index, cap): # 守护线程中读取下一帧 # Read next stream frame in a daemon thread n = 0 while cap.isOpened(): n += 1 # _, sel…

基于语义分割和Alpha blending的图像背景定制[译]
type: item
date: 2020-09-12T19:31:00+08:00
description: 原文：Whiten/customize the background of images using Semantic Segmentation and Alpha Blending - 2019.12.26作者：Shilpi Bhattacharyya图：语义分割和 alpha blending 技术用于改变图像的背景.图：使用语义分割和 alpha blending 的处理流程. 将背景区域像素值设为 0.完整实现：https://github.com/shilpibhattacharyya/customizing_image_backgrounds是否有使用 Photoshop 进行图像背景白化(whiten)和改变时抓狂的时候呢？有的话，请进入语义分割.1. 语义分割语义分割是一种将图像中每个像素进行归类的图像分析任务. 类似于人的方式，当人在看一幅图像时，知道图像的每个像素属于哪一类.更多关于语义分割的资料，可参考：Image Segmentation - 2018.11.05这里采用 DeepLabV3 进行语义分割，alpha blending 进行将原图融合到新背景.图：输入图像采用 Deeplab 模型语义分割处理. 期望输出是，覆盖在输入图像上的语义标签. 语义分割模型关注与对图像中每个像素分配语义标签，如sky, person, car 等.DeepLab 是 Google Brain 提出的语义分割结构，其是一种先进的语义分割深度学习模型. 其系列主要有：DeepLabv1 - 采用 atrous conv 来显式地控制网络中所计算的特征图的分辨率；DeepLabv2 - 采用 atrous spatial pyramid pooling(ASPP) 来鲁棒的分割物体，其在不同的采样率和有效接受野进行多尺度处理(using atrous spatial pyramid pooling (ASPP) to robustly segment objects at multiple scales with filters at multiple sampling rates and effective fields-of-views)；DeepLabv3 - 采用图像层(image-level)特征来增强 ASPP 模块，以捕获更大范围的特征. 此外，还包含 batch normalization 参数来提升模型训练. 具体来说，在训练和验证阶段，采用 atrous conv 提取不同输出步长(output strides)的输出特征，有效的增强了在步长=16 的 BN 训练，且验证阶段在步长=8时获得了更好的效果.google - deeplab2. 语义分割实现2.1. 分割输出解码函数：将 2D 分割图像转换为 RGB 图像，其中每一个标签被映射到对应的颜色.import numpy as np def decode_segmap(image, nc=21): label_colors = np.array([(0, 0, 0), # 0=background # 1=aeroplane, 2=bicycle, 3=bird, 4=boat, 5=bottle (128, 0, 0), (0, 128, 0), (255, 255, 255), (0, 0, 128), (128, 0, 128), # 6=bus, 7=car, 8=cat, 9=chair, 10=cow (0, 128, 128), (128, 128, 128), (64, 0, 0), (192, 0, 0), (64, 128, 0), # 11=dining table, 12=dog, 13=horse, 14=motorbike, 15=person (192, 128, 0), (64, 0, 128), (192, 0, 128), (64, 128, 128), (255, 255, 255), # 16=potted plant, 17=sheep, 18=sofa, 19=train, 20=tv/monitor (0, 64, 0), (128, 64, 0), (0, 192, 0), (128, 192, 0), (0, 64, 128)]) r = np.zeros_like(image).astype(np.uint8) g = np.zeros_like(image).astype(np.uint8) b = np.zeros_like(image).astype(np.uint8) for l in range(0, nc): idx = image == l r[idx] = label_co…