Ways To Implement Instance Segmentation By Location with SOLO and SOLOv2
Table of contentsIntroductionCentral Idea of SOLOPractical implementation of SOLOPros of SOLOLimitations of SOLOOverview of SOLOv2Practical implementation of SOLOv2 Introduction SOLO (segment objects by locations) is a simple and flexible framework applied for accomplishing instance segmentation in digital image processing and computer vision tasks. It is based on the notion of “instance categories” for instance segmentation in which each pixel within an instance of an object is assigned a category based on its location and size. SOLO was introduced by Xinlong Wang and Chunhua Shen at The University of Adelaide, Australia along with Tao Kong, Yuning Jiang 2 and Lei Li of the ByteDance AI Lab in July 2020 (revised version v3). Click here to read the research paper. Before moving on to the details of the SOLO approach, let us first understand the concept of instance segmentation. Instance segmentation is different from semantic segmentation method. Semantic segmentation refers to the process of associating every pixel of an image with a class label such as a person, flower, car and so on. It treats multiple objects of the same class as a single entity. On the contrary, instance segmentation treats multiple objects of the same class as distinct individual instances. Semantic segmentation Instance segmentation As shown in the above figure, semantic segmentation identifies all the entities belonging to the ‘person’ category whereas instance segmentation identifies each individual within the category as a different entity (person1, person2 etc.). All the objects comprising an image belong to a fixed set of semantic categories so semantic segmentation can be easily formulated as a dense per-pixel classification problem. However,  the number of instances in each semantic category may vary and hence, it is a challenging task to predict instance labels directly following the same paradigm as semantic segmentation. Central Idea of SOLO SOLO reforms the task of instance segmentation into two simultaneously performed classification tasks. It first divides an input image into uniform grids. If the centre of an object lies in a grid cell, that cell has to perform the following two tasks: Predict the semantic categoryThe segment that object’s instance(s) SOLO differentiates object instances in an image based on ‘instance categories’ i.e., the quantized centre locations and object sizes. The concept of instance categories allows us to Segment Objects by LOcations and hence the name, SOLO. Recent instance segmentation methods follow one of the two approaches: Top-down approach: Detect bounding boxes around the object(s) and then segment the instance mask in each bounding box to distinguish separate instances of the object (called ‘detect-then-segment’ approach) Bottom-up approach: Pull close pixels of the same instances and push away the pixels of different instances, thus creating an affinity relationship between them and assigning an embedding vector to each pixel. Then group similar pixels to delineate instances. The above paradigms are step-wise and more importantly ‘indirect’ i.e. they rely on precise detection of bounding boxes or embedding learning followed by pixels’ grouping process. On the contrary, SOLO is a ‘direct’ approach relying on full mask annotations. Practical implementation of SOLO Installation: The installation process is based on MMDetection (v1.0.0) (MMDetection is an open-source toolbox based on PyTorch and used for object detection). Prerequisites: Linux (Windows – not supported officially)Python 3.5+PyTorch 1.1 or higher (version higher than 1.5 is not tested)CUDA 9.0 or higherNCCL 2GCC 4.9 or highermmcv 0.2.16 Script for setting up SOLO with conda and linking the dataset path is as follows: Create conda virtual environment and activate it conda create -n solo python=3.7 -y conda activate solo Install PyTorch and torchvision conda install -c pytorch pytorch torchvision -y Install the compiled programming language – Cython conda install cython -y Clone the SOLO GitHub repository git clone https://github.com/WXinlong/SOLO.git cd SOLO Install build requirements followed by SOLO pip install -r requirements/build.txt pip install "git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI" pip install -v -e. Link the dataset path mkdir data ln -s DATASET_PATH data Once the installation is successfully completed, download the required model (list of provided models is available here) and then run the .py code file coded as follows: from mmdet.apis import init_detector, inference_detector, show_result_pyplot, show_result_ins Define the config file path e.g. config = ‘../config/solo/decoupled_solo_r50_fpn_8gpu_3x.py’ Download checkpoint from the model zoo and place it in ‘checkpoints/’ checkpt = '../checkpoints/DECOUPLED_SOLO_R50_3x.pth' Build the model from config file and checkpoint file model = init_detector(config, checkpt, device='cuda:0') Test a single image, say abc.jpg test_image = ‘abc.jpg’ test_result = inference_detector’(model, test_image) show_result_ins(test_image, test_result, model.CLASSES, score_thr=0.25, out_file="output.jpg") Train with a single GPU python tools/train.py ${CONFIG_FILE} Train with multiple GPUs ./tools/dist_train.sh ${CONFIG_FILE} ${GPU_NUM} Test with a single GPU python tools/test_ins.py ${CONFIG_FILE} ${CHECKPOINT_FILE} --show --out  ${OUTPUT_FILE} --eval segm Test with multiple GPUs ./tools/dist_test.sh ${CONFIG_FILE} ${CHECKPOINT_FILE} ${GPU_NUM}  --show --out  ${OUTPUT_FILE} --eval segm Visualize the results python tools/test_ins_vis.py ${CONFIG_FILE} ${CHECKPOINT_FILE} --show --save_dir  ${SAVE_DIRECTORY} Source: GitHub repository Pros of SOLO Totally box-free: SOLO is not restricted by (anchor) box locations and scales and hence benefit from the inherent advantages of Fully Connected Networks (FCNs).Direct instance segmentation: SOLO takes an image as input, directly outputs instance masks and the corresponding semantic class probabilities in a fully convolutional, box-free and grouping-free paradigms Limitations of SOLO inefficient mask representation and embedding learningnot high enough resolution for finer mask predictionsslow mask Non-Maximum Suppression (NMS) The above-mentioned bottlenecks of SOLO are eliminated by a dynamic and faster framework called SOLOv2. Overview of SOLOv2 SOLOv2 was proposed by Xinlong Wang and Chunhua Shen of The University of Adelaide (Australia), Rufeng Zhang of Tongji University (China) as well as Tao Kong and Lei Li of ByteDance AI Lab in October, 2020. Refer to the SOLOv2 research paper to understand its working and underlying concepts. SOLOv2 is a dynamic scheme for segmenting objects by locations. It divides the mask learning process into two parts: convolution kernel learning and feature learning. It predicts mask kernels dynamically according to the input while assigning appropriate location categories to different pixels. It then constructs a unified and high-resolution mask feature representation for instance-aware segmentation. The process of learning the mask kernels and mask features can be carried out separately with efficiency. To prevent duplicate predictions, it employs a matrix NMS algorithm Practical implementation of SOLOv2 Click here and follow the installation steps before proceeding further! Run the following command lines: wget https://cloudstor.aarnet.edu.au/plus/s/chF3VKQT4RDoEqC/download -O SOLOv2_R50_3x.pth python demo/demo.py \ --config-file configs/SOLOv2/R50_3x.yaml \ --input input1.jpg input2.jpg \ --opts MODEL.WEIGHTS SOLOv2_R50_3x.pth Set up the required dataset e.g. MS-COCO Train the model on COCO OMP_NUM_THREADS=1 python tools/train_net.py \ --config-file configs/SOLOv2/R50_3x.yaml \ --num-gpus 8 \ OUTPUT_DIR training_dir/SOLOv2_R50_3x Evaluate the model on COCO OMP_NUM_THREADS=1 python tools/train_net.py \ --config-file configs/SOLOv2/R50_3x.yaml \ --eval-only \ --num-gpus 8 \ OUTPUT_DIR training_dir/SOLOv2_R50_3x \ MODEL.WEIGHTS training_dir/SOLOv2_R50_3x/model_final.pth Source: GitHub repository Refer to the following links to have a deep understanding of SOLO and SOLOv2: SOLO Research PaperSOLO GitHub RepositorySOLOv2 Research PaperSOLOv2 GitHub Repository
Introduction SOLO (segment objects by locations) is a simple and flexible framework applied for accomplishing instance segmentation in digital image processing and computer vision tasks. It is based on the notion of “instance categories” for instance segmentation in which each pixel within an instance of an object is assigned a category based on its location […]
SOLO and SOLOv2 are simple and flexible framework for instance segmentation in digital image processing and computer vision tasks
[]
["CUDA", "Go", "TPU", "PyTorch", "AI", "ML", "computer vision", "Python", "object detection", "R"]
["AI", "ML", "computer vision", "PyTorch", "object detection", "TPU", "CUDA", "Python", "R", "Go"]
A zealous learner aspiring to advance in the domain of AI/ML. Eager to grasp emerging techniques to get insights from data and hence explore realistic Data Science applications as well.