review grabcut

Author: abidrahmank

source/py_tutorials/py_imgproc/py_grabcut/images/grabcut.jpg

@@ -15,7 +15,7 @@ Theory
 
 GrabCut algorithm was designed by Carsten Rother, Vladimir Kolmogorov & Andrew Blake from Microsoft Research Cambridge, UK. in their paper, `"GrabCut": interactive foreground extraction using iterated graph cuts <http://dl.acm.org/citation.cfm?id=1015720>`_ . An algorithm was needed for foreground extraction with minimal user interaction, and the result was GrabCut. 
 
-How it works from user point of view ? Initially user draws a rectangle around the foreground region (foreground region shoule be completely inside the rectangle). Then algorithm segments it iteratively to get the best result. Done. But in some cases, the segmentation won't be fine, like, it may have marked some foreground region as background and vice versa. In that case, user need to do fine touch-ups. Just give some strokes on the images where some faulty results are there. Strokes basically says "Hey, this region should be foreground, you marked it background, correct it in next iteration" or its opposite for background. Then in the next iteration, you get better results. 
+How it works from user point of view ? Initially user draws a rectangle around the foreground region (foreground region shoule be completely inside the rectangle). Then algorithm segments it iteratively to get the best result. Done. But in some cases, the segmentation won't be fine, like, it may have marked some foreground region as background and vice versa. In that case, user need to do fine touch-ups. Just give some strokes on the images where some faulty results are there. Strokes basically says *"Hey, this region should be foreground, you marked it background, correct it in next iteration"* or its opposite for background. Then in the next iteration, you get better results.
 
 See the image below. First player and football is enclosed in a blue rectangle. Then some final touchups with white strokes (denoting foreground) and black strokes (denoting background) is made. And we get a nice result.
 
@@ -25,7 +25,7 @@ See the image below. First player and football is enclosed in a blue rectangle.
 
 So what happens in background ? 
 
-    * User inputs the rectangle. Everything outside this rectangle will be taken as sure background (That is the reason it is mentioned before that your rectangle should include all the object). Everything inside rectangle is unknown. Similarly any user input specifying foreground and background are considered as hard-labelling which means they won't change in the process.
+    * User inputs the rectangle. Everything outside this rectangle will be taken as sure background (That is the reason it is mentioned before that your rectangle should include all the objects). Everything inside rectangle is unknown. Similarly any user input specifying foreground and background are considered as hard-labelling which means they won't change in the process.
     * Computer does an initial labelling depeding on the data we gave. It labels the foreground and background pixels (or it hard-labels)
     * Now a Gaussian Mixture Model(GMM) is used to model the foreground and background.
     * Depending on the data we gave, GMM learns and create new pixel distribution. That is, the unknown pixels are labelled either probable foreground or probable background depending on its relation with the other hard-labelled pixels in terms of color statistics (It is just like clustering).
@@ -50,7 +50,7 @@ Now we go for grabcut algorithm with OpenCV. OpenCV has the function, **cv2.grab
     * *rect* - It is the coordinates of a rectangle which includes the foreground object in the format (x,y,w,h)
     * *bdgModel*, *fgdModel* - These are arrays used by the algorithm internally. You just create two np.float64 type zero arrays of size (1,65).
     * *iterCount* - Number of iterations the algorithm should run.
-    * *mode* - It should be *cv2.GC_INIT_WITH_RECT* or *cv2.GC_INIT_WITH_MASK* or combined which decides whether we are drawing rectangle or final touchup strokes.
+    * *mode* - It should be **cv2.GC_INIT_WITH_RECT** or **cv2.GC_INIT_WITH_MASK** or combined which decides whether we are drawing rectangle or final touchup strokes.
 
 First let's see with rectangular mode. We load the image, create a similar mask image. We create *fgdModel* and *bgdModel*. We give the rectangle parameters. It's all straight-forward. Let the algorithm run for 5 iterations. Mode should be *cv2.GC_INIT_WITH_RECT* since we are using rectangle. Then run the grabcut. It modifies the mask image. In the new mask image, pixels will be marked with four flags denoting background/foreground as specified above. So we modify the mask such that all 0-pixels and 2-pixels are put to 0 (ie background) and all 1-pixels and 3-pixels are put to 1(ie foreground pixels). Now our final mask is ready. Just multiply it with input image to get the segmented image.
 ::
@@ -75,13 +75,13 @@ First let's see with rectangular mode. We load the image, create a similar mask
     
 See the results below:
 
-    .. image:: images/grabcut_rect.png
+    .. image:: images/grabcut_rect.jpg
         :alt: Segmentation in rect mode
         :align: center
 
-Oops, Messi's hair is gone. What is Messi without his hair? We need to bring it back. So we will give there a fine touchup with 1-pixel (sure foreground). At the same time, Some part of ground has come to picture which we don't want, and also some logo. We need to remove them. There we give some 0-pixel touchup (sure background). So we modify our resulting mask in previous case as we told now. 
+Oops, Messi's hair is gone. *Who likes Messi without his hair?* We need to bring it back. So we will give there a fine touchup with 1-pixel (sure foreground). At the same time, Some part of ground has come to picture which we don't want, and also some logo. We need to remove them. There we give some 0-pixel touchup (sure background). So we modify our resulting mask in previous case as we told now.
 
-*What I actually did is that, I opened image in paint, added another layer to image. Using brush tool in the paint, I marked missed foreground (hair, shoes, ball etc) with white and unwanted background( like logo, ground etc) with black. Filled remaining background with gray. Then loaded that mask image in OpenCV, edited original mask image we got with corresponding values in newly added mask image. Check the code below*
+*What I actually did is that, I opened input image in paint application and added another layer to the image. Using brush tool in the paint, I marked missed foreground (hair, shoes, ball etc) with white and unwanted background (like logo, ground etc) with black on this new layer. Then filled remaining background with gray. Then loaded that mask image in OpenCV, edited original mask image we got with corresponding values in newly added mask image. Check the code below:*
 ::
 
     # newmask is the mask image I manually labelled
@@ -92,7 +92,7 @@ Oops, Messi's hair is gone. What is Messi without his hair? We need to bring it
     mask[newmask == 0] = 0
     mask[newmask == 255] = 1
 
-    cv2.grabCut(img,mask,None,bgdModel,fgdModel,5,cv2.GC_INIT_WITH_MASK)
+    mask, bgdModel, fgdModel = cv2.grabCut(img,mask,None,bgdModel,fgdModel,5,cv2.GC_INIT_WITH_MASK)
 
     mask = np.where((mask==2)|(mask==0),0,1).astype('uint8')
     img = img*mask[:,:,np.newaxis]
@@ -109,7 +109,10 @@ So that's it. Here instead of initializing in rect mode, you can directly go int
 Additional Resources
 =======================
 
+
+
 Exercises
 ============
-#. Here, you can make this into a interactive sample with drawing rectangle and strokes with mouse, create trackbar to adjust stroke width etc. Such an example is already there in OpenCV samples. Please have a look at it.
 
+#. OpenCV samples contain a sample ``grabcut.py`` which is an interactive tool using grabcut. Check it. Also watch this `youtube video <http://www.youtube.com/watch?v=kAwxLTDDAwU>`_ on how to use it.
+#. Here, you can make this into a interactive sample with drawing rectangle and strokes with mouse, create trackbar to adjust stroke width etc.