Building a Security System using Computer Vision

By

Computer Vision. Does it sound familiar?  

No? 

I am sure the following pictures should look familiar.


or this?



or this?



All of the above, are few out of many applications of Computer Vision technology. 

Computer Vision is a field focused on equipping computers, to derive information from the digital images and videos. In short, it has the power to convert a camera lens into an eye. 

When a Tesla car runs in autopilot mode, it automatically gains understanding of most of the things on the road, including road path, vehicles, people, traffic sign boards, traffic lights etc. This happens in just the same way, a human driver driving a car understands what his eyes are watching. Actually it is a number of cameras(similar to human eye) viewing the surroundings, then communicating with the processing system(similar to human brain) in Tesla car, which understands what it is viewing and then takes the required action. 

This programmatic communication and the resultant action, is what Computer Vision makes possible for us. In addition to cameras there are several sensors too, which detects physical objects and bodies around. Though in this article, our focus would be on the vision aspect of information gathered. 

There are several subdomains of computer vision including i) object detection, ii) scene reconstruction, iii) event detection, iv) video tracking, v) 3D pose estimation, vi) learning, vii) indexing, viii) motion estimation, ix) visual servoing, x) 3d scene modeling and xi) image restoration. We'll use computer vision to build a surveillance security system.

What we'll build - A security system, which will track the motion of any unwanted intruder and on detecting that motion would alarm us with a sound. 

Surveillance and Security are sometimes two separate systems. But we'll build a product which would be a combination of these two, with tracking as well as alarming features. 

Following 2 videos explains the output of the product we'll build today:

Video 1 - Here we see an unwanted intruder, who is moving in the backyard of a house. I dowloaded this video from Youtube:

No security system installed

Video 2 - Once we have developed this security system, it will automatically alarm us(with a sound) of any such unwanted motion of an intruder. 

Following video(with alarm sound) is the output of our product:

Security system is installed


Technologies we'll use : 

1. OpenCV - In my opinion, it is one of the finest Computer Vision technology suite available today. Two key reasons supporting this:

  • It has rich cross platform support for desktop OS, including Windows, Linux, MacOS, FreeBSD etc and also mobile OS including Android, iOS, Maemo and Blackberry10. 
  • It supports GPU acceleration for real time operations.

2. Language - Python.

3. Development Environment(IDE) - Please use the one you are most comfortable with. I personally prefer Pycharm for my python coding. But again, please use the one you are most comfortable with.


Lets build our system:

For a moment, let us think like the end customer of our product. What would he/she expect this security system to do?

On asking this key question, we get 3 goals:

1. Capture the video of the required area.
2. Detect motion which is suspicious.
3. Produce alarm sound(on detecting suspicious motion).
   
We'll achieve all of these step by step.

NOTE - To build this system, we need to write code. And I will be explaining each step of the coding required to build this, along with the theoretical explanation. 
Hence this article would be technical from this point onwards. And programming experience is beneficial to follow along. 

If coding doesn't fascinate you, still there is lot of interesting content for you, around how motion is detected and different image processing techniques required to build this. 

Preliminary setup:

First create a python project in your favorite IDE, providing it a name you like, say ComputerVision. 
To get OpenCV library, use pip package manager and execute the following command in your IDE. 
In Pycharm you can open the terminal and simply execute the following pip install command:
pip install opencv-python

Then create your python code file(.py extension) inside the project. Provide it a name you like.
Import the OpenCV package using: 
import cv2

Step 1 - Capture video:

To capture the video, following is the code which I'll explain step by step:

import cv2
video_camera=cv2.VideoCapture("backyard_intruder.mp4")

while video_camera.isOpened():
    ret_value,frame=video_camera.read()

    if cv2.waitKey(5)== ord('x'):
        break

cv2.imshow("Akshay's cam", cv2.resize(frame,None,None,fx=0.40,fy=0.45))  


OpenCV's VideoCapture class facilitates capturing video from your camera or any existing video file. 
Create a vedio_camera object using the constructor of VideoCapture class, which takes one argument. This argument can be either the index of the actual camera(which we need to capture the video from) or the name of a video file. 
To use just the webcam of your laptop, pass 0 as the argument as follows: 
video_camera = cv2.VideoCapture(0)
For this exercise, I have used an intrusion video(backyard_intruder.mp4) which I downloaded from Youtube, as was captured by a surveillance camera. I'll build this security system on top of this video. 

To ensure the video is being captured, we would use the isOpened() method of VideoCapture class in a while loop, which returns True if the video capturing has been initialized already, else False and the while loop breaks.


To capture and show the video or camera's output on screen we need two things:

1. Read the captured video: 
This is achieved using read() method of  VideoCapture class. This method grabs and returns the next frame captured.
ret_value,frame2 = video_camera.read()

2. Show it on the screen:
Use OpenCV's imshow() method. This shows the image in a window. Takes 2 arguments 
1) Window name 
2) Image to show
cv2.imshow("Akshay's cam", frame)
I have named the window as "Akshay's cam".You can give it any preferred name. 
I am also adding a functionality to allow us to stop the video, by pressing a key('x' key) on the keyboard, using OpenCV's waitKey() method:
if cv2.waitKey(5) == ord('x'):
    break
This waits for 5 milliseconds and look for any pressing event of x key. As soon as you press x key it will close the window.

Now we have the code which shows the selected video, or anything captured by web cam of the laptop in a window named "Akshay's cam".


This opens up the following video(I am showing a still frame from the video):


Since this video wasn't appropriately fitting in the window, hence I have resized it using cv2.resize to get the whole video visible on my screen. Using fx and fy as the x axis and y axis resize percentages.
 

Step 2 - Detect motion:

How is motion detected? 🤔

Just look at this video stream of a heart, flying out of a magician hat into the clouds.


But wait !!  Is it a video? Isn't it just a booklet of still pages, when flipped in an order, producing this illusion of a video?

Yes both the statements are correct. 

It is a video & it is a sequence of still images. 
Because a video is actually a sequence of still images(frames), when displayed at a frame rate called FPS(Frames per second). I guess you might have heard this term FPS at some point.

Read this very carefully 👇
If the person holding this book, decide not to flip even a single page and just shows us the first page(as shown in below image), we would not see any motion. We would be able to see motion, only when he shows the second page(frame) which is slightly different from the first page(frame). 

Makes sense? 
If not, please read the above statement once again, because this is the fundamental concept behind motion detection.


We'll use this concept to detect the intruder's motion. We'll grab all the still frames from our video and calculate the difference of each frame with the previous frame. And any difference in frames would be a sign of motion. We'll get this difference using absdiff() method of OpenCV.

I am creating 2 frames frame1 and frame2 in sequence, inside the while loop and getting a difference difference_frames using absdiff() method of OpenCV. 
absdiff()requires 2 frames to compare as arguments.
while video_camera.isOpened():
    ret_value, frame1 = video_camera.read()
    ret_value, frame2 = video_camera.read()
    difference_frames=cv2.absdiff(frame1,frame2)
    if cv2.waitKey(5) == ord('x'):
        break

cv2.imshow("Akshay's cam", cv2.resize(difference_frames, None, None, fx=0.40, fy=0.45))

Above code detects motion and following video is the result of this code:



Now that we have the motion detected, we need to apply different image processing techniques such as changing color scale, smoothening using blurring, thresholding, dilation etc. Don't worry we'll look at each of these, with a proper video showing the result of applying each technique. Deeper details of these techniques, are out of the scope of this article and we'll focus on these, to the extent required for building this system.

i) Gray scale - For detecting the motion and form accurate contours(which we'll perform just in next steps), a gray scale image is always recommended over a colored image. Precisely for our use case, colors are more of a noise. 
Hence we'll convert the color from RGB to Gray scale using cvtColor method as follows, passing the source as difference_frames and color code as COLOR_RGB2GRAY :
gray_difference_frames=cv2.cvtColor(difference_frames,cv2.COLOR_RGB2GRAY)
Now when we show the grayscale differences in the window as follows, we'll see it gray scaled instead of RGB colored.
cv2.imshow("Akshay's cam", cv2.resize(gray_difference_frames, None, None, fx=0.40, fy=0.45))
Grayscale

ii) Blurring - To further reduce noise and smoothen the transition, we should smoothen the frames using a blurring technique. Though we have several options of blurring as part of OpenCV, such as blur(),GaussianBlur(), medianBlur() etc. But the one I'll use is GaussianBlur(). 
It takes 3 arguments -> source - which will be the gray scaled frames difference, Gaussian kernel size and Kernel's standard deviation in X direction(which we are setting to 0):

blur_frames=cv2.GaussianBlur(gray_difference_frames,(5,5),0)
Lets show this blurred result, and you will see the edges of this person are more blurred:
cv2.imshow("Akshay's cam", cv2.resize(blur_frames, None, None, fx=0.40, fy=0.45))
Blurred


iii) Thresholding - Thresholding is an important image segmentation technique to convert grayscale image into binary image and separates the foreground pixels from the background pixels. 
Using thresholding, we'll separate our focus entity's(intruder) foreground pixels from the background pixels. We'll use threshold method which takes in the src, thresh which I'll keep at 20(sufficient for this use case), maxval maximum value to be used, and type of thresholding which I am using THRESH_BINARY as follows. There are though other thresholding types such as THRESH_MASK,THRESH_OTSU,THRESH_TOZERO,THRESH_TRIANGLE etc which you can use on as needed basis:
_, threshold = cv2.threshold(blur_frames, 20, 255, cv2.THRESH_BINARY)

On passing this threshold to imshow() we get the following result:
cv2.imshow("Akshay's cam", cv2.resize(threshold, None, None, fx=0.40, fy=0.45))
Threshold



iv) Dilation - Dilation is a technique, which adds pixels to the boundary of the objects, such that the holes in the objects are filled and they become more visible. The opposite of dilation is erosion. 
We'll use OpenCV's dilate() method. The key arguments for this are the source and iteration(number of times dilation is applied), for which 3 is a sufficient value for our use case.
dilated_otpt = cv2.dilate(threshold, None, iterations=3)
On showing the dilated version using imshow(), we see more visible boundaries of the moving intruder. Just compare the following dilation video with the previous threshold video and you can feel the difference in the boundaries.

cv2.imshow("Akshay's cam", cv2.resize(dilated_otpt, None, None, fx=0.40, fy=0.45))

Dilated


v) Contours - Now is the time to create contours around our focus point which is the intruder. 
Well what is a contour? 
Definition - Image contouring is a process of identifying structural outlines of objects in an image. 
In simple terms, these green rectangles shown in the image below are the contours.



a) findContours() - I am using findContours() method of OpenCV and passing the dilated_otpt as image, mode which is a retrieval mode. Though there are different modes, I am choosing RETR_TREE which retrieves all of the contours and reconstructs a full hierarchy of even all nested contours. findContours() also requires ContourApproximationMode, which I am using CHAIN_APPROX_SIMPLE,to retain only the required points and remove all the redundant points, which is not achievable using CHAIN_APPROX_NONE. 
contours_fetched, _ = cv2.findContours(dilated_otpt, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
b) drawContours() - Use drawContours() for drawing the contour around intruder. This needs: 
> image, which I am passing my frame1, 
> contours, which is the output of findContours() ie' contours_fetched,
> contourIdx, keeping this as -1, because if this is negative then all the contours are drawn,
> color, which I am keeping as green (0, 255, 0). Remember these are RGB color, where I am keeping 0 for both Red and Blue and 255 for green. 
> thickness, I am setting this to 10, to make contours look a bit thick.

cv2.drawContours(frame1, contours_fetched, -1, (0, 255, 0), thickness=10)
Following video shows the resultant green contours:

Contours Generated


But wait !! 
If you replay above video and notice carefully, even the flies in the backyard, as well as the moving tree leaves due to wind, are also getting contours created around them. But we don't want to be alarmed, in the middle of the night, if an insect flies in the backyard or a leaf moves due to blowing wind. For this I'll retain only the required contours.

Retaining only the desired contours:

To achieve this, we'll filter all the contours which are lesser than a particular area. 
A contour's area can be retrieved using contourArea()method, to which I'll pass each contour by iterating over contours_fetched, which was the result of findContours() as shown previously. Only if the area of the contour is greater than 73000, then we would get the bounding rectangle coordinates and form the rectangle contour around it, else we'll continue the while loop. 
boundingRect(contour)returns the x coordinate, y coordinate, width and height of the bounding rectangle. 
Calling the rectangle(frame1,(x,y),(x+w,y+h),(0,255,0),thickness=10)by passing it frame1, start and end coordinate points of rectangle which are (x,y) and (x+w,y+h), we get the rectangle formed. For green color using (0,255,0)and a thickness of 10 of the green rectangle contour. 

Following code achieves this logic of filtering small contours and forming rectangle contours only around the intruder.
for contour in contours_fetched:
    if cv2.contourArea(contour)<73000:
        continue
    x, y, w, h = cv2.boundingRect(contour)
    cv2.rectangle(frame1,(x,y),(x+w,y+h),(0,255,0),thickness=10)
Finally we have now reached the point where we are now able to detect the motion of the intruder in the backyard and draw contours around his motion, as show in this video:
Required contours around intruder's motion


Step 3 - Alarm on detecting the motion:

Now the final step. When the motion of intruder is detected, we want an alarming sound to be played, so that we can take necessary action. This I'll achieve using winsound. Just import winsound using:
import winsound
This module has PlaySound utility function, to which we can pass a sound .wav file. I am passing a TF016.wav sound file. Also since this sound should play asynchronously as soon as the motion is detected, we'll use winsound.SND_ASYNC to ensure asynchronous alarm sound. 
This TF016.wav I have kept in the same root folder as my code. 
To play sound, there are also other different modules such as playsound, which you can simply install using pip install playsound. It is a light weight single function module, which has a function called playsound and supports different sound file formats. 

Now that the alarm sound has been integrated, let us look at the final output of our surveillance alert system:

Output with alarm sound on intruder's motion


My full source code(from my Pycharm IDE):



I hope this was helpful, to understand how we can use Computer Vision to build intelligent systems, with intelligence a bit similar to what human eyes and brain together can achieve.

Big Data, Cloud(AWS), Graph and Machine Learning professional.

Comments

New comments are not allowed.

Popular posts from this blog

UPSERT in Hive(3 Step Process)

By
Image
In this post we'll learn an efficient 3 step process, for performing UPSERT in hive on a large size table containing entire history. Just for the audience not aware of UPSERT - It is a combination of UPDATE and INSERT. If on a table containing history data, we receive new data which needs to be inserted as well as some data which is an UPDATE to the existing data, then we have to perform an UPSERT operation to achieve this. Prerequisite  – The table containing history being very large in size should be partitioned, which is also a best practice for efficient data storage, when storing large volume of data in any Big Data warehouse. Business scenario  – Lets take an example of click stream data of a website, as gathered from different browsers of visitors who visited the website. The site_view_hist table contains the clicks and page impressions counts from different browsers and the table is partitioned on hit_date(the date on which the visitor hitted or visited the website).
Read more

Parquet tools utility

By
Image
It is a well known fact, that parquet format is a binary format. And for humans, reading binary data is not as easy as opening a text file and understanding its contents. But no worries, today we'll learn how to read a parquet file . If you would like to understand all the internal details of a parquet file , you can please refer my previous article here . In this article I have tried to explain all the inner storage details of Parquet and answered most of the questions around its usage, efficiency and performance.                                                                                Parquet tools - As we know parquet is a binary format, we need an interpreter to translate its contents so that we can understand them. And this is what parquet-tools  utility project does for us. It lets us read parquet files. Note - At the time of writing this blog, I used parquet-tools utility already installed on my Hadoop cluster. But you can always install parquet-tools on you
Read more

Hive - Merging small files into bigger files.

By
Image
Since Hadoop's inception, we have learnt that it is good for processing small number of large files, instead of large number of small files. Part of the reason pertaining to this behavior is, its MapReduce Paradigm which works on splits of input data.  In our day to day data processing, there can be various situations where our processes can produce large number of small files. Assume a process generated output data of 300 MB, but this data is distributed in approximately 3000 files of 99 KB each. Today we'll learn how to merge such large number of small files into bigger files. Before that let us quickly understand the problems, such large number of small files can cause. Problems posed by large number of small files -   1.   I f we store the data of this table in GZIP format(3000 files each of 99KB), then a downstream process running a hive query(MapReduce job) on this data, would spin up 3000 mappers to process individual splits for each file, as the GZIP format is non
Read more

Parquet File format - Storage details

By
Image
Columnar storage has revolutionized Big data processing, since its inception.  Its power can be realized from the fact that Google Big Query, Hbase, Amazon Redshift, Azure SQL Data Warehouse and many more, all utilize columnar storage. As you are here today reading this article, it is obvious that you are curious to learn how Columnar storage internally works. Well, just stay tuned for next few minutes, as  I'll explain all the details of the  most popular columnar file format called Parquet.  We'll learn the details  by  actually opening  a parquet file and by  going deep, at the level of how things work under the hood at disc level.  We'll also learn why Parquet would save cost, when it is used as underlying file format with A WS services such as Athena . When to use Parquet format  -  Just check this less than 30 seconds conversation, going on between a Data Engineer(male) and a  BI Analyst(female) . When to use parquet - Whenever our dataset has large num
Read more

Skew Join Optimization in Hive

By
Image
Skew is a very common issue which most of the data engineers come across.  What is Skew - When in our data we have very large number of records associated with one(or more) particular key, then this data is said to be skewed on that key.  We'll understand this with a very simple example. Think about world's population data table, where each record is of one person and the number of records equals to total world population. As we all know China and India, each accounts for around 17-18% of world's population, hence such population table is referred to as skewed on keys country=China and country=India. If you focus on the following pie chart of world's population distribution by country, then this would make exact sense. Problem statement  - Assume we have a table which tracks the online visits to different online websites. In this table amazon.com  being very popular,  has 1 Billion rows whereas other not very popular sites ( www.abc.com , www.xyz.com..etc) i
Read more

Apache Spark - Sort Merge Join

By
Image
Apache Spark is a well known open source cluster computation system, for large scale data processing. If you are into Big Data and Machine Learning, Apache Spark may not need any introduction. Its capabilities include In-memory processing, Distributed SQL Analytics, Real Time stream processing, Graph processing, Machine Learning,  support for several programming languages  and many more.  " What happens under the hood, when we execute a Spark application? ".    thinking????? This is a very common question, which crops up in every Apache Spark user's mind. If you have the same question, I 'll answer this using Spark SQL by executing one of the most common data processing operation i.e. "Join". Spark being a vast technology, it is nearly impossible to cover all of Spark's details in a single article. And we all learn something new, every time we use a technology to solve a new challenge.  However, the details I'll cover here, should be enough to give
Read more

Apache Spark - Explicit Broadcasting Optimization

By
Image
Join operation is an indispensable part of majority of Big Data Projects. If you are using Apache Spark for your Big Data processing and using join operation, in this article I'll explain a strategy to optimize joins. The technique I'll explain is how to explicitly broadcast a table, in any Spark join operation and convert the join operation to Broadcast Hash Join. Why Broadcasting - W hen we perform a join operation in Spark SQL, b roadcasting proves very efficient in reducing data shuffling and hence serialization and deserialization of data over the network, which happens during a Sort Merge Join operation. If you haven't read my previous article on Spark's Broadcast Hash join, I highly recommend you to first go through that. This would provide you most of the deeper details of Spark's Broadcast hash Join. You can please find it here :- Broadcast Hash Join - By Akshay Agarwal Business use case  -  To explain this optimization, let's take the business
Read more

Spark Optimization - Bucketing

By
Image
Complex data computation is a part of every Big Data Project.  Do you have a business scenario requiring complex join on data having billions or trillions of values?  If you are using Apache Spark, stay tuned for next few minutes, as this article will address this expensive join problem, using Spark's optimization technique called Bucketing. Before reading this article, I recommend you to once go through my previous articles on Spark's Sort Merge Join and Broadcast Hash Join . They will set up your foundation for understanding bucketing technique. It is recommended but is not mandatory, as you would still be able to grasp the core concept. When to use Bucketing ?    For any business problem, where we need to perform a join on tables with very high cardinality on join column( I repeat very high ) in say millions, billions or even trillions and also this join needs to happen multiple times. Lets understand bucketing using the data of a hypothetical service provider company
Read more

Apache Spark - Broadcast Hash Join

By
Image
In this technical article, we'll discuss about one of Apache Spark's efficient join technique called Broadcast Hash Join. I will explain in detail how this operation is implemented internally in Spark at DAG(Directed Acyclic Graph) level and also different phases of DAG. We'll also uncover the details of division of spark application into jobs and stages.  Data  -  To explain this join concept, I'll take customer data of a hypothetical online service provider organization with international presence. Different customers across the world using its services holds accounts with this company. Moreover the customers uses different devices to access this company's website. Hence to fetch the details of different  customers(business_accounts) and the corresponding device used by them, we'll use two datasets: Business Accounts data   -  Data consisting of details of the customer's business account, created with this service providing organization.  The schema f
Read more

Graph Technology - In plain English

By
Image
Graph processi ng systems have become a prominent buzzword in computation world in recent years. Is this just a passing trend which is here today and would vanish tomorrow? There is no harm in being skeptical. But with knowledge graphs recognized as a top emerging technology trend by industry expert Gartner, it's evident that graph technology is here to stay.  So let's dive in and discover what makes graph technology special and why it's gaining momentum.   What industry experts are saying about Graph Technology: Knowledge graphs has been recognized by Gartner among the top 5 emerging technology trends, alongside IOT & Blockchain, that will blur the lines between humans and machines. I have included a snippet from Gartner's press release below. You can also find Gartner's official full press release   here  . Snippet from Gartner's Newsroom Press Release  An important note before we proceed further. If you do a simple google search about graphs, y
Read more