Abnormal Behaviour Detection in Crowds

 

Table of Contents

Abstract

1. INTRODUCTION

2. RELATED WORK

3. Methods

3.1 SFM (Social Force Model)

3.2 The modification SFM

3.2.1 Explanation to Modification SFM

3.2.2 Modification in algorithm

4.0 Analysis

4.1 Comparison between SFM (Social Force Model) and Modified SFM

4.2 Comparison between proposed social force model and pure optical flow

5. Research Trends

6. Conclusion

References

 

 

This report presents a review of literatures that focus on abnormal event detection techniques meanwhile introduced the Generalized Social Force Model with detailed algorithms and basic concepts of it, but meanly the report focuses on illustrating the Modified Social Force Model based on the Generalized Social Force Model, both of them were invented for detecting abnormal crowd events in videos. The Modified Social Force Model uses the particle advection method to present the movements of human crowds instead of tracking individual pedestrians in modelling phase. The method places a grid of particles on images, the moving particles represent pedestrians in the crowd. The modified model and algorithms will be used to analyse the group of behaviours from individual particles, the model computes the interaction forces. In event detecting phase, the modified model uses ‘bag of words’ approach, LDA (Latent Dirichlet Allocation) method, and the EM (Expectation Maximization) algorithm to detect and distinguish frames as normal frame or abnormal frame, and locate the abnormal area according to force flow, which means the area with ‘strong’ force flow also means the ‘strong’ interaction behaviour between individuals in that area. This report also provided the results of experiments based on sample datasets and real cases. The report mainly compared the Generalized Social Force Model with Modified Social Force Model theoretically. Meanwhile the comparison between Modified Social Force Model and pure optical flow method was given, the performances are evaluated according to Receive Operating Characteristic (ROC) curves and Area Under Curve (AUC) values. The result shows Modified Social Force Model has stronger capability theoretically, and modified model is more robust and precisely rather than pure optical methods. The report presents the future trends of analysis and research, the future trend will possibly be combine other subjects for instance micro-psychology, socio-psychological, human behaviouristics, even religious studies with existing models in order to make these models more anthropomorphism or personification, which might be helpful in simulating human crowd behaviours and analysing human behaviours even facial expressions, to help abnormal event detection in crowd.

Get Help With Your Essay
If you need assistance with writing your essay, our professional essay writing service is here to help!
Essay Writing Service

As the worldwide economy rapidly growth, people and governments start to concerns about public security since some breaking events that cause critical harm to society and brings loss of lives and property. Therefore, governments spend an enormous sum of recourses to take precautions about such events. At the present, and the past few years, mostly the major precaution was setting up wide ranges of Closed-Circuit-Televisions (known as CCTV) in crowded locations, such as squares, subway stations, and crowded streets. In the case that was mentioned above which lacks of ‘intelligence’ and costly on human recourses due to security people are supposed to operate and monitor manually, and the high workload or some cursoriness may cause the delay to corresponding responses when abnormal event happens, such as security staff gets tired or change shifts. Abnormal events are predictable and there are precursors of them, for instance, multiple London bombings in 2005, witnesses mentioned that bomb attacker was nervous and carrying an opening backpack and muttering to himself, the expressions of bomb attacker proved that the importance of the human behaviour analysis model. Under this circumstance, how to detect abnormal events efficiently in videos and real-time monitoring automatically in an intelligent way is one of major research topics and challenges in the area of computer vision intelligence.

The aim of abnormal crowd event detection is to analyse crowd motions and human activities to build mathematical models by sociology and human behavioural theories. There are three main approaches for modelling human crowds as Mehran has mentioned in his literature (Mehran, Oyama, & Shah, 2009):

(1)        Microscopic approach, which defines pedestrians’ motivation in movement and treats crowd behaviours as a result of a self-organization process.

(2)        Macroscopic approach, which focuses mainly on goal-oriented crowds. In this approach, a set of group-habits is determined based on the goals and destinations of the scene.

(3)        Hybrid approach. Which inherit from macroscopic models as well as microscopic ones.

Common challenges of conventional methods for abnormal crowd event detection is low resolution, clutter (too many people), interactions between pedestrians, and pedestrians may block the view. The solutions to these problems had been mentioned in literature, social force model will be the one, this model is capable to simulate crowd behaviours and determine the human activities by social force, due to the patterns of social forces are different in different scenes.

The analyzations about crowd behaviours was a very popular topic in computer vision area and it still is in current research trends. For how to understand and analysis the crowd behaviour, there are two mean approaches: object-based approach, which will do the segment the scene into individuals (P, et al., 2008). Another approach is the holistic approach which considers the crowd as an entity. In nutshell, the appropriate approach for modelling human crowds must be confirmed before choosing the appropriate approach to understand and analysis crowd behaviours.

This report focuses on abnormal crowd behavior detection using the social force model and the modified social force model, and explore the strengthes of the modified social force model, and performances while analysing human crowd. In section 1, the background and the status of current research progress has been introduced. In section 2, the review of literature and related work will be introduced, meanly the fundation researches of the literature will be introduced. In section 3, the introduction and describution to the generalized method and modified method will be given. In section 4, the analysis results for this topic will be introduced, the comparisions between generalized SFM (Social Force Model) and modified SFM, and the comparisions between using force flow and using pure optical flow will be provided, the expirements based on sample datasets and real cases will be given as well. In section 5 it covers the envision of future analysis and research trends, and some personal thoughts.

According to the literature, the scene modelling techniques had been used to capture features from the behaviour of pedestrians and traffic respectively, but not tracking individual objects, which are individual pedestrians. The difference between crowd behaviour detection and individual object detection is no individual pedestrians can be properly segmented in the image. (Reisman, Mano, Avidan, & Shashua, 2004). The paths of each individual pedestrian can be irregular and same to backgrounds, so it is hard to distinguish pedestrians and other things.

There is a schema for real-time crowd behaviour detection based on spatiotemporal theory, precisely the spatiotemporal of a video sequence. The algorithm is based on slices in the spatio-temporal domain. An image is a single intensity scan-line collected over several frames is representation is rich in information, easy to obtain and the relevant information can be calculated in real-time. (Reisman, Mano, Avidan, & Shashua, 2004). Therefore, when a group of pedestrians (crowd) is moving to irregular direction or velocity, intersecting lines in spatiotemporal will be generated.

The crowd behaviour detection and analysis is an active research topic, there are several complex models existing, for instance, the applications of switching linear models are numerous including human motion modelling (Antoni & Vasconcelos, 2005). This model is motivated to track multiple objects such as crowd by using a vector of sensors, and it is related to the dynamic texture mixture. And the method by using conformal mapping provides us a convenient way to solve problems with any pedestrian flow density and velocity of them, and if there is no necessary for analytical solution (common in practice), directions of motion of pedestrians can be determined rapidly. The present study provides a body of exact solutions, or the techniques to generate them, which can be used to validate a numerical model. The method described here is a valuable tool for understanding, as opposed to merely simulating, pedestrian flows of a single type of pedestrian. It is appealing in its graphic nature. It is indeed remarkable that conformal mapping holds for the highly non-linear, time-dependent equations for pedestrian flow. It is even more remarkable that it can be used to study such flows at both medium and high densities and to study the behaviour of any discontinuity between them. (Hughes, 2002).

In previous section it had mentioned that the social force model to simulate crowd behaviours and to determine the human activities by social force, the social force model was invented to compute and different forces and represent them in expressions, estimate the social force parameters to create a model of likely behaviours in the crowd (Mehran, Oyama, & Shah, 2009). Due to the patterns of social force are different in different scenes, this section will cover further details about social force model and the modification of the social force model which was mentioned in Mehran’s literature.

3.1 SFM (Social Force Model)

In general, social force model represents people’s different motivations and influence by expressions of different forces based on Newtonian mechanics, as Mehran has stated that for each individual object, the actual behaviours/motions are influenced by 2 factors, personal motivations, and environmental constraints (Mehran, Oyama, & Shah, 2009).

Many people have the feeling that human behaviour is ‘chaotic’ or at least very irregular and not predictable. This is probably true for behaviours that are found in complex situations. However, at least for relatively simple situations, stochastic behavioural models may be developed if one restricts to the description of behavioural probabilities that can be found in a huge population (group) of individuals. (Helbing & Molnár, Social force model for pedestrian dynamics, 1995). Pedestrians are considered as subjects to long-ranged forces, and their dynamics will follow the equation of motion, the concept of it is similar to Newtonian mechanics. The relationship between the velocity of each individual object and the velocity of global entity (crowd)is complementing each other, which means individual objects are influenced by the global entity. The velocity of single pedestrian also can be described as the result of a personal desire force and interaction forces.

In the social force model, the changes in velocity will be expressed as
vi
, and accelerationof pedestrian (
i
) with mass (
mi
), and the expression will be:
midvidt=Fa=Fp+Fint

In the expression above, Fa represents the actual force, and due to individualistic goals or environmental constraints. This force consists of two main parts: (1) personal desire force
Fp
, and (2) interaction force
Fint
(Mehran, Oyama, & Shah, 2009). The way for tracking all objects in the crowd was blocking the interaction force at a certain, large enough distance around 5 meters and book-keeping of neighbouring pedestrians. And in this way the simulation of 30,000 even more in real-time is no problem. (Johansson, Helbing, & Shukla, 2007)

As known that normally behaviours/motions of people are purposefully or motivational, thus
vip
to express the desired velocity of the pedestrian. Meanwhile, it has been mentioned above the motion of individual object (pedestrian) is influenced or guided by the crowd, therefore, the actual velocity
vi
will be different from
vip
(desired velocity), so every individual object will have intention to achieve
vip
(desired velocity), and this intention can be considered as
Fp
which represents the personal desire force, the expression of
Fp
will be:
Fp=1τ(vip–vi)

In the expression above, τ is the relaxation parameter,
vi
is the actual velocity,
vip
is the desired velocity.

The social force model considers the influence that caused by events like panic, such as collective motion: runaway from a dangerous environment, therefore the personal desired velocity
vip
will be replaced by the expression below:
viq=1–pivip+pi⟨vic⟩

In the expression above,
pi
is the weight parameter of panic,
⟨vic⟩
is the average velocity of pedestrians around. When
pi→0
represents individual behaviour,
pi→1
represents crowd behaviour, thus the social force model can be considered as the expression below:
midvidt=Fa=1τviq–vi+Fint

3.2 The modification SFM

3.2.1 Explanation to Modification SFM

The modification SFM (Social Force Model) is about to use the approach ‘particle advection’ to represent the motions of the crowd. The modification version is for avoiding track every single pedestrian, the particle advection can be described as motions leaves that are floating in water. This approach distributes particles on the image with uniform level of distribution, then take the Gaussian weighted average of the optical flow that surrounding the particle as the average velocity of this particle. In this case for floating leaves, when obstacles appear, the velocity of leaf will be impacted.

 

The modification method based on social force model aimed to prevent dynamic occlusions, extensive clutter et cetera. in crowd with high density, the particle advection method is used to analyse videos. To perform the particle advection method, a grid of particles is launched over the first mean-field operations will be made on moving particles instead of individual pedestrians. The Lagrangian trajectory corresponding to a particle at grid location is computed by numerically solving the ordinary differential equations. The fourth-order Runge-Kutta-Fehlberg algorithm along with cubic interpolation of the velocity field to solve this system. The above process is repeated for each block of the video. (Ali & Shah, 2007).

3.2.2 Modification in algorithms

Since the particles are considered as individuals from the crowd (Mehran, Oyama, & Shah, 2009), the algorithm is supposed to be modified as well as below:

If the position of a particle is:
(xi, yi)

The actual velocity of particle
i
will be:
vi=Oave(xi, yi)

In this case,
Oave(xi, yi)
expresses the average of the optical flow of particle
i
which has the position as
(xi, yi)
.

The desired velocity of particle
i
will be:
viq=1–piOxi, yi+piOave(xi, yi)

In the equation above,
Oxi, yi
is the optical flow that surrounding the particle
i
. Two different values ‘the average of the optical flow of particle
i
’ and ‘the optical flow that surrounding the particle
i
’ are for identifying the individual behaviours and crowd behaviours.

And in this modified SFM, the difference between expected velocity and actual velocity are influenced by the surrounding particles and the environment. Thus, the expression of interaction force will be as shown, if
mi
= 1:
Fint=1τviq–vi–dvidt

In general, same as social force model, once obstacles appear, crowd branches or another crowd joins, the velocity of individual particle will be different to average velocity, which means the desired velocity and the direction of flow field will be different. And after this modification to SFM, particles will move as the velocity of the crowd.

For event detection, suppose videos without abnormal events are used to train LDA (Latent Dirichlet Allocation), and a group of scenes and a set of their normal force flows were given, the corpus and distribution of underlying topics for normal crowd behaviour will be constructed. Then the modified EM (Expectation-Maximization) algorithm in ‘Journal of Machine Learning Research’ will maximize the possibility of corpus (M. Blei,, Y. Ng, & I. Jordan, 2003), and analyse the possibility of every clip of video sequence, after all determine frames are normal or abnormal by using threshold. (Mehran, Oyama, & Shah, 2009)

4.1 Comparison between SFM (Social Force Model) and Modified SFM

Previous sections have covered the generalized social force model and the social force model modified by Mehran. The previous social force model already has a high degree of accuracy with simulating the crowd dynamics, therefore for Mehran concludes that abnormal social force in the social force model represents abnormal events.

The social force is still parameter for simulating possible behaviours in the crowd, but for modification, Mehran gets rid of tracking individual objects, instead of using the particle advection method to represent the motions of the crowd, the theory about ‘leaves float’ describes this modification precisely.

The idea about the difference between generalized SFM (Social Force Model) and Modified SFM is in the crowd with a high level of density, it becomes extremely hard to track individuals precisely, but the behaviours and motions of individuals are restricted by the crowd, thus individuals in the crowd will be interaction particles while estimating the
Fint
(Interaction Force). Without tracking an individual object, the modified model works robust and precise, meanwhile the continuity of the crowd flow that was captured by particle advection cannot be captured by optical flow et cetera.

Compare the algorithms between generalized SFM with modified SFM again, the modified SFM has redefined the expressions of actual velocity and desired velocity, particles will move as the velocity of the crowd. It is hard to compute the interaction force in high-density crowd such as subway stations in rush hour due to crowded and occlusions, the modified version with particle advection offers a possible way to solve described issues.

As known that that is not enough if only the interaction force was computed. In modified SFM the feature matrix of force flow
Sf(t)
will be computed by stream of frames
I(t)
. After that the ‘bag of words’ method will do the organize features and do the learning work, the modified SFM uses LDA (Latent Dirichlet Allocation) and EM (Expectation-Maximization) algorithm to do the rest of works and distinguish normal frames and abnormal frames. These modifications can provide the area with higher degree of interaction behaviours by seeing the force flow ‘strong’ or not, it has proved that in his experiments, the modified SFM is capable to show the region of abnormal area instead of label individuals. From the above, the modified social force model was surely improved.

4.2 Comparison between proposed social force model and pure optical flow

Mehran has the proposed social force method on a public dataset of normal and abnormal crowd videos from University of Minnesota, the dataset contains 11 scenarios of escaping from event in 3 different scenes (Indoor/Outdoor). As introduced in last section, the method labelled frames to normal or abnormal. The result shows the proposed model is able to detect abnormal event in not-trained scenes, which means the proposed method can simulate the abnormal event instead of concerns environmental features.

The experiment also has used optical flow to detect abnormal events, the Spatio-temporal patches of the optical flow were used as visual words, and the LDA model was learned based on optical flow information. The same dataset and parameters were given. The experiment has used ROC (Receiver Operating Characteristic, as known as sensitivity curve) curves as the performance evaluation standard, ROC curves show the sensitivity to the same signal from spots on ROC curves, and basically the ROC curve is the combination of confusion matrixes. And in this experiment, ROC curves will be used to determine the capacity of methods with the AUC (Area Under Curve). For AUC, the area should be between 0.1 to 1, and the value equivalently visualized the capacity of each method, obviously people prefer higher values.

Figure 1, (Mehran, Oyama, & Shah, 2009)

According to the experiment result, figure 1 above shows the ROC curves for proposed method and pure optical flow method, meanwhile shows the performances for detecting abnormal events. The AUC values for proposed SFM method (Red curve) and pure optical flow method (Blue curve) are 0.96 and 0.84 respectively. Therefore, according to values of AUC and figure 1, the proposed SFM was performing better than pure optical way obviously.

Another experiment by using web datasets has been done and it was aiming to test the method and do the comparison again base on real cases. Multiple scenes of different events such as marathon running and normal street view for normal scenes, and scenes of street fights, people escape in panic for abnormal scenes.

Figure 2,  (Mehran, Oyama, & Shah, 2009)

The ROC curves (Figure 2) above shows the performances of proposed SFM (Red curve) and pure optical flow (Blue curve) in real cases, the experiment was tested 10 times, and figure 2 shows the average of the results. According to figure 2 above the performances of two experimented methods can be barely evaluated, the modified SFM remains more precise than pure optical flow method. The AUC values of performances of proposed SFM (Red curve) and pure optical flow (Blue curve) are 0.73 and 0.66 respectively.

Regarding the results from 2 experiments which were based on both sample datasets and real cases, with respect, the performance of the modified SFM is better than pure optical flow method. In last subsection, the proposed method (Modified Social Force Model) was surely improved based on the generalized social force model theoretically, and it did perform better than generalized SFM. In this subsection, ROC figures and AUC had proved the modified SFM performed better than pure optical method, therefore the modification made by Mehran was distinct.

According to the literature about reviewing the research trends and methods, it tells that the degree of the popularity of crowd behaviour analysis is high in computer vision area globally (Zhan, Monekosso, Remagnino, Velastin, & Xu, 2008).

According to the research, the vision intelligence about crowd behaviour analysis and event detection is normally in 4 main factors, object detection, object classification, object tracking, and behaviour understanding. The behaviour understanding is now the highest level in these four factors, in this report, the behaviour understanding will be a hot topic in future analysis trends.

Moreover, researchers and scholars have done pretty much contributions in abnormal event detection, but the performances of currently existing models and algorithms are still yet to be improved while, especially for handling crowds with high density and occlusions.

In recent years, some methods are designed to analysis human motivations and understand human behaviours, in order to detect abnormal events mainly for public security concerns, once the abnormal event was detected, the abnormal area even individual person can be reported, after manual confirmations the actions will be taken. Thus, within the scope in this report how to increase the precision accuracy of event detection models for crowd will be important, and models are supposed to replace manual works as much as possible.

According to the modifications to social force model that was made by Helbing, the necessity and importance of quoting psychological theories have been proved (Helbing, Farkas, & Vicsek, Simulating dynamical features, 2000). Psychological theories and socio-psychological have capabilities to improve existing models, for instance, the social force model, these theories provide opportunities for models to personating the simulations to the human crowd/individual behaviours.

In addition, the micro-psychology and face detection will be future analysis trend possibly. The results from micro-psychology research illustrate human motivations can be detected by micro-expressions, this research result can be applied to abnormal event detections but mostly face detections at present. Meanwhile, researches related to analysing human emotions and motivations are in progress, and these researches included detections based on facial expressions, head postures, body postures, and human gaits. These researches are capable to remodify the estimation standard for detecting individual behaviours and expressions and their acting forces.

In future research trends about the human crowd and behaviours, the research related to physiology characteristics and phycological characteristics of individuals (Pedestrians) will be crucial as well. For physiology, age differences and gender differences will impact the forces. For phycological, conformity and panic, et cetera will impact the individual behaviours even crowd behaviours.

In general, the future trends for research in abnormal event detection will possibly be integrated other subjects, for instance, micro-psychology, socio-psychological, human behaviouristic, even religious studies. The research trend will be the focus on anthropomorphism or personification. Furthermore, most of existing abnormal event detection models are not capable to process in real-time or online due to heavy time cost, meanwhile the patterns for normal events or abnormal events are not constants, which means the offline trained methods will be eliminated in someday. In existing models, the assumption was  For future analysis trend about abnormal event detection in human crowd, the algorithms and models that are self-adaptive with enough processing speed will be focused.  

This report has covered various classic works of literature and summarized value of them. The abnormal event detection in crowd is a meaningful and valuable topic in computer vision intelligence, and related techniques help people and society about better understanding human behaviours based on models and algorithms. In this report the comprehensive comparisons were given, the modified SFM has made improvements on algorithms. The result from experiments also showed the strength and how robust it is rather than optical flow methods. This report has introduced a better model for handling event detection in crowd with evidences. Furthermore, the modified model has proved the outstanding performance of force flow methods, which will provide the theory and experiments supplies for future analysis and researches about event detection in crowd, also as an instance for future modifications based on SFM, or any other models.

According to literatures about event detecting and human behaviours/expressions analysis, the importance of integrating subjects is remarkable for instance, Social Force Model. The future techniques that relate to event detection and crowd behaviour analysis will focus on resistance to noises and interferences, automatically detections, techniques will make great contributions to security of societies globally.

Ali, S., & Shah, M. (2007). A Lagrangian Particle Dynamics Approach for Crowd Flow Segmentation and Stability Analysis. Computer Vision and Pattern Recognition Computer Vision and Pattern Recognition, 2007. CVPR ’07 (pp. 1–6). IEEE.

Antoni, B. C., & Vasconcelos, N. (2005). Mixtures of dynamic textures. Tenth IEEE International Conference on Computer (pp. 641–647). Washington, DC: IEEE.

Helbing, D., & Molnár, P. (1995, May 1). Social force model for pedestrian dynamics. Physical Review, p. E 51:4282.

Helbing, D., Farkas, I., & Vicsek, T. (2000, 9 28). Simulating dynamical features. nature, pp. 487-490.

Hughes, R. L. (2002). A continuum theory for the flow of pedestrians. In Transportation Research Part B: Methodological,36(6) (pp. 507-535). Elsevier.

Johansson, A., Helbing, D., & Shukla, P. K. (2007). Specification of the social force pedestrian model by evolutionary adjustment to video tracking data. In Advances in Complex Systems (pp. 271-288). World Scientific Publishing.

M. Blei,, D., Y. Ng, A., & I. Jordan, M. (2003). Latent Dirichlet Allocation. In Journal of Machine Learning Research (pp. 993-1022).

Mehran, R., Oyama, A., & Shah, M. (2009). Abnormal crowd behavior detection using social force model. 2009 IEEE Conference on Computer Vision and Pattern Recognition (pp. 935-942). Miami, FL, USA: IEEE.

P, T., T, S., G, D., N, K., J, R., & T, Y. (2008). Unified Crowd Segmentation. In Computer Vision – ECCV 2008. Springer, Berlin, Heidelberg: Springer, Berlin, Heidelberg.

Reisman, P., Mano, O., Avidan, S., & Shashua, A. (2004). Crowd detection in video sequences. 2004 IEEE Intelligent Vehicles Symposium (pp. 66-71). University of Parma,Parma, Italy: IEEE.

Zhan, B., Monekosso, D. N., Remagnino, P., Velastin, S. A., & Xu, L.-Q. (2008). Crowd analysis: a survey. In Machine Vision and Applications (pp. 19(5-6)).