Discuss about the Big Data challenges in IoT and Cloud.
The evolution of technology has increased the volume of data generation. The large volumes of data are generated majorly due to the use of cloud and IOT environment. It is the biggest challenges to manage the security and privacy of the big data. The IT companies are looking forward to analyse the problem domain of big data challenges due to the tremendous growth in IOT and cloud computing services to develop new framework for managing security, integrity, accuracy, and reliability of the big data (Zaslavsky, and at.el., 2012). The decision making capabilities of the user is degrading due to the availability of large data because it is a difficult tasks to extract meaningful information from the voluminous data available on the Cloud and IOT environment. The big data is defined on the basis of five parameters which are classified as Volume, velocity, variety, veracity, and Value. The following diagram shows the structured view of 5V parameters of the big data:
The volume of the big data refers to the large storage capacity is required for handling the large data set of the IOT and Cloud application which is generated in terabytes in the form of records, transactions, tables, texts, and images. The velocity of generating the big data is increasing at a very fast rate due to the excessive use of real time analytics, streams of processes, and batch processing system. The value of the application depends on the statistical data being driven according to the query placed for fetching the data. The big data is available in large variety such as audio, video, text, images, social media platform, emails, and documents. Veracity is the accuracy and quality of the data which is used by the IOT and cloud application.
The generation of big data due to the inclusion of IOT and cloud architecture in developing intelligent system for the health care centre, parking system, e-commerce website, trading through online parameters, waste management system, smart cities, and others increases the complexities in managing the privacy and security of the data. The deployment of hardware, software, and network devices on the cloud increases pressure of data management. In this paper, we will focus on various challenges which are faced with the big data management system in IOT and cloud architecture.
The literature review was conducted for analysing the growth of big data and increasing challenges which are faced in the data management with the effective development in the field of IOT and cloud. The excessive growth in the advanced technology increases tremendous pressure on managing the big data for easy accessing and retrieval of the information.
Sources of Big data
There are various sources for increasing the big data volume such as development of the smart city, intelligent system for health care and waste management system, organizing home care centre for the patients, organizing systematic parking architecture, complexity in managing sensor data, and others (Ranjan, 2017). The big data can be collected from numerous platforms in the heterogeneous manner such as data sources, content format, data stores, data staging, and data processing. The following chart shows the different sources of big data.
Cloud computing and Big Data:
Cloud computing and big data are the two sides of the same coin. The increasing growth in the adoption of cloud computing services and data storage units by the organization results into the increasing growth of big data. The fault tolerance capability of the cloud architecture can be effectively resolved with the use of big data. The development of effective visualization of the data helps in increasing the decision making capabilities of the users. The complexity of the data set increases with the heterogeneity in the database. The correlation in the big data complexity requires architectural changes in the big data environment. The following diagram shows the use of big data in the cloud environment:
The clustering of the database can affect the storage capability of the database and network communication among the cloud devices. The availability of the data can be affected by the expansion of the cloud units (Satyanarayana, 2015). The cloud storage system helps in improving the on-demand accessing of services from the small and medium sized organization. The Hadoop file structure is used by the cloud architecture for storing the data on the cloud. It helps in improving the accessibility of data through the inclusion of map reduce practices. The map-reduce technique helps in accelerating the storage of big data through the cloud devices. The on-demand accessing of the information can be effectively resolved with the inclusion of NoSQL database, Hadoop file structure, and map reduce technique for storing big data.
IOT and Big Data
The organizations are moving towards the inclusion of artificial intelligence in the working architecture of the enterprise. The growth of artificial intelligence and sensor network increases the overloading of the data on the premises of data storage which results in increasing the complexities of the project (Sabarmathi, Chinnaiyan, and Ilango, 2016). The big data is the structured and unstructured format used for storing the data on the network which can occur due to the availability of different sources. The internet of thing is the operational platform for bringing sensor networks in the work practices for increasing artificial intelligence of the system. The big data is generated due to the inclusion of IOT and cloud architecture in developing intelligent system for the health care centre, parking system, e-commerce website, trading through online parameters, waste management system, smart cities, and others increases the complexities in managing the privacy and security of the data (Yang, at.el., 2013). The disruptive technology should be used for managing the big data of the real time system. The use of big data analytical tools is used for managing the data of the IOT architecture.
Big data challenges in IOT and Cloud
The rapid growth of big data results in many challenges and concerns which should be taken under consideration for the effective functionality of smart applications and cloud computing services. The heterogeneous data is collected from various sources which increase the complexity for managing it on the database. In the database, the data is stored in the proper format and data types (Schenker, 2015). The inclusion of non-traditional data types increases the complexities for the entire system. Some of the biggest challenges in the management of big data through the inclusive growth of IOT and cloud computing services are listed below:
|
Big Data Challenges |
Description |
|
Security issues |
The security is the major concern for the data management because the inefficiency of securing the data on the cloud and IOT architecture can results in the loss of data confidentiality and accuracy. The unauthorised accessing of the information from the cloud and IOT environment can affect the security parameters used for preserving the confidentiality and integrity of the information. The confidential information of the customers and users should be kept secured on the cloud database (Schroeder, Meyer and Taylor, 2013). The loss of confidential data can result in the big hazard to the users such as misuse of information, Illegal use of information, and theft from bank accounts, and other |
|
Privacy issues |
The security and privacy are the two sides of the same coin. Damage to security architecture of the big data can results in the loss of data privacy. The sensitivity of the information should be kept private so that unauthorised accessing of the information can be reduced and minimised. |
|
Heterogeneous data |
The heterogeneity in the variety of data available increases the complexities of the data storage. The variety of big data increases the pressure of managing the data on the database. The complexity in the data representation and management increases with the increase in heterogeneous variety of available data. |
|
Unauthorised accessing of data |
The unauthorised accessing and un-authentication of data can modify and manipulated the information stored on the database. The protection mechanism should be used for preserving the data from the modification and manipulation. |
|
Transferring of data on the cloud |
The complexity in transferring data on the cloud due to the increasing amount of data (Zheng, and at.el., 2013) |
|
Availability of sensor data |
The sensor data requires extra parameters for storing it into database of IOT and cloud architecture. |
|
Consistency of the system |
The consistency of the system can be exploited due to the technological advancement in the availability of the data. The big data have to be stored in different parameters which are the major hazards in managing the consistency of data storage system. |
|
Data transparency |
It is difficult for managing the transparency of the data due to the increase growth of data availability on the IOT and cloud platforms |
|
Data Configuration |
The re-configuration of the database architecture is required for managing the complexities of the increasing data (Zanoon, Haj, Khwaldeh, 2017). |
|
Data governance plan |
It is difficult to manage data governance plan with the big data environment because the data is available in different formats therefore it is difficult for applying same principles on every data |
|
Architectural issues |
The intelligent system requires an effective database which can handle the sensor data and information effectively for retrieving the information instantaneously in the real time environment. |
|
Issues in Innovation |
The growth in the innovation and technological advancement give rise to the excessive growth of big data in the IOT and cloud environment which is the major hazard for the management process of the data. |
|
Technical issues |
The complexity arises with the management of data transfer and storage in the heterogeneous environment of data storage. It is the complicated tasks for retrieving the data from different architectures. The automation in the services can increases the security and privacy issues with the data transfer. |
|
Hardware issues |
The deployment of new hardware increases the cost of the IOT architecture and complexity in managing interconnection with the new architecture and hardware devices. |
|
Standard issues |
The standard policy framework is not developed for managing the complexities of the big data in the IOT and cloud environment. The difference in the data types of the information collected from different sources increases the complexity in handling it. |
|
Organizational Issues |
The inclusion of social network in the organization increases the wastage of crucial time of the employees. The productivity of the organization can be affected. It is difficult for managing the big data of social network of the organization |
|
Data processing issues |
It is a complex tasks for managing the integration between the data generated from the different sources. The redundancy of the data stored can be increased. |
|
Heterogeneity in data processing |
In the database, the data is stored in the proper format and data types. The inclusion of non-traditional data types increases the complexities for the entire system. The classification of the data is the biggest challenge in the availability of the heterogeneous data. The data fusion methodology should be used for managing the heterogeneity of the data (Neves, 2016). |
There are numerous technologies and methodologies available for resolving the challenges faced in the management of big data associated with the IOT and cloud architecture. Some of them are discussed below
Big data analytics and tools:
The big data analytics and tools are the most effective solution for managing the complexities of the big data which is associated with the IOT architecture and cloud computing services (Marjani, 2017). The following diagram shows the big data analytics architecture used for managing the big data of the enterprises:
There are different tools of big data analytics which are used for handling the complexity and challenges associated with the management process of big data such as confidentiality, security, reliability, accuracy, and integrity of the information (Gholami, and Laure, 2016).
The big data is stored on the cloud which helps in increasing the availability of the data and improves the distribution of resources among the participating units. The management of big data on the cloud environment requires technological platform which results in effective management of security and privacy of the cloud big data. Some of the methodologies which are used by the cloud environment are stated below:
The Semantic web server is used for gathering and collecting data from various sources. The data can be collected in the homogenous and heterogeneous forms according to the requirement of the user. It helps in preserving the privacy and security of the information in the collecting process from various address space. It is based on the principles of data mining for fetching the relevant information from the data warehouse services. The intensive amount of big data is generated through the inclusion of semantic servers in the curriculum of the IOT architecture.
Fusion of data
Fusion of data is the multidisciplinary approach for managing the data generated from different sources and effectively classifying them under appropriate parameters. This methodology for managing the big data accumulated on the premises of the IOT and cloud architecture follows some guidelines and principles for managing it which are stated as below:
Middleware:
The middleware methodology is used for managing the interconnection between the services and the applications used in the heterogeneous environment of IOT devices (Ahmed, and at.el., 2017). The communication between the IOT devices and cloud resources and devices is effectively managed with the implementation of middleware. Some of the middleware and its relative examples are listed in the table below:
|
Type of Middleware |
Examples |
|
Middleware based on Message orientation |
MOM MQ JSM ESB |
|
Middleware based on transaction layer |
TPM Tuxedo |
|
End to End Middleware |
JXTA |
|
Mobile computing middleware |
OSA Parlay JAIN OMA |
|
Middleware based on GRID |
PVM MPI Schedulers |
|
Middleware based on RFID edge |
OAT system Sybase Oracle Tibco SeeBeyond IBM SAP Connectera Global Ranger |
|
CORBA middleware in real time analytics |
Real time CORBA |
|
Middleware based on process orientation |
Webmethods Tibco SeeBeyond IBM SAP |
Some of the proposed middleware which can be used for managing the big data of the IOT and cloud architecture are summarised below:
The following diagram shows the deployment of middleware in the IOT architecture for managing the interconnection and integration for the flow of information securely among the IOT devices
|
Comparison |
Big Data analytics and tools |
Cloud Computing Methodology |
Semantic Web Server |
Fusion of Data |
Middleware |
|
Definition |
The big data analytics and tools are the most effective solution for managing the complexities of the big data which is associated with the IOT architecture and cloud computing services. |
The big data is stored on the cloud which helps in increasing the availability of the data and improves the distribution of resources among the participating units. The management of big data on the cloud environment requires technological platform which results in effective management of security and privacy of the cloud big data. |
In the semantic web server environment, the data can be collected in the homogenous and heterogeneous forms according to the requirement of the user (Babu, and Koushik, 2017). It helps in preserving the privacy and security of the information in the collecting process from various address space. |
Fusion of data is the multidisciplinary approach for managing the data generated from different sources and effectively classifying them under appropriate parameters (Badiel, 2012). |
The middleware methodology is used for managing the interconnection between the services and the applications used in the heterogeneous environment of IOT devices. |
|
Efficiency |
It is efficient in managing the synchronised format by making use of Map reduce technology |
The security and privacy of the information can be effectively managed by making use of hash value algorithm and other cryptographic procedures such attributed based protocols, encryption technologies, and many more. |
The sensor data of the IOT environment can be effectively managed on the semantic web server |
The efficiency in the classification of heterogeneous data can be improved with the deployment of multidisciplinary approach of data fusion |
The web services can effectively managed the flow of information in managing the resources on the IOT architecture through the common platform of Middleware. |
|
Simplicity |
It is simple to use and implement |
It is simple to use and implement |
It is complex task for storing and retrieving sensor data |
It is complex task for storing and retrieving heterogeneous data |
It is simple to implement in the IOT architecture for managing integration of IOT devices |
|
Extension into other application |
Extensible |
Extensible |
Inextensible |
Extensible |
Inextensible |
|
Time Saving |
Time Saving |
Time Saving |
Time Saving |
Time Consuming |
Time Consuming |
|
Cost |
Low |
Low |
High |
High |
Low |
|
Feasibility |
Feasible |
Feasible |
Infeasible |
Infeasible |
Feasible |
|
Connectivity |
Yes |
Yes |
Yes |
Yes |
Yes |
|
Commercialization Issues |
It is difficult for implementing the big data analytics in the current working structure of the enterprise |
It can be easily implemented in the working architecture of the enterprise with minimum cost |
It is difficult for implementing the Semantic web server in the current working structure of the enterprise |
It is difficult for implementing the Data fusion in the current working structure of the enteprise |
It can be easily implemented in the working architecture of the enterprise with much higher cost than cloud environment |
The big data analytics and tools is the best suitable methodology for accompanied with the management of big data of IOT and cloud environment. The Hadoop file structure is used for storing the information in the data nodes on the distributed file structure for handling the scalability of the IOT and cloud architecture. The clustering of the IOT devices known as Hadoop cluster helps in managing the real time data of the intelligent system.
Working of Hadoop distributed file structure
The HDFS environment is used for transforming the information into different nodes located in the Hadoop clusters. It is based on the parameters of parallel processing for managing the flow of information. The fault tolerance capability of the system can be improved with the help of HDFS implementation. The replication of the copies is maintained on the data nodes for the processing of required information according to the query generated by the users. The master slave architecture is followed in HDFS for managing the data operation. The methodology helps in retrieving effective information from the large volumes of big data on the IOT and cloud architecture which results in increasing the decision making capabilities of the user. The Map reduce technology is the programming paradigm for storing the big data in the distributed file structure of Hadoop effectively. It helps in classifying the source of data in different parameters which increases the efficiency of the retrieval process.
The Hadoop distributed file structure and map reduce technology are best suitable methodology because it helps in securing the privacy and confidentiality of the big data. The management of the big data on the architecture of IOT and cloud can be done for faster retrieval of information as the demand rose by the user. The efficiency of interconnection and integration can effectively improve with the use of HDFS environment. The Hadoop file structure is advantageous for the organization in managing flow of information between IOT devices. The implementation of HDFS is capable of managing the scalability and expansion of the business environment effectively. The data generated from the new premises of the organization can be effectively handled by the HDFS environment. The functionality of the business based on cloud and intelligent system can be managed without any distraction and complexities. It is a cost effective platform for managing the confidentiality and privacy of the data set. The recipients of the information are provided with quality of service in managing accuracy of the information which helps in improving the decision making capability and productivity of the organization. The structured and unstructured data format of big data can be effectively handled by Hadoop. It is based on the principle of data mining for increasing flow of information between different units by retrieving data from the data warehouse. In the fraction of seconds, the processing of big data can be effectively done which increases the capability of faster retrieval of information. The fault tolerance capability of the IOT and cloud environment can be improved by replacing the data node with the replicated data node for managing the flow of information. The HDFS is featured with scalability and flexibility which helps in managing the big data through the expansion of the business units.
It has been researched that the rapid growth in the technological advancement will results in the growth rate of big data. The complexities of the big data are increasing with the rapid increase in the big data from different sources. The excessive growth in the advanced technology increases tremendous pressure on managing the big data for easy accessing and retrieval of the information. The effective tools should be used for managing the complexities of the big data so that the efficiency of the real time analytics and intelligent system can effectively improve and provide quality of service to the end users. The security and privacy are the two major concerns with the increasing growth of big data because it can affect the reliability and accuracy of the information which in turn affect the expected outcome of the intelligent system (Acharja, and Ahmed, 2016). The different methodologies are proposed for managing the confidentiality and integrity of the information provided to the end users. The security and privacy are the two sides of the same coin. Damage to security architecture of the big data can results in the loss of data privacy. The sensitivity of the information should be kept private so that unauthorised accessing of the information can be reduced and minimised. The Hadoop file structure associated with the map reduce technology helps in securing the big data from the unauthorised access so that it can eliminate the complexities of data modification and manipulation. It is recommended to use HDFS for preserving and managing the confidentiality, privacy and security of the big data in the IOT and cloud environment.
Conclusion
It can be concluded that the emphasis should be given to the big data challenges so that the effective big data management technologies and methodologies should be opted according to the requirement of the organization. The rapid growth of big data results in many challenges and concerns which should be taken under consideration for the effective functionality of smart applications and cloud computing services. The big data management methodologies help in preserving the privacy and security of the information in the collecting process from various address space. It is based on the principles of data mining for fetching the relevant information from the data warehouse services. The data can be collected in the homogenous and heterogeneous forms according to the requirement of the user. The data availability through the data mining helps in improving the decision making capabilities of the user because the limited amount of information is transferred to the user so that he can readily retrieve the information of its choice and take corrective action plan as required. The Hadoop file structure is used for storing the information in the data nodes on the distributed file structure for handling the scalability of the IOT and cloud architecture. The availability and security of the data can be effectively improved with the implementation of Hadoop distributed file structure.
References
Acharja, D., and Ahmed, K. (2016). A survey on big data analytics: Challenges, open research issues and tools. International Journal of advanced computer science and applications, 7(2). Retrieved from https://thesai.org/Downloads/Volume7No2/Paper_67-A_Survey_on_Big_Data_Analytics_Challenges.pdf
Ahmed, E., Yaqoob, I., Hashem, I., and Vasilakos, A. (2017). The role of big data analytics in internet of things. 1st ed. Retrieved from https://www.researchgate.net/publication/317617290_The_role_of_big_data_analytics_in_Internet_of_Things
Aly, H., Elmogy, E., Barakat, S. (2014). Big data on Internet of things: Application, Architecture, Technologies, Techniques, and future directions. 1st ed. Retrieved from https://www.ijcse.net/docs/IJCSE15-04-06-040.pdf
Babu, A., and Koushik, A. (2017). Comparative study on big data analytics framework in the cloud. International Journal of computer science and information technologies, 8(1). Retrieved from https://ijcsit.com/docs/Volume%208/vol8issue1/ijcsit2017080109.pdf
Badiel, A. (2012). IOT meets big data: Standardization considerations. 1st ed. Retrieved from https://nca.org.gh/assets/Uploads/Alidrssa-Badiel-IoT-meets-Big-Data.pdf
Cai, H., Xu, B., Jiang, L, and Vasilakos, A. (2014). IOT based big data storage systems in cloud computing: Perspectives and challenges. 1st ed. Retrieved from https://ieeexplore.ieee.org/document/7600359/
Cecchinel, C., Jimenez, M., Mosser, S., and Reveill, M. (2014). An architecture to support the collection of big data in the internet of things. 1st ed. Retrieved from https://www.i3s.unice.fr/~mosser/_media/research/umc14.pdf
Gholami, A. and Laure, E. (2016). Big data security and privacy issues in the cloud. 1st ed. Retrieved from https://aircconline.com/ijnsa/V8N1/8116ijnsa04.pdf
Hashem, I., Yaqoob, I., Anuar, N., Mokhtar, S., Gani, A., and Khan, S. (2015). The rise of big data on cloud computing: Review and open research issues. 1st ed. Retrieved from https://umexpert.um.edu.my/file/publication/00001293_109174.pdf
Marjani, M., Nasaruddin, F., Gani, A., Karim, A., Hashem, A., Siddiqa, A., and Yaqoob, I. (2017). Big IOT data analytics: Architecture, Opportunities, and open, research challenges. 1st ed. Retrieved from https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7888916
Neves, P., Schmerl, B., Bernardino, J., and Camara, J. (2016). Big data in cloud computing: Features and Issues. 1st ed. Retrieved from https://acme.able.cs.cmu.edu/pubs/uploads/pdf/IoTBD_2016_10.pdf
Ranjan, R., Thakker, D., Haller, A., Buyya, R. (2017). A note on exploration of IOT generated big data using semantics. 1st ed. Retrieved from https://www.cloudbus.org/papers/IoT-FGCS2017.pdf
Sabarmathi, G., Chinnaiyan, R., and Ilango, V. (2016). Big data analytics research opportunities and challenges: A review. 1st ed. Retrieved from https://ijarcsse.com/Before_August_2017/docs/papers/Volume_6/10_October2016/V6I10-0105.pdf
Satyanarayana, L. (2015). A survey on challenges and advantages in Big data. 1st ed. Retrieved from https://www.ijcst.com/vol62/1/24-LENKA-VENKATA-SATYANARAYANA.pdf
SChenker, N. (2015). Big data: Uses and Limitation. 1st ed. Retrieved from https://www.cdc.gov/nchs/data/bsc/bscpres_schenker_091913.pdf
Schroeder, R., Meyer E., and Taylor, L. (2013). Big data and the uses and disadvantages. 1st ed. Retrieved from https://m.sussex.ac.uk/webteam/gateway/file.php?name=schroeder-sussex-big-data.pdf&site=25
Sivakumar, S., Anuratha, V., Gunasekaran, S. (2017). Survey on Integration of cloud computing and internet of things using Application perspective. International Journal of emerging research in management and technology, 6(4). Retrieved from https://www.ermt.net/docs/papers/Volume_6/4_April2017/SV6N4-101.pdf
Yang, C., Huang, Q., Li, Z., Liu, K., and Hu, F. (2013). Big data and cloud computing: Innovation opportunities and challenges. 1st ed. Retrieved from https://www.tandfonline.com/doi/pdf/10.1080/17538947.2016.1239771
Zanoon, N., Haj, A., Khwaldeh, S. (2017). Cloud computing and big data is there a relation between the two: A Study. 1st ed. Retrieved from https://www.ripublication.com/ijaer17/ijaerv12n17_89.pdf
Zaslavsky, A., Perera, C., Georgakopolous, D. (2012). Sensing as a service and big data. 1st ed. Retrieved from https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.294.9753&rep=rep1&type=pdf
Zheng, Z., Wang, P., Liu, J., Sun, S. (2013). Real time big data processing framework: Challenges and solution. 1st ed. Retrieved from https://www.naturalspublishing.com/files/published/v6910010rnl56m.pdf
Essay Writing Service Features
Our Experience
No matter how complex your assignment is, we can find the right professional for your specific task. Contact Essay is an essay writing company that hires only the smartest minds to help you with your projects. Our expertise allows us to provide students with high-quality academic writing, editing & proofreading services.
Free Features
Free revision policy
$10Free bibliography & reference
$8Free title page
$8Free formatting
$8How Our Essay Writing Service Works
First, you will need to complete an order form. It's not difficult but, in case there is anything you find not to be clear, you may always call us so that we can guide you through it. On the order form, you will need to include some basic information concerning your order: subject, topic, number of pages, etc. We also encourage our clients to upload any relevant information or sources that will help.
Complete the order form
Once we have all the information and instructions that we need, we select the most suitable writer for your assignment. While everything seems to be clear, the writer, who has complete knowledge of the subject, may need clarification from you. It is at that point that you would receive a call or email from us.
Writer’s assignment
As soon as the writer has finished, it will be delivered both to the website and to your email address so that you will not miss it. If your deadline is close at hand, we will place a call to you to make sure that you receive the paper on time.
Completing the order and download