Home >> Publication
Sensors
2019³â 19±Ç 1È£ p.1 ~ p.18
Improvement of Kafka Streaming Using Partition and Multi-Threading in Big Data Environment
Improvement of Kafka Streaming Using Partition and Multi-Threading in Big Data Environment
Bunrong Leang
Sokchomrern Ean, Ga-Ae Ryu and Kwan-Hee Yoo
- Abstract -
The large amount of programmable logic controller (PLC) sensing data has rapidly
increased in the manufacturing environment. Therefore, a large data store is necessary for Big Data
platforms. In this paper, we propose a Hadoop ecosystem for the support of many features in the
manufacturing industry. In this ecosystem, Apache Hadoop and HBase are used as Big Data
storage and handle large scale data. In addition, Apache Kafka is used as a data streaming pipeline
which contains many configurations and properties that are used to make a better-designed
environment and a reliable system, such as Kafka offset and partition, which is used for program
scaling purposes. Moreover, Apache Spark closely works with Kafka consumers to create a
real-time processing and analysis of the data. Meanwhile, data security is applied in the data
transmission phase between the Kafka producers and consumers. Public-key cryptography is
performed as a security method which contains public and private keys. Additionally, the
public-key is located in the Kafka producer, and the private-key is stored in the Kafka consumer.
The integration of these above technologies will enhance the performance and accuracy of data
storing, processing, and securing in the manufacturing environment.
- Key Words -
Hadoop ecosystem, public-key cryptography, data processing, data streaming, real-time analysis, secured PLC sensing data
Hadoop ecosystem, public-key cryptography, data processing, data streaming, real-time analysis, secured PLC sensing data
- ³í¹® ±¸ºÐ -
±¹¿ÜÀü¹®ÇмúÁö, SCI
- »ç»ç Á¤º¸ -
This research was funded by [Ministry of Trade, Industry & Energy (MOTIE, Korea) under Industrial Technology Innovation Program, Development of intelligent operation system based on Big Data for production process efficiency and quality optimization in nonferrous metal industry] grand number [10082578].
- Downloads -
Improvement of Kafka Streaming using Partition and Multi-Threading in Big Data Environment.pdf