Improved Optimization and Speed up in Big Stream Data Processing
Main Article Content
Abstract
To provide the low latency, higher throughput & speedup in stream processing, there should be systematic flow design which can accept the continuous incoming stream and provide it to the different operators to work parallel on the incoming stream. In this paper, we have implemented the proposed pipeline and watermark on Apache Beam along with google cloud dataflow as a runner. The experiments have been carried on stock market dataset, by considering the prices of oil, us dollar and gold as essential dependent parameters. Result of experiments proved that there is a relationship exist between the stock price and those dependent parameters. Now as the prediction of stock market essentially required other dependent parameter to be present which are originally from distributed environment, any parameter delay affect the result of prediction and introduced the good optimization and low latency. To implement the effective stream processing, we have used pipeline and watermark concept to handle and reduced any such delay and increase the speedup in big data stream processing.
Downloads
Metrics
Article Details
You are free to:
- Share — copy and redistribute the material in any medium or format for any purpose, even commercially.
- Adapt — remix, transform, and build upon the material for any purpose, even commercially.
- The licensor cannot revoke these freedoms as long as you follow the license terms.
Under the following terms:
- Attribution — You must give appropriate credit , provide a link to the license, and indicate if changes were made . You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
- No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
Notices:
You do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation .
No warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material.
