CSR: Small: Collaborative Research: Systematic Approaches for Real-Time Stream Data Services

Primary Investigators

• KD Kang (Lead PI), Associate Professor, Department of Computer Science, State University of New York at Binghamton

• Steve Goddard, Professor, Department of Computer Science and Engineering, University of Nebraska--Lincoln

Project Period: 9/1/2011 - 8/31/2015

Project Summary

Data-intensive real-time applications, including transportation management, military surveillance, and network monitoring, need to handle massive amounts of stream data in a timely fashion. It is challenging to support real-time stream data services (RTSDS) due to stringent timing constraints, potentially unbounded continuous stream data, bursty stream data arrivals, and workload variations due to data value changes. This project will develop cost-effective methods and a runtime system for RTSDS. The project will systematically investigate methods and tools to support real-time continuous queries for RTSDS even in the presence of dynamic workloads. Specifically, the project will study 1) real-time continuous query modeling, 2) new performance metric design, 3) adaptive query scheduling design, and 4) tardiness control and load shedding, for both single node and clustered RTSDS. The project will also have prototype implementation and testbed evaluations. The results and findings of this project will advance and seamlessly integrate real-time computing and stream data management.

Publications

Journals

• Y. Zhou, K. D. Kang, "Deadline Assignment and Feedback Control for Differentiated Real-Time Data Services", IEEE Transactions on Knowledge and Data Engineering, To Appear.
• J. Oh, K. D. Kang, "A Predictive-Reactive Method for Improving the Robustness of Real-Time Data Services," IEEE Transactions on Knowledge and Data Engineering, Volume 25, Issue 5, pages 974 - 986, May, 2013.
• M. H. Suzer, K. D. Kang, C. Basaran, "Active Queue Management via Event-Driven Feedback Control," Computer Communications, Elsevier, Volume 35, Issue 4, Pages 517-529, February 2012. (closely related but already in press before the funding decision)
• K. Kapitanova, S. H. Son, K. D. Kang, "Using Fuzzy Logic for Robust Event Detection in Wireless Sensor Networks", Ad Hoc Networks, Elsevier, Volume 10, Issue 4, pages 709-722, June 2012. February 2012. (closely related but already in press before the funding decision)

Conference Papers

• Y. Wang, M. C. Vuran, S. Goddard, "Stochastic Performance Trade-offs in the Design of Real-time Wireless Sensor Networks, IEEE International Conference on Computing, Networking, and Communications (ICNC '15), Anaheim, California, USA, February 16-19, 2015.

• L. Chen, K. D. Kang, "A Framework for Real-Time Information Derivation from Big Sensor Data,", 12th IEEE International Conference on Embedded Software and Systems (ICESS '15), New York, USA, August 24 - 26, 2015.

• M. H. Suzer, K.D. Kang, "Predictive Thermal Control for Real-Time Video Decoding", In Proceedings of the Euromicro Conference on Real-Time Systems (ECRTS '14), Madrid, Spain, July 8-11, 2014.

• M. Xie, K.D. Kang, C. Basaran, "Moim: A Multi-GPU MapReduce Framework," In Proceedings of the International Symposium on MapReduce and Big Data Infrastructure, Dec. 3 - 5, 2013, Sydney, Australia.

• D. Driscall, K. D. Kang, "Dynamic Packet Size Adaptation for Efficient Bluetooth Communication," International Conference on Wireless Networks, July 22 - 25, 2013, Las Vegas, USA.

• S. Denman, K. D. Kang, "ACRE: A Method for Supporting Strong Consistency and Adaptivity in Replicated Data Storage," International Conference on Internet Computing and Big Data, July 22 - 25, 2013, Las Vegas, USA.

• C. Basaran, K. D. Kang, Y. Zhou, M. H. Suzer, "Adaptive Load Shedding via Fuzzy Control in Data Stream Management Systems," IEEE International Conference on Service Oriented Computing & Applications (SOCA '12), Dec. 17 - 19, 2012, Taipei, Taiwan.

• C. Basaran, K. D. Kang, "Supporting Preemptive Task Executions and Memory Copies in GPGPUs," In Proceedings of the 24th Euromicro Conference on Real-Time Systems (ECRTS '12), July 11 - 13, 2012, Pisa, Italy.

Open Source Software

• Adaptive load shedding in stream data management systems: source code and data, paper
• Dynamic packet size adaptation in Bluetooth: source code, paper

Project Outcomes

Data-intensive real-time applications, including transportation management, military surveillance, and network monitoring, need to handle massive amounts of data streams in a timely fashion. It is challenging to support real-time stream data services (RTSDS) due to stringent timing constraints, potentially unbounded continuous stream data, unpredictable stream data arrivals, and workload variations due to data value changes. To address the challenges, this project has develop cost-effective methods and a runtime system for RTSDS. The project has systematically investigated and developed methods and tools to support real-time continuous queries for RTSDS even in the presence of dynamic workloads. Key outcomes include the following: 1) various control theoretic techniques are applied to support robust real-time data services, 2) systematic data stream load shedding techniques are developed to avoid overload, 3) effective deadline assignment and RTSDS differentiation schemes are developed, 4) predictive thermal control techniques are developed to avoid overheating the processor due to intense real-time data stream processing, 5) a new real-time scheduling mechanism is developed to leverage the massive parallelism provided by GPGPUs (general graphic processing units), 6) a real-time sensor data analytics framework, called RTMR (Real-Time MapReduce), is designed and implemented to support parallel in-memory processing of sensor data streams in real-time, 7) For real-time streaming data collection and delivery, trade-offs between delay, throughput, and lifetime are quantified to support stochastic network analysis approaches, 8) Using network calculus tools, it is proven that stochastic comparison of scaled service curves of network paths is equivalent to quantile comparison of delays, which accommodates stochastic design tools compared to traditional average-based utility metrics, and 9) a stochastically dominant path selection (STOOP) solution is developed and implemented for real-time streaming path decisions with stochastic information. The expected broader socio-economic impact of this project is significant due to many important applications that can be supported by RTSDS. For broader dissemination and adoption in industry as well as academia, the developed open source software and data used for experiments in addition to the publications are made publicly available at the project web site: http://www.cs.binghamton.edu/~kang/rt-stream.htm . Moreover, underrepresented groups of students were advised to work on relevant projects and publish research results at peer-reviewed conferences and journals.