Streaming operator placement for distributed stream processing

Details Streaming operator placement for distributed stream processing Streaming operator placement for distributed stream processing

2008-06-19

2011-11-15

Bhatia, Ajay (Department: 2445)

Electrical computers and digital processing systems: multicomput

Computer network managing

Network resource allocating

C718S105000, C709S231000

Reexamination Certificate

active

08060614

ABSTRACT:
A streaming operator assignment system and method for determining a streaming operator assignment that minimizes overload in a data processing system. Embodiments of the streaming operator assignment system include an optimization goals definition module, which defines optimization goals in terms of fundamental quantities that system administrators and application writers want to control, such as minimizing the worst case latency over all periods of time, or minimizing how much the system is backlogged with work. Embodiments of the streaming operator assignment system also include an optimization goals solution module that optimizes and solves a selected optimization goal. A specialized optimization technique is used to find the best operator (or load) assignment using the optimization goals to measure of the value of the assignment. This technique minimizes an optimization goal by iterating over all possible operators assignments over all possible nodes to find the operator assignment that minimizes the desired optimization goal.

REFERENCES:
patent: 5671403 (1997-09-01), Shekita et al.
patent: 6032144 (2000-02-01), Srivastava et al.
patent: 6691101 (2004-02-01), MacNicol et al.
patent: 7010538 (2006-03-01), Black
patent: 7200117 (2007-04-01), Chiu et al.
patent: 7272707 (2007-09-01), Liu et al.
patent: 7277950 (2007-10-01), Chapweske
patent: 7496683 (2009-02-01), Lang et al.
patent: 7831718 (2010-11-01), Chapweske
patent: 2004/0220923 (2004-11-01), Nica
patent: 2005/0021511 (2005-01-01), Zarom
patent: 2006/0085558 (2006-04-01), Solomon
patent: 2006/0129528 (2006-06-01), Miyamoto et al.
patent: 2008/0028095 (2008-01-01), Lang et al.
patent: 2008/0304516 (2008-12-01), Feng et al.
patent: 2009/0300183 (2009-12-01), Feng et al.
patent: 2009/0319687 (2009-12-01), Goldstein et al.
patent: 2010/0036779 (2010-02-01), Sadeh-Koniecpol et al.
Yang Guo, Zihui Ge, Bhuvan Urgaonkar, Prashant Shenoy, Don Towsley, Dynamic cache reconfiguration strategies for cluster-based streaming proxy, Computer Communications, vol. 29, Issue 10, Monitoring and Measurements of IP Networks, Jun. 19, 2006, pp. 1710-1721, ISSN 0140-3664, DOI: 10.1016/j.comcom.2005.09.017.
U. Srivastava, K. Munagala, and J. Widom. Operator placement for in-network stream query processing. In PODS, 2005.
Badrish Chandramouli, Jonathan Goldstein, Roger Barga, Mirek Reidewald, and Ivo Santos. Accurate Latency Estimation in a Distributed Event Processing System. In Proceedings of the 27th International Conference on Data Engineering (ICDE '11), Hannover, Germany, Apr. 2011 (to appear).
Peter Pietzuch, Jonathan Ledlie, Jeffrey Shneidman, Mema Roussopoulos, Matt Welsh, and Margo Seltzer. 2006. Network-Aware Operator Placement for Stream-Processing Systems. In Proceedings of the 22nd International Conference on Data Engineering (ICDE 2006). IEEE Computer Society, Washington, DC, USA.
Avnur, et al., “Eddies: Continuously Adaptive Query Processing”, ACM SIGMOD Record, vol. 29, Issue 10, Jun. 2000, p. 261-272.
Babu, et al., “Continuous Queries over Data Streams”, ACM SIGMOD Record , vol. 30, Issue 3, Sep. 2001, pp. 109-120.
Chen, et al., “NiagaraCQ: A Scalable Continuous Query System for Internet Databases”, In SIGMOD, 2000, 12 pages.
Dalvi, et al., “Pipelining in MultiQuery: Optimization”, Proceedings of the twentieth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, 2001, pp. 59-70.
Abadi, et al., “The Design of the Borealis Stream Processing Engine”, Second Biennial Conference on Innovative Data Systems Research, 2005, 13 Pages.
Abdelnaser Adas, “Traffic Models in Broadband Networks”, Communications Magazine, IEEE, vol. 35, Issue 7, Jul. 1997, pp. 82-89.
Ahmad, et al., “Network-Aware Query Processing for Stream-based Applications”, Proceedings of the 30th VLDB Conference, 2004, pp. 456-467.
Babcock, et al., “Models and Issues in Data Stream Systems”, Technical Report. Stanford InfoLab, 21st ACM Symposium on Principles of Database Systems, 2002, pp. 1-30.
Carney, et al., “Monitoring Streams—A New Class of Data Management Applications”, Proceedings of the 28th international conference on Very Large Data Bases, 2002, pp. 215-226.
Chandrasekaran, et al., “TelegraphCQ: Continuous Dataflow Processing for an Uncertain World”, Proceedings of the 2003 ACM SIGMOD international conference on Management of data, 2003, 12 pages.
Chekuri, et al., “On Multi-dimensional Packing Problems”, SIAM Journal on Computing, vol. 33, Issue 4, 2004, 22 pages.
Cranor, et al., “Gigascope: A Stream Database for Network Applications”, Proceedings of the 2003 ACM SIGMOD international conference on Management of data, Jun. 9-12, 2003, pp. 647-651.
Demers, et al., “Towards Expressive Publish/Subcribe Systems”, 2006, pp. 1-18.
Garofalakis, et al., “Resource Scheduling for Composite Multimedia Objects”, Proceedings of the 24rd International Conference on Very Large Data Bases, 1998, pp. 74-85.
Gross, et al., “Fundamentals of Queueing Theory”, Third Edition, Feb. 1998, 1 pages.
“The Internet Traffic Archive”, http://ita.ee.lbl.gov/.
Kifer, et al., “Detecting Change in Data Streams”, Proceedings of the Thirtieth international conference on Very large data bases—vol. 30, 2004, pp. 180-191.
“The State of the Art in Distributed Query Processing”, http://www.accessmylibrary.com/coms2/summary—0286-28755794—ITM.
Pietzuch, et al., “Network-aware operator placement for stream processing systems”, Proceedings of the 22nd International Conference on Data Engineering, 2006, 12 pages.
Shah, et al., “An Adaptive Partitioning Operator for Continuous Query Systems”, Proceedings of the 19th International Conference on Data Engineering (ICDE'03), IEEE, 2003, pp. 25-36.
Willinger, et al., “Self-Similarity Through High-Variability”, IEEE/ACM Transactions on Networking, vol. 5, No. 1, Feb. 1997, pp. 71-86.
Wu, et al., “High Performance Complex Event Processing Over Streams”, Proceedings of the 2006 ACM SIGMOD international conference on Management of data, Jun. 27-29, 2006, pp. 407-418.
Xing, et al., “Providing Resiliency to Load Variations in Distributed Stream Processing”, Proceedings of the 32nd international conference on Very large data bases, Sep. 12-15, 2006, pp. 775-786.
Xing, et al., “Dynamic Load Distribution in the Borealis Stream Processor”, Proceedings of 21st International Conference on Data Engineering, ICDE 2005, Apr. 5-8, 2005, pp. 791-802.
Garofalakis, et al., “Multi-dimensional resource scheduling for parallel queries”, In Proceedings of the 1996 ACM SIGMOD International Conference on Management of Data, Jun. 1996, pp. 365-376.

Affiliated with

Also associated with

Rating

Streaming operator placement for distributed stream processing does not yet have a rating. At this time, there are no reviews or comments for this patent.

If you have personal experience with Streaming operator placement for distributed stream processing, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Streaming operator placement for distributed stream processing will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFUS-PAI-O-4266828