| |
The increasing prevalence of large-scale stream-processing applications
has created the demand for a generic overlay infrastructure capable of
sustaining large numbers of simultaneous streams. These streaming
applications arise as distributed data sources throughout the Internet
deliver data in real-time. Applications like Internet traffic
monitoring, real-time analysis of RSS feeds, and planetary scale
e-science experiments demand a robust, efficient network overlay.
To address this issue, I describe a Stream-Based Overlay Network (SBON)
that allows multiple applications to establish stream queries, leveraging
Internet resources for in-network processing. A key challenge is
network-aware optimization of queries. In particular, the global impact
of thousands of queries on the network and changes in network and node conditions must be taken into account. The SBON uses a novel adaptive
query optimization technique based on a cost space, a virtual metric
space that encodes routing costs in the network. The SBON performs
decentralized optimization decisions in terms of operator placement,
decomposition, and cross-query reuse using geometric algorithms in the
cost space. A deployment on PlanetLab shows that this approach minimizes
network usage while providing low latency to applications while and
adapting to dynamic changes. By abstracting away the details of stream
query setup and optimization, the SBON greatly simplifies the development
of Internet-wide stream-processing applications. |
