Scientific instruments, such as radio telescopes, colliders, sensor networks, and simulators generate very high volumes of data streams that scientists analyze to detect and understand physical phenomena. The high data volume and the need for advanced computations on the streams require substantial hardware resources and scalable stream processing. We address these challenges by developing data stream management technology to support high-volume stream queries utilizing massively parallel computer hardware. We have developed a data stream management system prototype for state-of-the-art parallel hardware. The performance evaluation uses real measurement data from LOFAR, a radio telescope antenna array being developed in the Netherlands.

Transportation–related problems, like road congestion, park-ing, and pollution are increasing in most cities. In order toreduce traffic, recent work has proposed methods for vehiclesharing, for example for sharing cabs by grouping “closeby”cab requests and thus minimizing transportation cost andutilizing cab space. However, the methods proposed so fardo not scale to large data volumes, which is necessary tofacilitate large scale collective transportation systems, e.g.,ride–sharing systems for large cities.This paper presents highly scalable “trip grouping” algo-rithms, that generalize previous techniques and support in-put rates that can be orders of magnitude larger. The follow-ing three contributions make the grouping algorithms scal-able. First, the basic grouping algorithm is expressed as acontinuous stream query in a data stream management sys-tem to allow for very large flows of requests. Second, follow-ing the divide–and–conquer paradigm, four space–partition-ing policies for dividing the input data stream into sub–streams are developed and implemented using continuousstream queries. Third, using the partitioning policies, par-allel implementations of the grouping algorithm in a paral-lel computing environment are described. Extensive experi-mental results show that the parallel implementation usingsimple adaptive partitioning methods can achieve speed–upsof several orders of magnitudes without significantly effect-ing the quality of the grouping.

Scalable execution of continuous queries over massive data streams often requires splitting input streams into parallel sub-streams over which query operators are executed in parallel. Automatic stream splitting is in general very difficult, as the optimal parallelization may depend on application semantics. To enable application specific stream splitting, we introduce splitstream functions where the user specifies non-procedural stream partitioning and replication. For high-volume streams, the stream splitting itself becomes a performance bottleneck. A cost model is introduced that estimates the performance of splitstream functions with respect to throughput and CPU usage. We implement parallel splitstream functions, and relate experimental results to cost model estimates. Based on the results, a splitstream function called autosplit is proposed, which scales well for high degrees of parallelism, and is robust for varying proportions of stream partitioning and replication. We show how user defined parallelization using autosplit provides substantially improved scalability (L = 64) over previously published results for the Linear Road Benchmark.

Scalable execution of expensive continuous queries over massive data streams requires input streams to be split into parallel sub-streams. The query operators are continuously executed in parallel over these sub-streams. Stream splitting involves both partitioning and replication of incoming tuples, depending on how the continuous query is parallelized. We provide a stream splitting operator that enables such customized stream splitting. However, it is critical that the stream splitting itself keeps up with input streams of high volume. This is a problem when the stream splitting predicates have some costs. Therefore, to enable customized splitting of high-volume streams, we introduce a parallelized stream splitting operator, called parasplit. We investigate the performance of parasplit using a cost model and experimentally. Based on these results, a heuristic is devised to automatically parallelize the execution of parasplit. We show that the maximum stream rate of parasplit approaches network speed, and that the parallelization is resource efficient. Finally, the scalability of our approach is experimentally demonstrated on the Linear Road Benchmark, showing an order of magnitude higher stream processing rate over previously published results, allowing at least 512 expressways.