Professor Frank Eliassen, University of Oslo (project leader)
Professor Carsten Griwodz, Simula Research Laboratory
Professor PŒl Halvorsen, Simula Research Laboratory
Dr. Viktor S. Wold Eide, University of Oslo (post doc)
Anh Tuan Nguyen, University of Oslo (PhD scholar)
Pengpeng Ni, Simula Research Laboratory (PhD scholar)
HŒkon Stensland, Simula Research Laboratory (PhD scholar)
The main objective of the project was to investigate and provide solutions for how the provision of multimedia streaming services can be made robust with respect to adaptivness towards an open-ended range of devices operating in highly heterogeneous and possibly dynamically varying networking environments. To achieve this overall goal the sub-goals of the project were:
1. Design and prototype a middleware framework that enable and simplify the development of context-aware self-adaptable multimedia streaming services, and support automatic adaptation and personalization of multimedia streaming services.
2. Develop and evaluate a component-based distribution infrastructure for real-time streaming of resource demanding media (such as video) that is pluggable into the middleware and that can be configured and reconfigured by the middleware.
3. Provide a demonstration of use of the middleware framework and plug-in distribution infrastructure for a real and non-trivial test case.
The next generation of streaming services will be the basis of personalized applications, where everybody will be able to receive media content tailored to his/her own preferences, character, and context.
Personalization already exists both on the Internet and with mobile technologies. There, operators learn about their customers, draw their profile and push them content. In our scenario, producers deliver their services across the networks of several operators. For consumers who use an interactive application, they have to deliver their service independently of network technology and end-device. Heterogeneity of networks and terminals is one of the sources of complexity in the above scenario. A challenge is to provide each consumer with the best possible experience when considering context and individual preferences, while maintaining efficiency and scalability in all systems that are involved in the delivery from
producer to consumer. Context changes in this work are limited to include changes in computational resource availability (CPU, memory, network) and the joining and leaving of devices.
Streaming systems have to cope with variability on a shorter time scale. The streamed content itself changes over time, which often translates into variable resource requirements, for example, with respect to bit rates and processing needs. The resource availability may also change over time.
The overall challenge resulting from the discussion above is how the provision of many-to-many multimedia streaming services can be made robust with respect to delivery to an open-ended range of devices operating in highly heterogeneous and possibly dynamically varying networking environments, and where the distribution system is partially shared and without centralized control, and where consumers have individual context-dependent preferences with respect to the quality and modality of the streaming media.
The project has produced results in three main areas that contribute to the goals of the project in different ways. For each of the areas we first describe the results and then how they relate to the goals.
The overall goal of this research was to achieve adaptive and robust Peer-to-Peer (P2P) streaming services, which are believed to be the next generation of P2P streaming on the Internet. P2P technologies have emerged as a powerful and popular paradigm for bringing such emerging multimedia services to a large number of users. The essential advantage of P2P systems is that the system capacity scales up when more peers join, as peer upload capacity is utilized. However, providing satisfactory streaming services over P2P networks is challenging because of their inherent instability and unreliability and the limited adaptability of traditional video coding techniques. On one hand, different from dedicated servers, users may not have enough bandwidth to serve other users. In addition, users can join and leave the system at any time as there are no guarantees on their contribution to the system. On the other hand, traditional video coding techniques do not support scalable modes efficiently as such modes come along with high computation cost. Consequently, in traditional P2P streaming systems, the bit rate (the video quality) of media streams is determined based on the capacities of the low-end users, i.e. the lowest common denominator, to make sure that most of their users can perceive acceptable quality. This causes two critical limitations of the current P2P streaming systems. First, users perceive the same quality regardless of their bandwidth capacity, i.e., no differentiated QoS. Second, with the current best-effort Internet and peer dynamics, the streaming quality at each peer is easily impaired, i.e., no continuous playback.
Recently, multiple layer codec research has become more refined, as SVC (the scalable extension of the H.264/AVC standard) has been standardized with a bit rate overhead of around 10% and an indistinguishable visual quality compared to the state of the art single layer codec. The hypothesis of this research work is that the adaptable coding technique can bring significant benefits to P2P streaming as it enables adaptability in P2P streaming. In addition, to improve the robustness of the system to network fluctuations and peer dynamics, network coding and social networking are also applied.
Several major contributions are provided under this topic. First, an architectural framework for P2P video streaming has been developed. This open architecture identifies key functionalities such as neighbor selection, quality adaptation, sender selection and peer coordination as pluggable components so that different design options of each component can be added and used in different conditions.
Secondly, to use SVC in P2P streaming, a segmentation method to segment SVC streams into scalable units is proposed such that they can be delivered adaptively by the P2P paradigm. The method is demonstrated to be able to preserve the scalability features of a stream, i.e., adaptation can be applied on segments and the re-generated stream at each peer is a valid stream.
Third, a novel and complete adaptive P2P streaming protocol, named Chameleon, has been developed. Chameleon uses the segmentation method to use SVC and combine it with network coding in P2P streaming to achieve high performance streaming. The design of Chameleon would serve as a sample design of unstructured layered P2P streaming systems with its key components and their interactions. The adaptability of Chameleon would change the behavior of P2P streaming systems. Chameleon users can receive the best possible video quality according to their available bandwidth capacity rather than receiving the same quality. Rather than suffering playback skips when bandwidth drops, they can perceive reduced quality but continuous playbacks. These features would improve user satisfaction in using P2P streaming services. Other work could consider Chameleon as a baseline to investigate more problems and bring adaptive P2P streaming to the real world.
Fourth, a neighbor selection protocol and a peer sampling-based membership management protocol for layered P2P streaming are proposed. Our study on membership management for layered P2P streaming contributes to the research area in two ways. First, it demonstrates that traditional membership management protocols are not sufficient for layered P2P streaming, e.g., high capacity peers may receive low video quality because they are surrounded by low capacity peers. Second, our proposed membership management protocols show that by taking peer capacity into account, a gossip-based protocol is good enough to boost an adaptive P2P streaming protocol. Other work could inherit the findings about the good features of quality-aware overlays for adaptive P2P streaming and the current version of the protocols to build a more practical protocol when taking other network metrics into account.
Fifth, to better deal with peer dynamics, Stir, a social-based P2P streaming system, is suggested. Stir is the first attempt to apply social networking in P2P streaming. Allowing social activities during streaming sessions, Stir offers more entertaining streaming services, e.g., users can communicate and share opinions directly about the content they are watching together. In addition, with Stir, faithful users who have spent time in the system and have many friends are rewarded with a high priority in receiving satisfactory streaming quality. Such personalized services are important in attracting users and have not existed in current P2P streaming systems. With our approaches, differentiated QoS and personalized services, which have only been available in client-server streaming systems, are now possible in P2P streaming. Others could learn from Stir: How does the idea of spontaneous social networking work? How could it be exploited efficiently in P2P streaming systems?
Finally, as the approaches are about different aspects of adaptive and robust P2P streaming, to complete the picture, Chameleon++, which combines Chameleon and Stir, as been developed. The design and the evaluation of Chameleon++ demonstrate the feasibility and the benefits of the approaches, and the consistency of the study.
Towards the goals of the project, the common criteria of the above solutions are adaptiveness and robustness. The aim of Chameleon is to achieve a basic solution for adaptive and robust P2P streaming, which can adapt to heterogeneous users (different user capacities -the heterogeneity can come from different end-user devices- and different user preferences), and environments (network conditions). Stir is the first step to offer personalization services in P2P streaming systems as user can build up their profile and have friendships with other users to share common interests. The component-based design of Chameleon and Stir offers an open architecture so that different design options of each component can be added and used in different conditions. Some key components of the designs can be implemented in form of middleware, e.g., neighbor selection component to support different on-top applications. The proposed overlay construction methods offer simple but flexible and efficient mechanisms to build up a quality-aware and context-aware overlay for adaptive and robust P2P streaming. Although the proposed solutions are evaluated by simulation, realistic simulation settings give reasons to believe that they can work well in practice. Finally, Chamelon++ contributes to the integration of individual results into a single platform.
To provide universal multimedia experience, multi-media streaming services need to transparently handle the variation and heterogeneity in the operating environment. Scalable video coding (SVC) techniques are highly attractive for designing such a robust video streaming system. To cope with the fluctuation of the available bandwidth, scale video coding technique can adaptively change some video properties so that the size of the streamed video segment fits. Examples of tunable video properties include picture size, texture details, and the number of video frames to be delivered. The change of these video properties results unavoidably in different types of visual distortion. For instance, decreasing texture details and picture size will result in crude and blurry picture quality, while dropping video frames will cause motion jerkiness. Moreover, when the video properties are tuned up and down regularly or irregularly under the constraints of the available bandwidth, additional annoying effects will occur due to the quality fluctuation in the streamed video. For instance, when video quality is fluctuated up and down fast enough, people will notice an uncomfortable flicker effect. Such kind of visual artifacts cannot be measured accurately by any current known objective video quality metric.
To make the best use of SVC techniques and provide user the best multimedia experience possible, we performed a series user studies to subjectively evaluate different visual artifacts in SVC. If well designed and carefully conducted, subjective evaluation is the most intuitive and reliable way of examining the results of multimedia experience. Any objective quality metrics should be examined by matching their prediction to the subjective results. Knowledge gained by user studies about human's reaction to different kinds of visual distortion provides also guidance to system design and other research works with SVC assumption.
By user studies, we find that the impact of visual distortion is related to video content characteristics, viewing distance and peripheral vision etc. One of our studies reveals that the flickering effect caused by frequent switching between layers in SVC compliant bit-streams is highly related to the switching period. When the period is above a certain threshold, the flickering effect disappears and frequent layer switching is not considered as harmful. Another study examined user perceived video quality in 3D virtual worlds. Our results show that the avatars' distance and angle to the virtual screen in 3D worlds contribute mostly to the visual experience, i.e., for a wide extent of distortion, there exists always a feasible virtual distance from where the distortion is not detectable for most of people, which makes sense to perform video adaptation.
Conducting subjective quality evaluation test is usually both time consuming and expensive. There is a lack of enough facilities for conducting these tests. To help with this, we developed a toolbox to simulate reference impairments resulting from SVC techniques. We developed Randomized Pair Comparison (R/PC), an easy-to-use, flexible, and robust test method for subjective quality assessment. The R/PC method reduces significantly the burden on the test participants and hence can be easily applied to laboratory experiments and self-controlled field studies. Based on this method, we prototyped an audiovisual quality assessment tool on the popular iPhone OS platform. This facilitates and automates the test procedures to the extent possible. Hence subjective tests can be easily conducted in various locations and among people with wide range of backgrounds, meanwhile the system designers are still able to obtain robust results efficiently and economically.
In relation to the project goals, the H.264 SVC extension, which was released after the ROMUS project started, provides just the expected middleware framework design. But how to make the best use of the design and how to evaluate it remain as open issues, which makes the motivation of this research work. We performed user studies to subjectively evaluate the framework, aiming to provide guidelines to design real time video streaming infrastructure adopting SVC techniques. How to ensure the actually delivered visual experience becomes the main focus of this work. The guidelines can be integrated into the quality adaption component of the open architecture framework described above. At the time of writing this has not yet been done.
Many multicore processor scheduling approaches exist, but to the best of our knowledge, a challenge is still to find mechanisms that can schedule dynamic workloads of communicating operations while taking both the processing and communication requirements into account. For such applications, we believe that feasible scheduling can be performed in two levels, i.e., dividing into the task of placing a job onto a processing unit and the task of multitasking time-slices within a single processing unit.
We have done several investigations in the field of general purpose computing on graphics processing units (GPGPU). Here we both have analyzed the performance characteristics of GPUs, using the nVIDIA CUDA framework to investigate the properties of the processors. We have also tried to offload several applications like video encoding and cheat detection to a GPU. Our investigations also shows that programmers have to think differently when programing heterogeneous architectures like Cell and GPU architectures, and existing frameworks like OpenCL for programming these processors does not hide the architectural differences. In collaboration with the iAD SFI we have developed a framework for distributed processing on computer nodes in a cluster. This framework called P2G has support for multimedia workloads with soft deadlines, and allows the programmer to develop applications in a single language.
The R&D tasks conducted by the project mainly follow from the above description of the project. This includes:
á Development of an open architectural framework for adaptive and robust peer to peer video streaming
á Development of a segmentation method to segment SVC streams into scalable units such that they can be delivered adaptively by the P2P paradigm.
á Development of a novel and complete adaptive P2P streaming protocol, named Chameleon. Chameleon uses the above segmentation method for SVC and combines it with network coding to achieve high performance P2P streaming.
á Development of a quality-ware neighbor selection protocol and a peer sampling-based membership management protocol for layered P2P streaming.
á Development of a framework for distributed multicore processing of multimedia workloads.
á Development of Stir, a novel social-based P2P streaming system.
á Performed a series of user studies to subjectively evaluate different visual artifacts in SVC.
á Development of a toolbox to simulate reference impairments resulting from SVC techniques.
á Development of a simulator for evaluating our P2P video streaming middleware architecture.
We believe the suggested project organization has worked out satisfactory for the progress of the project. In particular the project has
á organized project workshops twice a year reporting and discussing results achieved so far and future work and promoting team work.
á participated in partnership programs with international partners to expose the project work to a critical review
á organized reading groups with PhD and master students to address state-of-the art in areas constituting particular challenges to the project
á focus on organizing the research work around Òthe next publicationÓ. This helped in defining running research goals and detailed plans and clearly defining the expected contribution of each result.
The ROMUS project adopted research methods that are considered to be the most reliable and trustworthy in the research community (systems research) to whom we direct our publications. Thus, the principal research method has been simulations and prototyping as the main methods of validation. Prototyping is a very resource demanding method of research but nevertheless often deemed necessary in systems research, as it requires extensive programming to perform the implementation work of the prototypes and conduct the experiments. For the large-scale P2P video streaming experiments, simulation was chosen as the main method to evaluate the proposed solutions. This turned out to be the only feasible method given the time and resource constraints at hand.
To do a PhD thesis at an international good level effectively spending 2½ years on the research work and writing, is a challenge in general. Doing it in the area of systems research in the same amount of time is probably even harder. At the time of this writing, only one of the PhD students in the Romus project has submitted and defended the thesis. However, in spite of this I am glad to report that the other two PhD students are in their final writing phase and ready to submit within a few months.
The proposed solutions and models of the research activities above are the invaluable and fundamental measures for the scientific quality of our research outcomes.
The development of a simulation model to simulate and evaluate novel adaptive and robust P2P video streaming solutions is essentially important for long-term research in the field. For the long run research and development, it could provide an efficient, large-scale, and close-to-practical simulation platform for conducting experimental research and bridging the gap between innovative research ideas and realization.
Peer-to-Peer (P2P) streaming services are believed to be the next generation of streaming on the Internet. It is therefore essential to find solutions to important limitations of P2P streaming technologies. We believe the basic solutions and findings about the good features of social- and quality-aware overlays for adaptive P2P streaming and the current version of the protocols offered by Romus could be inherited by other initiatives to build a more practical protocol taking other network metrics into account.
Our work on video quality assessments, provide tools and guidelines for how to design real time video streaming infrastructure adopting SVC techniques. This is crucial for being able to provide the best possible user experience with minimal visual distortion in situation with dynamically varying resource availability such as network bandwidth.
Multicore processor scheduling to handle multimedia workloads on individual peer nodes will be important in the future and may improve the multimedia experience of the user even further. An important contribution of this project in this respect is the development of a programming framework called P2G that has support for multimedia workloads with soft deadlines. This framework simplifies the task of the programmer to develop multimedia applications exploiting multicore processing in a distributed environment.
At the time of writing, two PhD candidates have not submitted their thesis yet, but as mentioned this will be done within the next months. One PhD candidate have submitted and successfully defended his thesis.
Last updated: 20110802