In the DMJ project we use the term "media journaling" as a blanket term to refer to a broad class of applications in which users have the capability to capture multimedia content and related information of real-world multimedia activities or sessions as they happen, as well as the ability to process the media so as to create a segmented synchronized multimedia record of events.
The focus of our research is on capturing and indexing distributed multimedia content in real time. The project develops an architectural framework addressing challenges like on-line automatic content analysis, indexing and annotation of live networked multimedia sessions. The DMJ framework is a generic application that can be reused as the basis for developing domain specific content indexing and annotation applications. It also includes a higher-level application programming interface for formulating high-level media content queries and functions for transforming high-level media content queries into distributable configurations of media content analysis algorithms. This allows for many times faster development of journaling applications.
The uniqueness of our architecture lies in the combination of probabilistic knowledge-based media content analysis with QoS and distributed resource management to handle real-time requirements. This is all packaged into a generic framework that is tailorable to specific application domains.
Issues of media journaling are similar to those found in the area content-based multimedia indexing. We may think of a user specification of journaling requirements as a complex content-query specification. However, rather than performing the content analysis on stored data, on-line analysis requires real-time processing of (live) networked sessions.
Automatic indexing of multimedia content is often based on computationally expensive feature extraction algorithms and only works satisfactory limited to specific domains. In contrast an end-user may want to access live content in terms of high-level domain concepts under a variety of processing environments ranging from complex distributed systems to single laptops or set-top-boxes.
The above reasoning suggests that a knowledge-based approach combined with open resource-aware distributed processing technology, might be a promising approach to the challenges of media journaling. First of all, for automatic feature extraction to work well, specific domain knowledge is needed during analysis. One promising approach to integrating domain knowlegde (semantics of user domains) into the content analysis process is to combine (in domain specific ways) low-level quantitative content querying (i.e. query by colour histogram, texture, pitch, etc.) into higher-level qualitative content querying. Furthermore, a probabilistic approach to detection and recognition of content is needed, since evidence of content extracted from media streams can be either missed or hallucinated (false positives) by the involved algorithms.
Secondly, a distributed solution is generally required to cope with
Our approach to the above challenges is to build upon the methodology of dynamic object-oriented Baysian networks (DOOBNs). In DOOBN, uncertainty can be handled explicitly and domain concepts may be effectivly represented as dynamic Bayesian network objects. A DOOBN consists of a hierarchy of dynamic Bayesian network (DBN) objects. Using DOOBNs, the specification of media indexing tasks is simplified by
Futhermore, the DMJ system supports different media processing strategies. DBN objects are executed periodically in time slices. A DBN time slice rougly corresponds to the sampling period of the DOOBN. Media features can be extracted either eagerly (i.e. all media feature algorithms are executed within the time limit of a DBN time slice), or on a peer need basis. The latter media feature extraction strategy is founded on hypothesis driven media classification such that media features are extracted only when the expected information gain relative to the extraction cost is significant. This again allows more flexible use of feature extraction algorithms under resource restrictions such that the reliability of content detection can be traded off against the available computing resources.
High-level content queries are transformed by the DMJ system into a hierarchy of Bayesian network objects. Each object is realized as a software component which may execute on any host computer. During execution, some components monitor other components and react to particular changes in their state. For example, a high-level DBN object realizing face recognition may monitor a lower-level face detection DBN object detecting the occurence of faces in a video stream. Monitoring in the DMJ system is realized using the publish/subscribe paradigm, leading to an event based component interaction model.
The configuration of DBN software components and their deployment in a distributed processing environment (DPE) is the result of a chosen media processing strategy in which qualitative and quantitative requirements of ther user are traded off against the available resources of the DPE. This requires knowledge about the resources of the execution environment and the performance and quality characteristics of each component.
Last modified Monday, 01-Sep-2003 10:47:06 CEST