A unified software architecture for video convergence solutions

Long gone are the days when people used pigeons to send messages or visited the Colosseum to see gladiator fights. The face of human civilization has drastically transformed on counts of both communication as well as entertainment. The former has become more universal and the latter more personal. Imagine this:

Alex manages a team of fifty in an advertising agency in New York. He is required to attend a client meeting that evening, but needs to leave early for a dentist’s appointment. He calls his business manager over his desk videophone and discusses the agenda of the presentation, which he simultaneously streams. On his way to the dentist, Alex sends an SMS to his secretary about the meeting over his car’s media panel, where he had earlier received the graduation pictures of his brother from Vienna. He then tunes the panel to play FM radio. While waiting at the clinic, he calls in to the ‘place-shifter’ installed on his home network from his mobile, and watches his favorite series, ‘Lost’. Later from home, Alex dials into the agency’s network and videoconferences into the meeting. His secretary has set up the conference phone to live stream the presentation to his home phone. He discusses his review comments with the clients over the call and they conclude the meeting with a business approval for the advertising campaign. They make the announcement to all employees by broadcasting the news to their desk videophones. Alex also streams the video recording of the conference call with business partners in India, who provided the special effects for the presentation campaign.

Motivation


A plethora of this and more communication and entertainment scenarios cross each other’s paths every day. There is a wide range of products in the market today that address each of these requirements, independently. Imagine an innovative solution that blends all these functionalities into a universal design that OEM/ODMs can customize for the end user. Such a product has been realized at Ittiam Systems in the form of its Video Convergence Solutions (VCS) offering. VCS is a software architecture design solution aimed at providing the core video communication functionality (half or full duplex, over heterogeneous networks) and a wide range of entertainment and media sharing features (Figure 1). More importantly, it addresses the convergence applications that emerge out of these application scenarios.

Design challenges
Convergence applications are still an emerging market area and require the challenges to be ‘learnt’ rather than being ‘known’. At the outset, it may appear that a gadget having multiple applications can serve the purpose. However, an application such as a video communication with a record and streaming (client/server) capability needs more depth than simply running two independent applications – calling for true convergence of different application scenarios. By integrating these application specific requirements as ‘features’ of an architecture (vis-à-vis multiple applications), we can make sure that the resource utilization (in terms of processor(s), memory, DMA) is optimized while allowing for better controllability of the individual features. For instance, a 3-way conference requires scheduling of around 30 components on a single or multi core solution (typically comprising of a host processor and a DSP). Similarly, a feature like acoustic echo cancellation comes with the need for platform (OS, processor) portability in a generic fashion while taking care of the specific needs for each platform (RTOS vs. non-RTOS) without affecting the overall performance.

A key aspect of providing a solution such as VCS is not only to understand the requirements, but also to architect it to suit the target application(s). This requires acute foresight to project these application scenarios and the competence to deliver the best for each choice of processor, OS and hardware peripherals that the OEMs/ODMs choose to build their solutions for the end consumers (Figure 2).


Overview of Ittiam’s VCS Architecture
Ittiam’s VCS design solution is unique in terms of the architecture concepts that it embodies. The architecture comprises of well-defined building blocks that interconnect via well-defined interfaces (Figure 3). The core media engine is cleanly abstracted from the application layer (such as user interface) allowing it to be ‘plugged in’ to a host of application scenarios. Abstractions for OS and hardware specific functionality in the form of abstraction layers (OS Abstraction Layer, Driver Abstraction Layer) are particularly crucial architecture elements that ensure the portability of the solution for multiple target platforms while extracting optimal performance for that choice of hardware and OS.

The media engine is based on a scalable, modular framework architecture (Media Engine Framework) that facilitates seamless integration of components (audio/speech/image/video codecs, etc). Also critical to the architecture are the concepts of uniform and efficient component management via an abstraction layer that also allows efficient utilization of processors across single core as well as multi-core variants. The architecture builds in the ability to efficiently utilize the host of peripherals and processors in the most optimal manner to be able to extract the best capability. This flexibility is a key ingredient in addressing both the entertainment and communication applications.

Performance aspects of Ittiam’s VCS
In order to maintain focus on addressing the needs of each application scenario, the architecture employs various functional blocks to take care of key aspects of performance – media, network, standards and robustness. These key functional blocks designed are to deliver best performance in a required direction. Of special significance in this regard is the core media engine, which encompasses several IPs in a wide range of video codecs (H.264, H.263, MPEG4, MJPEG), audio & speech codecs (ITU G series, iLBC, GSM AMR, AAC-LC, MP3, etc), quality enhancement modules (silence suppression, comfort noise generation, pre and post-processing components (resizers, deinterlacers, etc.) and ensures that functionality like audio-video-synchronization and constant bit rate transmission at all times. Furthermore, the innovative framework design combines these IPs in a flexible manner. For example, the architecture can be configured to work as a regular VoIP phone with its support for standard telephony features like call progress and DTMF tones and act as a VoIP to PSTN gateway as well.

The Network Subsystem that takes care of aspects ranging from protocol compliance (packet transmission, payload formats, etc) and management of multiple media streams, and provides functionality to take care of jitter, clock-skew, packet loss and network delay amongst other impairments. This subsystem can be configured as a delay optimized system (communications scenario) as well as a quality-optimized system (entertainment scenario). It also facilitates efficient handling of data buffers that apply to each of these scenarios.

In order to ensure the reliability of all these features, it is important that the architecture design be complemented by test cases designed to cover the entire breadth and depth of the offering. Multiple production test cycles of Ittiam’s VCS have made the solution robust, even under varied operating environments like wired and wireless LANs, network simulators as well as the public network. Moreover, the solution has been verified to interoperate successfully with many standard VoIP/video phones, streaming servers and clients that illustrate its usability in controlled as well as field environments across multiple application scenarios that it targets.

It is these strengths of Ittiam’s VCS that have enabled Ittiam’s offerings on 3 different operating systems and 4 platforms to several customers spread across Asia (Japan, China, Korea), USA and Europe.


Conclusion
With a rich portfolio of about 10 patents at algorithm, modules and system level, Ittiam’s VCS offers wide customizability and programmable interfaces options to support myriad applications (desktop phone, media player, streaming server / client, photo album, video camera, access control device, video security, etc) on a varied range of platforms (low power, high density, multitude of operating systems, etc).

With the advent of more powerful processors, a general trend towards confluence of personal media, personal communication (ex: mobile phones) or security communication devices (example: hazard job aids, etc) and need to transfer information (or data) efficiently across distances, a design solution like Ittiam’s VCS is geared up to take on the challenges that the company foresees as the future.

Click here for Illustrations:

Figure 1, Figure 2, Table 1