Hiperwall video wall software has always been great at accepting and displaying lots of sources of several types, but sources and their uses have changed over the years as what they were showing evolved. This post contains my experiences with the changing landscape of sources used in Hiperwall video wall systems around the world, particularly for control rooms and similar, rather than signage applications.

Early days

Our first source type was the Sender, followed closely by the Streamer. These standalone apps send data to the video wall computers in very different ways making their uses unique.

Sender

The Sender software (now called HiperSource Sender) captures the screen or part of the screen of the computer it runs on and sends that to the Hiperwall system. Sender can run without even being installed on the computer (for environments that restrict software installation). The encoding of the Sender’s video stream is completely CPU-based, so it runs on nearly any PC, Mac, or Linux box, and the performance scales with the CPU and network speed. Sender can send its data directly to the video wall if it is on the LAN or can use the HiperController as an intermediate via an encrypted channel. This allows Sender sources to come from anywhere on the Internet. We extended this capability with a product now called HiperCast (was Share) that can deliver multiple Sender sources to multiple Hiperwall systems around the world.

Our clients use Sender for monitoring dashboards, social media feeds, desktop sharing, and more. Sender provides multiple captures on a single machine, runs in VMs, and supports KVM control of the source machine. The flexibility and utility of Sender means customers want to use it in many situations where high frame rate is not required. Sender also scales and resizes handily, so if a user changes the resolution or orientation of the screen of a Sender PC, it can adjust automatically.

While Sender is our oldest source type, it has undergone many performance and capability improvements over the years and remains the simplest and perhaps most flexible of our sources. Sender was built at a time when most of our customers were running custom applications to monitor and control their systems, so it was perfect for delivering those output screens to the video wall. Since the nature of such applications has changed to be more web-based, other solutions, especially HiperSource Browser, discussed later, grew in popularity.

Streamer

The HiperSource Streamer software is designed to deliver high frame rate, high quality video streams to the Hiperwall video wall. It uses hardware accelerated video capture, compression, and encoding to send a bandwidth-efficient video stream. Streamer uses our patented approach to synchronize playback on different display computers to make the frames sync across display boundaries for a seamless experience even if the video spans multiple display tiles. Streamer provides display capture, like Sender, but also supports capture cards, so it can stream video content captured from HDMI, SDI, or analog video sources to the wall. For display capture, Streamer also supports KVM control of the Streamer PC, which means content and applications on that PC can be interacted with while being shown on the video wall. Because Streamer requires hardware support to encode the stream, it must run on a moderately powerful PC on the Hiperwall LAN.

Since Streamer is great for high frame rate video streams, our customers use it to play videos, show presentations that have artistic transitions, share VMS consoles or other applications where high frame rate matters, and to show TV stations to monitor news and weather. Streamer has evolved over the years to support more hardware types and to manage capture cards in a flexible manner. Streamer continues as the go-to source type for video-style streaming.

Streamer+

With version 8 or the Hiperwall video wall software, we added a complete re-imagining of the Streamer idea in a product caller Streamer+. This new product was built from the ground up for performance, allowing more streams, more desktop captures, more capture card inputs, and higher resolution and frame rate than the original Streamer on supported hardware. The new Streamer+ has been the go-to choice for new installations and even upgrade customers since it was released.

Streaming Evolution

While Sender and Streamer are well suited to sending desktops and captured streams to the Hiperwall, our customers also wanted to send networked video streams from IP Cameras, video encoder boxes, and VMS gateways to their Hiperwall video walls. Since we already had a powerful synchronized streaming protocol from the Streamer, we adapted that to make HiperSource IP Streams. The IP Streams source software can ingest many types of network streams from ONVIF cameras, RTSP encoders, RTP and HTTP sources, and more. It then wraps the streams with our patented approach to synchronize playback across the multiple displays of the Hiperwall video wall and delivers the streams to the display computers. Because the IP Streams software is so efficient, a single moderate PC can ingest and process around 75 streams for simultaneous delivery and playback on a Hiperwall system.

With the addition of HiperSource IP Streams, and to provide more flexibility to our customers, we combined all the source license types into a single HiperSource license type. Therefore, customers could easily switch sources as their needs changed and no further license updates were needed. This simple source interchangeability has worked very well for our customers who may not know exactly what they want to use when they are defining their system. Now they can pick and choose and change as needed.

IP Streams has become a significantly popular source type for many of our clients’ applications. Some need to display streams from IP cameras, possibly via a VMS gateway, such as those provided by Milestone and Genetec. Some customers in secure or otherwise restricted environments use video encoder boxes to take the HDMI output of a computer and convert it to an RTSP stream that the IP Streams software delivers to the Hiperwall. Thus, their secure, mission-critical computers never have to be on the same network as the Hiperwall system, and the only interface to the Hiperwall network is a video cable. IP Streams also delivers content from TV decoder boxes, now that MPEG 2 streams are supported.

Large scale revolution

While Sender and Streamer have been able to capture web pages and send them to the video wall from the start, we wanted to take web content to the next level, so we developed HiperSource Browser. It is a real web browser, based on the Chromium engine, so it supports current web technologies, but it scales in amazing ways. Like most web browsers, it supports tabs with different content (web pages or PDFs, etc.), but each tab is actively rendered and sent to the Hiperwall video wall simultaneously. This means one PC can deliver several web page sources at once while the user does something else entirely. HiperSource Browser also scales in size, allowing a single web page capture to be huge. A control room customer with an enormous video wall uses HiperSource Browser to send several dashboards with 10s of millions of pixels each to their giant Hiperwall. That isn’t a typo – tens of millions pixels worth of custom dashboard data each! Other customers use Browser for normal-size web content.

Because the world has migrated from custom applications to web-based application and dashboards, HiperSource Browser has an incredibly bright future as more and more customers switch over to modern infrastructure. Its flexibility to send multiple content items of differing, possibly enormous, size makes it strongly in demand. And because it works in virtual machine environments and doesn’t interfere with other uses of the PC, it is very friendly to the staff and operators that use Hiperwall video walls. Browser is built on the Sender protocol, so it can send across the Internet via an encrypted channel and HiperCast can deliver Browser streams to multiple Hiperwall systems at once.

The future

The current broad range of sources described here support our customers’ current use cases, but technology is always changing. Performance and feature improvements are obvious next steps, but we are always examining new use cases and customer needs. New streaming protocols are becoming popular for both audio and video, so we are watching market acceptance of those. Integration with collaboration or other products could also be in the cards as the world recovers from disruptions over the last years. We have great sources to get your content to your Hiperwall video wall system, but we’re far from done.

Hiperwall video walls have supported playing content with audio since the beginning, but we had avoided adding audio controls to the system because of the complexities described in this article. That changed with a product version we released several months ago. This article describes the journey the development team took to make the robust and powerful audio controls in the current product. Customers with maintenance contracts can upgrade to the current version and get these new audio management capabilities for no additional cost.

Why Audio?

Hiperwall makes video wall software, so why is there a need for audio in such a visual medium? Many of our customers use their Hiperwall video wall in a control room environment, where audio might be a distraction. Therefore, whatever audio features we add must allow operators to easily mute the system to avoid disturbing the people monitoring and using the control room video wall. Many other customers, however, use their Hiperwall video wall in some sort of public (or employee) facing application where audio is commonly used. In the past, we advised such customers to use a multi-input audio mixer to manage their audio with fine-grain control.

Why is Audio Management a Challenge?

The Hiperwall video wall software is a distributed and parallel computing system that uses multiple display computers to draw content on video wall displays, including LCD, projector, and LED walls, which allows it to scale from very small systems to enormous systems with hundreds of displays or thousands of square feet of LED tiles. Each of those display computers can have audio output and perhaps even speakers attached. Beyond that, a Hiperwall video wall can have tens or hundreds of sources and content items, many of which can have audio channels. Therefore, video wall audio control not only has to manage display computer output volume, but also volume levels for perhaps hundreds of content items, all in an easy-to-use and scalable manner.

Project Beginnings

One of our significant partners expressed the need for volume control for the display computers, because some of their customer projects could use those capabilities. Having looked into volume controls in the past, I thought, “I know how to do that. I’ll do it.”

In the early days of Hiperwall software development, I made the Hiperwall Daemon, a small program that runs on each of the display computers to manage them, transfer content and issue commands, and make sure the display software is running. Having the Daemon control the output volume of its display computer is easy but communicating to make it do so was the challenge. I enhanced the state transfer protocol to allow individual volume control for each of the possibly many displays in the system. The protocol needed to be concise, yet scalable, so the mechanism I added extended an existing control scheme to add volume values when needed. I also added a volume control slider and status reporting to the “Walls” tab in the controller software so volume levels can be quickly and easily changed for one display or an entire wall at a time. The volume values also had to be consistent between fault-tolerant controllers, so I made sure the controllers coordinated.

Content Audio Control Development

After discussing this new capability with the team, we decided that we couldn’t just stop at device volume control, but needed to address content volume as well. Managing volume of individual content items is more technically challenging than just changing device output volume. Because each display computer can have many contents playing audio simultaneously, each object needs separate audio controls both on the controller and built into the display software. Managing audio levels for multiple items is a lot like using an audio mixer like the Windows mixer (right-click on the speaker in your system tray and choose “Open Volume Mixer” to see what I mean). Each object can have a volume setting, but the display computer volume limits that volume to its maximum.

The development team members who make the display software developed mechanisms to adjust the volume level for each object open on their display(s), and that worked very well. We added object volume to the extensive list of properties we manage for each object (like size, position, rotation, transparency, etc.). We added a volume slider to the controller so the operator could change the volume of a selected content object. We the made mechanisms to propagate the volume properties throughout the system and between fault-tolerant controllers. We also extended the existing Environment mechanism so volume levels are saved with each Environment and restored when the Environment is loaded. We even added volume control to our XML-based web services style interface so third-party programs can control content volume. But we weren’t done yet…

Audio Muting Policies

Being able to quickly and easily mute audio content is critical – we do it all the time with our TVs and radios and phones, but when you have many outputs and potentially hundreds of audio sources, it becomes a lot more complicated. We first had to decide which operations made sense. Of course, we needed the ability to mute all content, but we had to decide what it meant to mute audio. Was it just setting the volume of all the content items to 0? That’s easy, but it doesn’t remember the old value in case we want to unmute some or all of the content. Your TV remembers the volume it was before it was muted, so we decided customers would expect that kind of behavior, even though a TV is only playing one thing at a time, while we can show many items at once. Therefore, we had to make muting reversible rather than just setting an object’s volume to 0. This meant maintaining a bit more state and a lot more logic to mute and unmute properly. Of course, muted objects can be unmuted as a group or individually.

Beyond muting everything, we also devised a “mute all but this” mechanism, which allows the operator to define an object that should be the focus of attention by eliminating any potentially distracting audio from other source. While this is not something TVs would have, it is very beneficial in a video wall environment with many sources that include audio.

Convenient and Simple Controls

Because audio has the potential to be disruptive and distracting, we had to make sure the controls to mute everything were easy to find and always available. In addition to the “Mute All” capability as part of the audio controls, we added very obvious system Mute button to the controller so it can be used quickly and easily. Like its “Mute All” counterpart, this Mute button can mute or unmute all the content at once. It has intuitive icons to show its state rather than words like the button in the audio controls.

In addition to the fault tolerance controller software, we have another way of controlling the video walls called HiperOperator. This is a very easy-to-use graphical application that several possibly remote operators to manipulate content on the video wall simultaneously. In keeping with the simplicity of HiperOperator, we had to add very simple audio controls to it. HiperOperator makes it very easy to manipulate properties of individual objects, including applying filters, rotating, and even making the transparent using a simple menu. We added audio volume and muting to the menu, making it extremely convenient to set audio properties of content items. We also added the object’s volume level to the descriptive label shown with each object in HiperOperator.

As with the controllers, HiperOperator needed a very simple way to mute and unmute all audio in the system, so we added a button/icon in the corner of the display that toggles the muted state when clicked. All of these volume and mute states need to be coordinated across all controllers and HiperOperators, too, so if one user performs a volume control operation, it is reflected everywhere. As anyone who builds distributed systems knows, such coordination is easier said than done, but because we were building on already robust protocols and communications links, it turned out very nicely.

Development of audio control capabilities was a very interesting and challenging activity, but more for policy reasons than technical ones. The underlying technology to change display computer output volume or individual object volume is not particularly difficult. Rather, defining behaviors and capabilities was by far more challenging. Since multi-input, multi-output audio is rare except in audio mixers, we had to do lots of prototyping and debating of how things should work. We worked with our partners to get feedback on our designs and made some changes based on their suggestions. The new audio features are the result of a great collaborative effort among the dev team and with our technical services group and our partners.