My previous post was all about audio latency and how we can scientifically measure audio latency in networked performance contexts. Now, let's focus on video. Even though we are more tolerant of video delays than audio delays, video latency has a huge impact on stage aesthetics, flow and audio-video synchronization of networked performances.

The first step in managing video latency is to know how to properly measure it. There are many approaches to measuring video latency, but few of these methods are particularly well-designed for real-life scenarios, such as networked music and dance performances.

In this post, I write about some strategies and practical ways to measure video latency in networked performance scenarios, inspired by recent research and developed from years of experience running networked music courses for the MCT masters programme at the University of Oslo. Hopefully, this can guide you toward better and more scientific measurement practice.

Contents

1. How to Measure
2. What to Measure
3. Measurement Strategy
4. Measurement Procedure
5. Summary
6. Sources

---

How to Measure

Put simply, video latency is the time it takes an image or a video feed to travel from A to B. In technical systems, video delays are usually caused by various encoding and decoding stages that involve numerous hardware or software operations that take time to complete.

A clever way to measure video latency is to film a time-critical source with a camera, like a clock, and pass the signal through a dedicated signal chain before displaying it live on a screen nearby. Then, a secondary measurement system/camera is used to capture the source and the time-delayed source displayed on the screen. The time difference between the time-critical source and the same source displayed on-screen is the video latency of the system.

This general technique, and many others, was presented by Sven Ubik and Jiri Pospíšilík in their article from 2021, entitled Video Camera Latency Analysis and Measurement. But networked performances are chaotic endeavors, usually with limited time and resources allocated for scientific measurements. This makes it challenging to use Ubik and Pospíšilík's method off the shelf. Instead, we should build simpler measurement schemes more suitable for networked concert contexts that are based on scientific methods.

For example, I prefer to use more conventional equipment when in the field, modifying the time-critical sources, measurement devices, and editing procedures proposed in Ubik and Pospíšilík's method. For my sources, I usually switch out precision clocks for hand-clapping. Dealing with precision clocks and high-framerate devices to capture digital numbers can be a real hassle in the field. Hand-clapping gives a surprising amount of accuracy, is equally reliable, and doesn't require extra hardware. Of course, the accuracy will depend on what measurement camera is used. Smartphone cameras can be perfect measurement devices, but make sure you can control and log the framerate (FPS) before you use them. Knowing the FPS is vital when calculating the latency. Finally, as video is usually shot between 30 and 60FPS, free video editing software such as DaVinci and iMovie is more than enough advanced for post-analysis of the measurement footage to determine the latency.

In the end, experimenting with different techniques and practices is necessary to find suitable methods depending on the context. But using scientifically-based techniques is always a good starting point. Then, the question simply becomes what kind of quality and accuracy you're after. Keep in mind that the more simplifications and shortcuts you include, the lower the quality and accuracy you should expect from your experiments.

What to Measure

To measure video latency, we must first know what to measure. In other words, what is the measurement object ? Finding this object is a process of filtering out irrelevant factors and can be different things depending on the context. For instance, one measurement object could be interpreted as being only between the transmission computers, while another could be from time-critical source to PC-screen, or from TV-screen to TV-screen, etc.

For networked performances, it's common to consider the entire video chain as the measurement object, known as the glass-to-glass or end-to-end latency. The end-to-end latency is the time it takes for an image to be sent from one camera until it's displayed on-screen on the remote location end, including the time it takes the image to travel to from the screen to the eyes of the performers. This metric is particularly valuable as it informs us what it's actually like to be a performer in the given scenario.

However, the image latency is not all that matters in a performance. Aesthetic factors, such as the image quality and framerate, are equally important for producing quality concert experiences. In fact, aesthetic factors are probably what matters most. This is why we should only care about the latency of a video transmission if the quality of that transmission is above a certain standard. This will ensure that the latency you are measuring is representative of a real-world scenario that is aesthetically acceptable for audiences to sit through. For clarity, we can call this expanded definition for the Stable end-to-end latency.

It should be stated that aesthetic video factors are usually network dependent, specifically on its bandwidth limitations. Therefore, always get to know the bandwidth of the performance network and dial-in aesthetic factors before doing any kind of latency measurements.

Measurement Strategy

After the measurement object is determined, it's time to construct a measurement strategy. There are two main strategies we can use when measuring latency in networked systems, standard one-way and Round-Trip Time (RTT). When adopting an RTT strategy for video latency measurements, the delayed image is sent to the remote location and looped back to the source before being measured (Carôt 2009). The downside of the RTT strategy is that it requires the same technical setup in each location to know the one-way latency. And from experience, similar technical setups are rare.

Alternatively, it's possible to adopt a one-way measurement strategy as demonstrated in the hand-clapping diagram in the How to measure section above. This method can be suitable if the performance locations are particularly close, such as in the same building, or use the same local network.

On the other hand, a one-way measurement strategy is challenging to perform if the performance locations are remote. But it's not impossible. For example, if you have access to a common sync source, like a global clock, a video feed of the clock can be used as the time-critical source and compared against a local instance of the same clock. However, although this method has been proven viable in several cases (Clim 2021, Fiebrink 2016), global clock dependencies are risky in the field and can be problematic for the future validity of your measurements.

Regardless of which strategy you choose, it's wise to give some extra thought to designing the video signal chain in advance. Video transmissions usually require a huge amount of bandwidth, much more than audio, so all processing stages (like format conversions, encoding, decoding, etc.) will most likely have a heavy impact on the latency. Try to have as few hardware and software stages as possible in the signal chain. Also, the power of your streaming PC matters! If resources are scarce, always prioritize having your most powerful laptops or PCs do the video encoding. This can significantly improve latency.

Measurement Procedure

Setting up networked performance systems is usually pretty hectic. There are many concerns and very little time to manage them. Having a detailed plan can ensure precision and scientific validity in chaotic times. And the better the plan is, the less you have to think and stress when the iron is hot. Below I have sketched out an example plan with five individual steps that, when followed, will give accurate and precise video latency measurements.

Step 1 - Verify The Measurement System

First thing is first, set up and verify the measurement system. Use your designated measurement camera and film a simple clock or a timer for a few seconds. Then, verify the framerate and quality of your camera in the editing software. You can also go one step further and connect the networked performance camera to a local TV and do some test measurements. This can be time well spent to get used to running the experiments. This is also a good time to run some bandwidth tests and configure any network settings.

Step 2 - Fine-Tune The Video Quality

As mentioned, the key to unlocking a meaningful networked performances is to achieve high quality and stable video transmissions. Therefore, spend good time fine-tuning the video system and balance quality vs. latency vs. stability. This is an iterative process that involves adjusting various framerate, resolution and compression settings. A good way to analyze the video quality is to film the performers in action watch the result on-screen. Use your eyes.

Step 4 - Measure The Latency

Finally, you can measure the latency of your system. Since these measurement practices are essentially statistical exercises, so the more data you collect have, the better.

Step 5 (extra) - Measure With Alternative Parameters

In addition, it can be well worth collecting data with different settings to get a better understanding of the system and gain important clues to how to make it more efficient. For instance, experimenting with different resolutions, framerates, compression settings, and even without certain modules in the chain, will give you insights into how each component contributes to the overall latency.

To measure the latency with alternative parameters, make a table beforehand where you systematically list all the configurations you want to test.

Summary

In this post, I've shared some theories and practical strategies to achieve good video latency measurements in networked performance systems. The post has covered how to best measure the latency, what you should aim to measure, the strategies you can use, and what a good measurement procedure should look like.

To me, the three most important factors when measuring video latency in a networked performance systems are: 1) considering the entire signal chain from end-to-end, 2) ensuring the video system is producing glitch-free and acceptable video tranmissions (stable), and 3) using a scientific measurement method that accurately isolates the desired measurement object. It's also a good idea to plan ahead with concrete steps.

Good luck!

Sources

Carôt, A., & Werner, C. (2009). Fundamentals and principles of musical telepresence. Journal of Science and Technology of the Arts. https://doi.org/10.7559/CITARJ.V1I1.6

Clim, A. (2021). Video latency: definition, key concepts, and examples. The MCT Blog. Online at: https://mct-master.github.io/networked-music/2021/11/15/alenacl-video-latency-clarifications.html

Hope, C. (2024). Unit 6: Networked Music Performance. Mutor Music Technology Online Repository. Online at: https://mutor-2.github.io/HistoryAndPracticeOfMultimedia/units/06/index.html

Oda, R., & Fiebrink, R. (2016). The Global Metronome: Absolute Tempo Sync For Networked Musical Performance. Proceedings of the International Conference on New Interfaces for Musical Expression (NIME). https://doi.org/10.5281/zenodo.1176096

Ubik, S., & Pospíšilík, J. (2021). Video Camera Latency Analysis and Measurement. IEEE Transactions on Circuits and Systems for Video Technology, 31(1).