Markerless motion capture system opens up biomechanics for a wide range of fields

Markerless motion capture system opens up biomechanics for a wide range of fields

Researchers develop markerless motion capture system to push biomechanics "into the wild"

Researchers at CAMERA have developed technology that can analyze body movement from 2D video footage without the need for markers. Credit: University of Bath

Researchers at CAMERA, the University of Bath’s Centre for Analysis of Motion, Entertainment and Research Applications, have developed open access software that analyzes motion capture data, without using markers. They have shown the markerless system to offer clinicians, sports coaches and physiotherapists an unobtrusive way of analyzing body movements from video footage that is comparable to using markers.

Motion analysis traditionally relies on attaching light-reflective markers onto specific points on the body; the movement of these markers in 3D space is then calculated using data from an array of cameras that film the person’s movements from different angles.

Placing markers accurately on the body can be time-consuming to set up and can sometimes interfere with the person’s natural movements.

To overcome this, the team at CAMERA led by Dr. Steffi Colyer, has developed a non-invasive markerless system using computer vision and deep learning methods to measure motion by identifying body landmarks from regular 2D image data.

Using the same images to evaluate the performance of their fully automated system, they found the results were comparable to that of a traditional marker-based motion capture system. The system works on similar technology to that used by commercial systems, but is available as an open-source workflow and can be adapted to user’s needs more easily.

The team has released a unique dataset to allow other researchers to evaluate new markerless algorithms and further progress the fields of computer vision and biomechanics.

The team used an open source computer vision system, OpenPose, to estimate the position of the joints on a 2D video image of a person running, jumping and walking. They then fuse the data in 3D and input those data into open-source modeling software called OpenSim, which fits a skeleton to the joints and allows the whole body motion to be obtained.

The fully synchronized video and marker-based data used in this study, along with the code underpinning the markerless pipeline are now available and are fully described in a paper recently published in Scientific Data.

Researchers from CAMERA, the University of Bath’s motion analysis center, have developed mo cap technology that could be used by clinicians, physios and sports coaches to analyze body movement for free. Credit: University of Bath

Dr. Colyer said, “The trouble with using markers is that they can be tricky to place on a participant accurately and reliably and this process can take a long time, which isn’t very practical for many participants and applications (for example elite athletes or clinical populations).

“Our markerless system estimates the joint positions from video alone without the need for any equipment to be placed on the participant or any preparation time. This opens the door for us to capture motion data more readily in settings outside of the laboratory and the outcomes for the movements we analyzed are comparable to traditionally-used techniques with markers.

“Our pipeline is open source, which means that anyone with some expertise in the area can use it for free to get movement data from normal video footage.

“This could be useful for physiotherapists, clinicians and sports trainers in a wide range of applications including sports performance and injury prevention or rehabilitation. Additionally, the accompanying data set provides the first high-quality benchmark to evaluate emerging algorithms in this rapidly evolving field.

“We have used the system to measure the biomechanics of skeleton athletes during their push-starts and we have recently taken it out on to the tennis and badminton courts to unobtrusively monitor how much work the players are performing during training and match play.”

Markerless motion capture system opens up biomechanics for a wide range of fields Read More
10,000x Faster: AI Discovers New Microscopy Techniques in Record Time

10,000x Faster: AI Discovers New Microscopy Techniques in Record Time

XLuminA Automated Optical Discovery Process
Artistic visualization of XLuminA’s automated optical discovery process. The setup shows laser beams being guided through a network of optical elements including beam splitters, spatial light modulators and mirrors. This represents how XLuminA explores vast experimental configurations to discover novel super-resolution microscopy techniques. The glowing paths highlight the system’s ability to find optimal routes for light manipulation automatically, enabling breakthrough optical designs previously unexplored by human researchers. Credit: Long Huy Dao and Philipp Denghel

XLuminA, an AI framework, enhances super-resolution microscopy by exploring vast optical configurations, rediscovering established techniques, and creating superior experimental designs.

Discovering new super-resolution microscopy techniques often requires years of painstaking work by human researchers. The challenge lies in the vast number of possible optical configurations in a microscope, such as determining the optimal placement of mirrors, lenses, and other components.

To address this, scientists at the Max Planck Institute for the Science of Light (MPL) have developed an artificial intelligence (AI) framework called XLuminA. This system autonomously explores and optimizes experimental designs in microscopy, performing calculations 10,000 times faster than traditional methods. The team’s groundbreaking work was recently published in Nature Communications.

Revolution in Microscopy: The Rise of Super-Resolution Techniques

Optical microscopy is a cornerstone of the biological sciences, enabling researchers to study the smallest structures of cellular life. Advances in super-resolution (SR) methods have pushed beyond the classical diffraction limit of light, approximately 250 nm, allowing scientists to see previously unresolvable cellular details. Traditionally, developing new microscopy techniques has relied on human expertise, intuition, and creativity—a daunting challenge given the vast number of possible optical configurations.

For example, an optical setup with just 10 elements selected from 5 different components, such as mirrors, lenses, or beam splitters, can generate over 100 million unique configurations. The sheer complexity of this design space suggests that many promising techniques may still be undiscovered, making human-driven exploration increasingly difficult. This is where AI-based methods offer a powerful advantage, enabling rapid and unbiased exploration of these possibilities.

“Experiments are our windows to the Universe, into the large and small scales. Given the sheer enormously large number of possible experimental configurations, its questionable whether human researchers have already discovered all exceptional setups. This is precisely where artificial intelligence can help,” explains Mario Krenn, head of the Artificial Scientist Lab at MPL.

AI’s Role in Discovering New Optical Configurations

To address this challenge, scientists from the Artificial Scientist Lab joined forces with Leonhard Möckl, a domain expert in super-resolution microscopy and head of the Physical Glycoscience research group at MPL. Together, they developed XLuminA, an efficient open-source framework designed with the ultimate goal of discovering new optical design principles.

The researchers leverage its capabilities with a particular focus on SR microscopy. XLuminA operates as an AI-driven optics simulator which can explore the entire space of possible optical configurations automatically. What sets XLuminA apart is its efficiency: it leverages advanced computational techniques to evaluate potential designs 10,000 times faster than traditional computational methods.

“XLuminA is the first step towards bringing AI-assisted discovery and super-resolution microscopy together. Super-resolution microscopy has enabled revolutionary insights into fundamental processes in cell biology over the past decades – and with XLuminA, I’m convinced that this story of success will be accelerated, bringing us new designs with unprecedented capabilities,” adds Leonhard Möckl, head of the Physical Glycoscience group at MPL.

Carla Rodriguez Crop
Dr. Carla Rodríguez, scientist in the research group of Dr. Mario Krenn at MPL. Credit: Jan Olle

XLuminA: A Breakthrough in Optical Simulation

The first author of the work, Carla Rodríguez, together with the other members of the team, validated their approach by demonstrating that XLuminA could independently rediscover three foundational microscopy techniques. Starting with simple optical configurations, the framework successfully rediscovered a system used for image magnification.

The researchers then tackled more complex challenges, successfully rediscovering the Nobel Prize-winning STED (stimulated emission depletion) microscopy and a method for achieving SR using optical vortices.

Finally, the researchers demonstrated XLuminA’s capability for genuine discovery. The researchers asked the framework to find the best possible SR design given the available optical elements. The framework independently discovered a way to integrate the underlying physical principles from the aforementioned SR techniques (STED microscopy and the optical vortex method) into a single, previously unreported experimental blueprint. The performance of this design exceeds the capabilities of each individual SR technique.

“When I saw the first optical designs that XLuminA had discovered, I knew we had successfully turned an exciting idea into a reality. XLuminA opens the path for exploring completely new territories in microscopy, achieving unprecedented speed in automated optical design. I am incredibly proud of our work, especially when thinking about how XLuminA could help in advancing our understanding of the world. The future of automated scientific discovery in optics is truly exciting!” says Carla Rodríguez, the study’s lead author and main developer of XLuminA.

Expanding the Capabilities of Microscopy Through AI

The modular nature of the framework allows it to be easily adapted for different types of microscopy and imaging techniques. Looking forward, the team aims to include nonlinear interactions, light scattering, and time information which would enable the simulation of systems such as iSCAT (interferometric scattering microscopy), structured illumination, and localization microscopy, among many others. The framework can be used by other research groups and customized to their needs, which would be of great advantage for interdisciplinary research collaborations.

Reference: “Automated discovery of experimental designs in super-resolution microscopy with XLuminA” by Carla Rodríguez, Sören Arlt, Leonhard Möckl and Mario Krenn, 10 December 2024, Nature Communications.
DOI: 10.1038/s41467-024-54696-y

10,000x Faster: AI Discovers New Microscopy Techniques in Record Time Read More