Motion-capture technology for USAR: only a matter of time

Published:  05 November, 2014

Dr Ian Greatbatch and Dr Andrea Kleinsmith argue the case for integrated technology in fire and rescue to sense responders’ vital functions and location.

Since the turn of the century most western governments have embraced the concept of urban search and rescue, although of course the concept of training to extricate survivors from collapsed structures can be traced back to the early days of modern firefighting operations and in particular lessons learned in the UK during the Blitz. Although a residual capability has been ever present globally it is only in the last 20 years that specialist units have been developed.

As we all know, the work of USAR teams has always been arduous and complex. They could be working at extreme height, suspended from cranes, or in the superstructures of skyscrapers, or more commonly in the hazardous environment of a collapsed building. This environment is obviously dusty, can contain toxic materials such as asbestos or combustible or noxious gases released by the collapsed structure. As USAR teams face the risk of being crushed or trapped by further collapse, they build shoring and use construction tools and techniques to attempt to make their environment safer.

This in turn means working in cramped conditions whilst using heavy breaking and construction equipment, which is unwieldy, heavy, vibrating and (in the case of nail guns) potentially a lethal weapon in its own right. In some cases this equipment has to be used whilst using BA sets, or even hanging in SWAH (safe working at height) kit, having been lowered into position from above. In all USAR jobs, as in firefighting in general, there is a requirement for personnel to wear PPE at all times and this can itself be heavy, uncomfortable and cumbersome, potentially leading to longer term strain on crew.

Clearly all of these factors make the role of an USAR technician difficult and with the potential for injury – either short term as a result of a further collapse or accident in the response, or longer term from an injury resulting from repetitive use of heavy equipment with poor posture. On top of the physical risks of injury, the wider circumstances surrounding USAR work can have other impacts. The likelihood is, given the nature of the events, that there will be significant numbers of traumatised casualties, fatalities and traumatised survivors on scene, which can in turn psychologically traumatise responders. This trauma may be extremely hard to spot, and has the potential to flare up without warning at some point in the future.

This we all know already.

The problem is that monitoring of the physical and emotional demands of firefighters is technically difficult to achieve, in real time and in situ – ie the fireground. Vibration and dust affects common monitoring equipment, the cramped conditions and distance away from the command post also makes wired monitoring equipment ineffective. We could wear heart and breathing rate monitors under tunics, but these alone only give a limited picture of overall physical performance.

Psychological monitoring is even trickier. It requires trained staff to carry it out, buy-in from operational staff and the time and money to run the program effectively. To further compound the situation, rescue workers are notoriously difficult to talk into taking a break. They will often minimise how tired or upset they are feeling in order to remain on the job, often well beyond what is safe for them or – worse – beyond their capability to properly help those trapped. Their drive to help people and be at the heart of the action is why they do the job they do, but it does make managing their mental and physical health more difficult.

 This all comes within the context of a fire and rescue sector, where firefighters are being asked to work longer, and where resources are precious; so losing firefighters to avoidable long term injuries should be a priority.

But what if we could monitor the physical and emotional state of USAR workers, unobtrusively, remotely and without wires or bulky sensors? What if we could tell the emotional state of a firefighter, their levels of fatigue, or even how upset they were? What if we could pick up on activity likely to cause a long term injury (such as using a powertool in a stress position) and either instruct the firefighter to stop, or rotate operators frequently enough to minimise damage? What if we could tell if a firefighter was too tired to continue safely, despite what they were telling us?

The same kind of technology that allows actors to portray apes, dinosaurs, robots or mythical creatures such as Gollum in movies may be part of the answer to this problem. Motion capture suits are routinely used in the video game and movie industries to capture the physical movement of people and electronically capture it to apply it to a CGI model on screen.

If this motion capture technology was used to track body movements, innovative Affective Computing techniques could then be applied to that body movement data to identify the physical and psychological states of firefighters and USAR workers. It could be used to inform a crew manager or on-scene commander that a firefighter was either engaging in a potentially damaging activity, or were too tired or too upset to continue. It could also work in a similar way to the Draeger Bodyguard BA Safety system, and identify when firefighters are unconscious or trapped in a more sophisticated way.

This technology could also be expanded to monitor crew to ensure that they were not too tired, upset or even intoxicated to operate machinery of vehicles. We could have ‘smart’ machinery that refused to work for any operator that was not behaving in a normal manner.

The term Affective Computing was coined by Professor Rosalind Picard at MIT in the late 1990s.  One branch of Affective Computing focuses on building systems that automatically recognise a person’s emotional and mental states through different kinds of verbal and nonverbal communication channels.  We recently proposed a system, whereby motion capture sensors could be sewn into the leggings and tunic and fitted within the helmet of firefighters to track body expressions as the nonverbal communication channel in order to build such a system.

Although motion capture technology has been around for a long time, it has not been possible to use it very easily in the real world situations in which firefighters often work. The sensors themselves were quite bulky, heavy, and required cabling between the sensors and the recording hardware.  However, motion capture technology has advanced considerably in the last few years and there are now systems on the market that are much smaller and more lightweight than previous systems, and can operate via wifi.

Many of these sensors combine contain accelerometers and gyroscopes, thus giving the ability to track not only the numerical position of the sensors on the body in 3D space, but also the speed of movement as well as angle rotations of different body joints.  This allows for a complete, precise representation of the body to be tracked and recorded.

Affective Computing uses artificial intelligence techniques to automatically associate different emotional and psychological states with different body expressions; just like humans do in real face-to-face interactions.  We start with human behaviour experts who would review example body movement recordings and judge the state of the person. This judgment gets combined with the numerical representation of the body and then a model is built. Once the model is trained it can be used in real time to recognize others.

The technology currently exists, but now we need to do the research to understand how best to apply it to the domains of USAR, firefighters and other emergency services. There is a virgin market out there for a genuinely new piece of firefighting technology. In our opinion it is only a matter of time before this becomes a standard component of firefighter PPE, sewn into every firefighter tunic in the world; the question is – who will exploit it first?

The authors

Dr Ian Greatbatch is Senior Lecturer in GIS at Kingston University London, where he works primarily in operational research concerning search and rescue, fire and the emergency services.

Dr Andrea Kleinsmith is a Postdoctoral Research Fellow in the Virtual Experiences Research Group at the University of Florida where she investigates the use of virtual humans in simulation training; examining the role that virtual patients can play in training medical students' interpersonal communication skills.

  • Operation Florian

Sign up: eMagazine & eNewsletter

The latest issues in your inbox.

Company Profiles

Renka´s Fire Engine No. 1

Firefighting, fast, safe, everywhere. Michael Renka GmbH, based in Germany, is a manufacturer of firefighting vehicles, pumps and rescue equipment.

Waterax - We move water

Trusted by wildland firefighters around the world, WATERAX sets the industry standard by developing innovative, portable fire pumps and water-handling equipment designed to withstand demanding applications and rugged environments.

Big Water Flow for Industrial and Municipal Firefighting Applications

Protecting the lives of the public and firefighters while limiting the structural damage caused in large scale fires is our primary mission

We are committed to improving lives and doing business in the right way

We have a unique mix of capability and culture that we refer to as 3M Science and we strive to develop products that improve people’s daily lives in a multitude of ways.

HazSim - Bringing situational HazMat training to life

HazSim, LLC provides innovative simulation training to ensure your team works safely and effectively. HazSim Pro simulation equipment is in use by hundreds of fire departments, training schools, industrial fire teams, and private trainers across the US, Canada and further afield as well as the US Army.

The ultimate in innovation, quality and service

For 60 years Lehavot has been delivering the world’s most advanced fire detection and suppression automatic systems

Advancing rescue technology

The specialist supplier of quality PPE and Rescue Equipment to Emergency Services.

Revolutionizing fire fighting foam technology

The one-stop resource for fire fighting foam concentrates and custom-designed foam suppression systems hardware.

Trust the best, let us be your foam solution

AUXQUIMIA is a Spanish company whose main activity is the design, manufacture and commercialization of firefighting foam concentrates.

Williams Fire & Hazard Control offers a full line of specialized fire response equipment for oil and gas platforms

From storage tanks and pipeline emergencies to offshore platforms and vessels at sea, Williams' response personnel and specialized equipment quickly address adverse fire emergencies.

The leader in truck-mounted hydraulic platforms

Our mission is to provide the best and the safest solution to professionals that work at height.

If you want quality, you want Zico

Since its inception Ziamatic Corp has provided the men and women of the fire service with products designed to make their jobs safer and easier.

The independent alternative

Dafo Fomtec AB is a privately owned company with head office in Stockholm Sweden and manufacturing in Helsingborg in the south of Sweden.