Mosaic ATM developed a scalable, camera-based framework to better understand fatigue progression in safety-critical transportation operations.
Take Our Content to Go
Fatigue is not an on-or-off condition. It develops gradually, manifests differently from person to person, and can become difficult to recognize before performance begins to deteriorate.
For safety-critical transportation operations, that creates a difficult technology challenge. A monitoring system must do more than recognize an isolated eye closure or head movement. It needs to interpret multiple behavioral signals over time, distinguish meaningful changes from normal variation, and provide information operators and organizations can trust.
Through the Fatigue Assessment for Transportation Engineer Determination (FATED) Small Business Innovation Research project sponsored by the Federal Railroad Administration (FRA), Mosaic ATM explored a new approach to that challenge. Phase I focused on developing and evaluating a computer vision-based framework for detecting fatigue and reduced vigilance in locomotive engineers using non-intrusive video monitoring.
Rather than treating fatigue as a binary fatigued/not fatigued determination, Mosaic developed the foundation for a system designed to characterize operator readiness as a continuum.
The Challenge: Looking Beyond Simple Fatigue Alerts
Many fatigue-monitoring approaches rely on individual indicators or predetermined thresholds: an eye remains closed for a certain period, a person’s head reaches a specific angle, or another predefined condition triggers an alert.
Those techniques can provide useful information, but fatigue is more complicated.
Phase I research reinforced two particularly important challenges. First, people express fatigue differently, making universal thresholds difficult to apply reliably. Second, systems that generate too many nuisance alerts can undermine user confidence, an especially important consideration when monitoring technology is introduced into an operational workplace.
For FATED, Mosaic’s goal was therefore broader than detecting isolated signs of drowsiness.
The research asked: Can multiple visual indicators be combined over time to provide a more nuanced and interpretable picture of changing operator readiness?
Building on Mosaic’s Transportation AI Experience
FATED builds on computer vision and driver-readiness research Mosaic previously conducted through the Visual Integration of Language Models in Automotive Safety (VILMAS) project.
VILMAS explored techniques including gaze estimation, eye-state monitoring, computer vision, deep learning, and visual-language models for understanding driver behavior and environmental context. FATED extended portions of that foundation into the rail domain while shifting the emphasis from driver distraction and situational context toward fatigue progression and sustained vigilance.
This continuity allowed Mosaic to build on previously investigated technologies while focusing Phase I resources on one of FATED’s central research questions: how to transform individual computer vision observations into a meaningful assessment of an operator’s changing state.
Mosaic’s Approach: Multiple Indicators, One Readiness Framework
During Phase I, Mosaic developed an end-to-end computer vision pipeline capable of processing video and extracting a range of fatigue-related behavioral indicators.
The approach combines facial analysis with body-posture estimation. Facial landmarks provide information related to eye activity, gaze, mouth movement, head position, and facial behavior, while skeletal pose estimation provides additional information about posture and movement.
Figure 1. FATED operator-readiness conceptual framework showing how multiple mental, physical, and behavioral indicators can contribute to the progression from fatigue and drowsiness to sleepiness and sleep.
Instead of allowing any one observation to determine whether someone is fatigued, FATED organizes those observations into multiple levels of analysis:
- Low-level visual signals capture frame-by-frame measurements such as eye state, gaze direction, facial landmarks, head pose, and body position.
- Mid-level behaviors examine how those signals evolve over time to identify patterns including blink frequency and duration, prolonged eye closure, yawning, nodding, gaze changes, and posture changes.
- Higher-level assessment is designed to combine those behaviors over longer periods into an interpretable measure of operator readiness—reflecting progression rather than a single yes/no fatigue determination.
That distinction is central to the FATED concept.
A single yawn may mean very little. A changing combination of yawning frequency, blink behavior, head movement, eye closure, and posture over time may provide considerably more useful information.
Turning Video into Interpretable Fatigue Indicators
To evaluate the concept, Mosaic standardized and analyzed more than 30 hours of publicly available drowsiness video representing 60 participants across self-reported alert and fatigue conditions. The dataset also illustrated an important human-factors challenge: individuals do not exhibit alertness or fatigue in precisely the same way.
Initial Phase I analysis identified observable differences between alert and fatigued recordings in behaviors such as yawning, blink characteristics, eye closure, and facial activity. These findings were consistent with indicators identified through the project’s human-factors research and provided early evidence that the selected features could help distinguish changing fatigue states in realistic video.
Figure 2. Example of the FATED Phase I computer vision pipeline extracting facial landmarks and behavioral features associated with fatigue, including gaze, head position, eye activity, and mouth movement.
Mosaic also created an analytics environment to examine those signals over time. Rather than relying exclusively on instantaneous measurements, the team evaluated the temporal structure of fatigue-related behaviors. Time-series and time-frequency analysis helped reveal patterns associated with behaviors such as prolonged eye closure, head nodding, and changing eye dynamics that may be difficult to capture through simple threshold rules alone.
This analytical capability created a bridge between raw computer vision output and the eventual higher-level readiness model.
Designing for Individual Differences
One of the most important findings from Phase I was also one of the most intuitive: there may not be a single definition of “normal” behavior that works equally well for everyone.
The analysis showed meaningful variability in how individuals expressed fatigue. As a result, the FATED framework includes the ability to consider individual baselines and calibration rather than relying entirely on fixed population-wide thresholds.
That approach could ultimately help a system distinguish between an individual’s ordinary behavior and a meaningful change in that behavior.
It is also important from a human-factors perspective. In operational environments, confidence in an automated assessment depends not only on algorithmic performance but also on whether people understand what the system is measuring and perceive its conclusions as credible.
A Modular Architecture Built for Continued Development
FATED was designed as more than a collection of research algorithms.
Mosaic implemented a modular video-processing architecture that moves information from camera input through detection, tracking, feature extraction, temporal aggregation, higher-level analysis, and eventual downstream alerting or visualization.
Breaking those capabilities into independent services allows individual algorithms to be improved or replaced without redesigning the entire system. It also provides a foundation for integration with different cameras, computing platforms, dashboards, APIs, and potentially additional sensing technologies in the future.
Phase I successfully demonstrated this end-to-end architecture with both live and recorded video, establishing a foundation for continued maturation toward real-time edge deployment.
What Phase I Demonstrated
Phase I was a feasibility effort rather than an operational deployment, but it answered several important research questions.
Mosaic demonstrated that multiple fatigue-related visual cues, including eye behavior, facial movement, gaze, head position, and posture, can be extracted concurrently from conventional video. The team established an initial library of low- and mid-level fatigue indicators, demonstrated measurable differences between alert and fatigued recordings, built an analytical environment for studying how those indicators change over time, and developed a modular software architecture capable of supporting further system maturation.
Just as importantly, Phase I identified areas requiring additional work.
Testing exposed computational constraints when multiple vision models were executed simultaneously on CPU-only hardware. Rather than treating this simply as a limitation, the team used the finding to define a clear optimization path involving hardware acceleration, model scheduling, and edge-computing optimization.
That is a fundamental purpose of early-stage R&D: identifying technical risk while there is still an opportunity to engineer around it.
Keeping Humans at the Center of Safety AI
The FATED research also reinforced that a successful fatigue-monitoring system cannot be designed around algorithmic accuracy alone.
Trust, explainability, privacy expectations, intervention strategies, and individual variability all influence whether monitoring technology will ultimately be accepted and useful in an operational environment.
For that reason, the recommended next stage of FATED research places significant emphasis on individual calibration, stakeholder engagement, careful design of intervention strategies, and explainability tools that could show which indicators contributed to a readiness assessment rather than simply presenting an unexplained alert.
For Mosaic, that principle extends beyond rail.
Effective artificial intelligence in safety-critical systems must combine machine perception with human factors, operational context, and transparent decision support.
Looking Ahead
Phase I established the technical and human-factors foundation for advancing FATED from a research framework toward a more mature rail-grade capability.
Recommended future work includes further validation across individuals and operating conditions, probabilistic fusion of multiple fatigue indicators, individualized calibration, optimization for real-time edge computing, refinement of intervention strategies, and expanded engagement with rail stakeholders.
The underlying architecture may also have applications beyond rail wherever operators must maintain vigilance over long periods, including fleet operations, heavy equipment, trucking, maritime transportation, and other safety-critical environments.
FATED demonstrates the broader capability Mosaic ATM brings to transportation research: combining computer vision, machine learning, systems engineering, data analytics, and human-factors expertise to move emerging AI concepts from research questions toward operationally meaningful solutions.
Mosaic ATM develops practical AI, machine learning, and decision-support technologies for complex transportation environments. Contact our team to learn how we can support your next research, prototyping, or operational analytics challenge.
