Runway Assignments

Assigning flights to runways at an airport is a crucial function that influences airport operations and aircraft performance. Air traffic controllers develop these plans and set the runway a flight must land or depart from. As with any operational decision-making process, there are many contributing factors, constraints, and events to account for, including airport configuration, the configuration of neighboring airports, aircraft size and engine type, noise/environmental regulations, and traffic volumes. Controllers must consider requests from flight crews for specific runways out of operational necessity and preference. The assignment rules evolve as new procedures and routes are implemented. As part of the NASA-led ATD-2 project, Mosaic developed an ML-driven approach to predict the runway assignments that controllers would make.

Past Approaches

Previous research on runway assignments has focused on optimizing the process (e.g., to improve throughput). While that body of work suggests potential benefits, there are significant barriers to implementation. Other work, including earlier in ATD-2, has focused on interviewing operational personnel to learn the underlying heuristic that they use in making runway assignments. This approach is costly because of the time and effort involved, makes capturing exceptions and interactions in the rule-based approach difficult, and can be difficult to update as the operation evolves.

The approach in this work assumes that controllers will continue to use their expertise and available automation tools. The machine learning models will replicate the runway assignment heuristics used by controllers. By developing this data-driven approach and proving its effectiveness against the existing static approaches already developed for some airports, runway assignment models can then be easily developed for additional airports at low cost.

Modeling Approach

To develop an accurate machine learning model, Mosaic’s aviation data scientists needed to model the decision heuristics used by controllers in their runway assignments. One of the critical objectives in this research is to have the ability to deploy the model in a real-time system to compare performance against existing ATD-2 approaches to evaluate future automation and decision support features. As outlined above, Mosaic wanted to apply a genuine data-driven approach, meaning it does not require significant adaption data maintenance. These objectives were selected to allow the models to be tested against dozens of airports across the U.S., provided that the data formats matched the training data. Mosaic generated data formats using the ATD-2 data fuser to cut down on feature and data engineering efforts.

The modeling process for the arrival and departure runway assignment problem was developed to be generic and quickly deployable to several U.S. airports.

The overall machine learning flow followed standards set out by  CRISP-DM and ASUM-DM. The CRISP-DM can be seen in the figure below.

Mosaic identified a target value as the historical runway identifier used by the flight. Feature engineering was developed based on literature review, brainstorming, and most importantly, collaboration with available air traffic controllers. After an exploratory data analysis, additional filtering was conducted to prepare the data for the model training algorithm. This filtering process picked up on anomalies, which skew results and imputed missing values deemed critical to the algorithm. The training data was now ready to be run into a machine learning algorithm.

Feature	Arrivals	Departures
Aircraft engine class	x	x
Wake turbulence category	x	x
Planned fix	x	x
Airport configuration	x	x
Time since airport configuration changed	x
Time until estimated operation	x
TBFM-assigned runway	x

ML Prototyping

The problem lends itself to multinomial classification algorithms. There are several classification algorithms, and Mosaic’s aviation data scientists tested several, including a classic logistic regression (logit), and XGBoost. These results are needed to be tested against a baseline to determine performance metrics, and Mosaic reported the most common target value in the training dataset as the prediction. More sophisticated baseline heuristics and classification algorithms will be tested against in future iterations.

Preliminary Predictive Results

As the research is still ongoing, all the below results are preliminary but indicate the potential benefits of a machine learning approach to the problem. It is important to note these results were generated during the COVID-19 pandemic, which led to significant disruptions for aviation operations. Mosaic observed significant differences in the data and took these effects into the presentation of the results.

Predicting Runway Assignments | Arrivals

Initial models focused on Dallas-Fort Worth International Airport, as it is a test site for the systems being developed on the ATD-2 project. The accuracy can be represented in terms of a percentage of correct predictions made by the models. Initial results for the two ML models are listed below.

Airport Identifier	Logistic Regression Accuracy	XGBoost Classifier Accuracy
KDFW	.700	.669

Predicting Runway Assignments | Departures

The departure models were validated against airports listed in the table below. Again, the accuracy metrics can be represented as a percentage of how well the model projected the departing runway.

Airport Identifier	Accuracy
KSFO	.751
KEWR	.891
KORD	.712

Mosaic explored additional airports using pre-pandemic data and found similar accuracy scores. Several improvements are planned to expand and solidify these preliminary results.

Predicting Runway Assignments vs ATD-2 Baseline

The air traffic controller runway assignments are used in the ATD-2 systems at Charlotte Douglas International Airport and Dallas-Fort Worth. As part of the project, these models are regularly updated with controller feedback and provided a great baseline to compare the ML values against. Performance is displayed below.

Airport Identifier	Departure Accuracy	Arrival Accuracy
KCLT	0.933	0.876
KDFW	0.837	0.549

Several trends are apparent from the baseline results. The departure runway accuracy exceeds what is achieved with current ML models, meaning the ML models need to include more features, model types, and further tuning. The arrival accuracy performance at Dallas-Fort Worth indicates that the decision heuristics employed by controllers have likely changed, but the baseline models used in the ATD-2 system have not been updated to reflect these new practices.

Predicting Runway Assignments Future Improvements

Machine learning shows initial promise for learning how controllers make runway assignment decisions when aircraft land or depart, which was the goal of this research effort. The overall approach enables broad deployment across U.S. airports by following careful data preparation and modeling flows. Fused with other models to predict NAS operations, Mosaic and NASA believe this data-driven ML approach will facilitate automated testing and deployment of advanced predictive decision-support tools.