# Donghwan Shin

Senior Lecturer (Associate Professor) in Software Engineering, School of Computer Science, University of Sheffield, UK.
Email: d.shin@sheffield.ac.uk

I am a Senior Lecturer (Associate Professor in the American system) in the School of Computer Science at the University of Sheffield, a member of the AI x Software Engineering & Testing (ASET) group, and a PI of the Intelligent Testing Lab. I received my PhD, MS and BS in Computer Science from KAIST, advised by Prof. Doo-Hwan Bae, and then spent four years as a research associate and research scientist in Prof. Lionel Briand’s SVV group at the SnT Centre, University of Luxembourg.

## Research

### Testing for AI

Making AI-enabled autonomous systems, such as self-driving cars, safe and reliable through simulation-based, search-based and multi-fidelity testing, failure simplification, and debugging.

### Mutation Testing

Understanding whether artificial faults can stand in for real ones, and making mutation analysis effective and scalable, from theory to efficient tools for Rust and safety-aware mutation for ADS.

### Log Analysis

Using software execution logs to solve software engineering problems, including model inference, log parsing, anomaly detection, and failure prediction.

## Publications

### 2026

- **Branch-Level Fault Localization in ADS Planning via Temporal Coverage Analysis**. Sangmin Woo, Dohyun Kim, Donghwan Shin, Yongdae Kim. *ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2026)*, 2026. [doi](https://dl.acm.org/doi/10.1145/3832135)
  Summary: Fault localisation for the planning module of autonomous driving systems is hard because faults show up only over time in a driving scenario. This work localises faults at the level of code branches by analysing coverage over time during failing and passing runs. Key implication: developers can trace an ADS planning failure back to specific branches instead of whole modules.
- **Evolution-Aware Regression Test Prioritization of ML-Enabled Systems Using Gradient-Based Behavior Vectors**. Eunho Cho, Donghwan Shin, In-Young Ko. *IEEE/ACM International Conference on Automated Software Engineering (ASE 2026)*, 2026. [doi](https://dl.acm.org/doi/10.1145/3832783.3834368)
  Summary: ML components change through retraining and fine-tuning, so a single update can hurt some tests and help others. GBV-PD represents each test by a compact projection of its loss gradient and checks how the model's parameter update aligns with it, so tests can be prioritised without running the updated model. Key implication: regression testing of evolving ML systems becomes cheaper, with cached vectors reused across updates.
- **DT-Drive: A Tool for Deterministic Replay-Based Testing and Debugging of Autonomous Driving Systems**. Sanjeetha Pennada, Matthew Leach, Olek Osikowicz, Donghwan Shin. *ASE 2026 Tools and Demonstrations Track*, 2026. [doi](https://dl.acm.org/doi/10.1145/3832783.3834601)
  Summary: Simulators such as CARLA can give different results on each run, which makes ADS test outcomes unreliable. DT-Drive records a scenario, separates the ego vehicle from it, and replays it deterministically, so different driving agents can be injected into the same scenario or conditions such as weather can be changed. Key implication: it gave 100% deterministic results on 128 CARLA benchmark scenarios, enabling fair comparison and counterfactual debugging.
- **A Comprehensive Empirical and Theoretical Analysis of Batching Algorithms for Efficient, Safe, Parallel Mutation Analysis in Rust**. Zalán Lévai, Donghwan Shin, Phil McMinn. *ACM Transactions on Software Engineering and Methodology*, 2026. [doi](https://dl.acm.org/doi/10.1145/3787851)
  Summary: Mutation analysis is rarely used for Rust because valid mutants are hard to generate and running them is slow. This work evaluates many non-interacting mutations together in the same process (mutation batching), with fast approximation algorithms for the NP-hard batching problem, implemented in mutest-rs. Key implication: across 22 Rust programs, batching cut mutation analysis runtime by up to 52.3%.
- **Dynamic Mutation Scheduling: Highly Parallel, Efficient Evaluation of Mutations for Rust Programs through Program Splitting**. Zalán Lévai, Donghwan Shin, Phil McMinn. *IEEE International Conference on Software Testing, Verification and Validation (ICST 2026)*, 2026. [pdf](https://drive.google.com/file/d/1Wis8ZQDiAAPQQf_wmoFjrpefSLg6JKnK/view?usp=sharing)
  Summary: Static mutation batching leaves threads idle while the slowest mutation in a batch finishes. Dynamic Mutation Scheduling instead assigns compatible mutations, those touching distinct parts of the program, to workers as soon as they become free. Key implication: on 25 Rust programs it reduced wasted time and evaluation runtime by up to 75.6% compared with static batching.
- **mutest-rs: Flexible, Efficient Mutation Analysis Tool for Rust Programs, using Extensive Static Analysis**. Zalán Lévai, Donghwan Shin, Phil McMinn. *ICST 2026 Tools and Demo Track*, 2026. [pdf](https://drive.google.com/file/d/19LgFhVSDLZ0MdAwHTkcbij-WKh17LLHz/view?usp=sharing)
  Summary: No mature mutation analysis tool existed for Rust. mutest-rs integrates with the rustc compiler and uses extensive static analysis to generate valid mutants, plus Rust-specific meta-mutants for efficiency. Key implication: Rust developers get a practical, open-source way to measure how thorough their tests are (mutest.rs).
- **Multi-Fidelity Bayesian Optimization for Simulation Based Autonomous Driving Systems Testing**. Olek Osikowicz, Phil McMinn, Wei Xing, Donghwan Shin. *IEEE Intelligent Vehicles Symposium (IV 2026)*, 2026. [pdf](https://drive.google.com/file/d/1Lz68CE63_qdQI8fcoRdEWkxmejL-v6DB/view?usp=share_link)
  Summary: Realistic driving simulations take minutes to hours each, which limits simulation-based ADS testing. MFBO-Drive treats critical scenario generation as multi-fidelity optimisation, choosing both which scenario to run and at what fidelity. Key implication: across more than 100,000 MetaDrive scenarios it improved cost-effectiveness by 16.8% over state-of-the-art Bayesian optimisation with comparable test quality.
- **Simulation Nondeterminism and Its Impact on Flaky Tests for Autonomous Driving Systems**. Olek Osikowicz, Phil McMinn, Donghwan Shin. *Automated Software Engineering (Springer), accepted*, 2026.
  Summary: Simulation-based ADS testing assumes the simulator is deterministic. This study reruns benchmark scenarios in CARLA and MetaDrive and traces where executions diverge. Key implication: 43.6% of CARLA scenarios were flaky (versus 3 of 1,000 in MetaDrive), mostly due to scenario initialisation, so ADS test results must be checked for nondeterminism before they are trusted.
- **Automated Testing of Prevalent 3D User Interactions in Virtual Reality Applications**. Ruizhen Gu, José Miguel Rojas, Donghwan Shin. *Automated Software Engineering (Springer)*, 2026. [doi](https://link.springer.com/article/10.1007/s10515-026-00620-1)
  Summary: Existing VR testing tools handle simple point-and-click events, not realistic 3D interactions such as grabbing or socketing objects. The authors identify four common interaction types, model them as an Interaction Flow Graph, and build XRintTest to exercise them automatically, evaluated on a new 456-interaction benchmark. Key implication: XRintTest covered 93% of interactions versus 8% for random input and found real interaction bugs.
- **Collaborative Multi-Agent Testing for Emergent Failure Discovery in Autonomous Driving Systems**. Ruizhen Gu, Konstantinos Koufos, Donghwan Shin, Vahid Garousi, Mehrdad Dianati. *1st Workshop on Generalization in Autonomous Driving (co-located with ICRA 2026)*, 2026. [pdf](https://drive.google.com/file/d/1zNF7nSfAvRtBa1SVUWHPUcQi7p7CQZm1/view?usp=share_link)
  Summary: ADS failures often emerge from interactions between perception, planning and control rather than from one module. CREAD coordinates a perception fuzzer, a metamorphic validator and an orchestrator agent through a shared blackboard. Key implication: in HighwayEnv it found about 2.1 times more failures per 100 scenarios than a single-agent baseline (work in progress).
- **Towards Safety-Aware Mutation Testing for Autonomous Driving Systems**. Donghwan Shin. *arXiv preprint*, 2026. [arxiv](https://arxiv.org/abs/2606.26456)
  Summary: ADS testing lacks a principled answer to when testing is enough, and coverage metrics miss the component interactions behind many crashes. This vision paper proposes injecting time-limited faults into the messages between driving modules, with mutation rules derived from STPA safety analysis. Key implication: mutation testing could give ADS a safety-grounded adequacy criterion and guide scenario generation and repair.
- **A Systematic Mapping Study on the Debugging of Autonomous Driving Systems**. Nathan Shaw, Sanjeetha Pennada, Robert M Hierons, Donghwan Shin. *arXiv preprint*, 2026. [arxiv](https://arxiv.org/abs/2601.04293)
  Summary: Testing of autonomous driving systems is well studied, but debugging after a test fails is not. This systematic mapping study surveys fault localisation and repair approaches for ADS, identifies gaps, and proposes common terminology. Key implication: the field is fragmented but promising, and the study gives a map and a research agenda for ADS debugging.
- **Detecting Multiple Semantic Concerns in Tangled Code Commits**. Beomsu Koh, Neil Walkinshaw, Donghwan Shin. *arXiv preprint*, 2026. [arxiv](https://arxiv.org/abs/2601.21298)
  Summary: Developers often bundle several unrelated changes into one commit, which makes history hard to understand. This work treats detecting the concerns in a tangled commit as multi-label classification and evaluates small language models that can run locally. Key implication: a fine-tuned small model beat a state-of-the-art LLM, and adding commit messages improved accuracy by up to 31% at negligible cost.

### 2025

- **Using Causal Inference to Test Systems with Hidden and Interacting Variables: An Evaluative Case Study**. Michael Foster, Robert Hierons, Donghwan Shin, Neil Walkinshaw, Chris Wild. *International Conference on Evaluation and Assessment in Software Engineering (EASE 2025)*, 2025. Distinguished Paper Award. [doi](https://dl.acm.org/doi/10.1145/3756681.3756967)
  Summary: Causal testing works well for complex, nondeterministic systems, but only when all relevant variables are observed and do not interact. This work adds effect modification and instrumental-variable methods to a causal testing tool and evaluates it on three requirements of the CARLA simulator. Key implication: reliable test outcomes are possible even with hidden and interacting variables, without heavy instrumentation or tightly controlled data.
- **On the Promises and Challenges of AI-Powered XR Glasses as Embodied Software**. Ruizhen Gu, Jingqiong Zhang, José Miguel Rojas, Donghwan Shin. *ACM International Conference on AI-Powered Software (AIware 2025)*, 2025. [pdf](https://eprints.whiterose.ac.uk/id/eprint/233948/) [doi](https://ieeexplore.ieee.org/document/11334291)
  Summary: AI-powered XR glasses combine spatial computing with foundation models, creating a new kind of embodied software. This vision paper frames them as a software paradigm and identifies key challenges: security and privacy, validating spatial capabilities, and explainability. Key implication: it sets a research agenda for building reliable, trustworthy XR systems in the foundation-model era.
- **Unseen Data Detection using Routing Entropy in Mixture-of-Experts for Autonomous Vehicles**. Sang In Lee, Donghwan Shin, Jihun Park. *ASE 2025 New Ideas and Emerging Results Track*, 2025. [pdf](https://drive.google.com/file/d/1Zd-y9s3hILR-3F5sqb-bXdgbWqTqC11r/view?usp=sharing) [doi](https://dl.acm.org/doi/10.1109/ASE63991.2025.00332)
  Summary: Detecting inputs unlike the training data is critical for autonomous driving, but output-based detectors suffer from overconfidence. Routing Entropy uses how uncertainly a Mixture-of-Experts model routes an input among its experts as an out-of-distribution signal. Key implication: on an MoE segmentation model for driving perception, it works on its own and significantly improves detection when combined with output-based methods.
- **XRintTest: An Automated Framework for User Interaction Testing in Extended Reality Applications**. Ruizhen Gu, José Miguel Rojas, Donghwan Shin. *ASE 2025 Tool Demonstration Track*, 2025. [doi](https://dl.acm.org/doi/10.1109/ASE63991.2025.00363)
  Summary: XR apps rely on 6-DoF controller interactions that existing tools cannot simulate. XRintTest builds an interaction graph of a Unity XR scene and uses it to explore the scene and trigger interactions automatically. Key implication: it reached 97% coverage of trigger and grab interactions, about 9 times more effective and 5 times more efficient than random exploration.
- **Can Test Generation and Program Repair Inform Automated Assessment of Programming Projects?**. Ruizhen Gu, José Miguel Rojas, Donghwan Shin. *ICST 2025 Education Track*, 2025. [pdf](https://drive.google.com/file/d/1QokEJuQBmod9KLA73PRBFmOxaixRRuG5/view?usp=sharing) [doi](https://ieeexplore.ieee.org/document/10988955)
  Summary: Can automated test generation and program repair help grade complex student projects, not just small exercises? A case study on a real Java project with 296 incorrect submissions found generated tests missed bugs in over half of them, and repair fully fixed only 2.1%. Key implication: current tools are not yet reliable for automated assessment of realistic programming projects.
- **Empirically Evaluating Flaky Tests for Autonomous Driving Systems in Simulated Environments**. Olek Osikowicz, Phil McMinn, Donghwan Shin. *International Flaky Tests Workshop (FTW 2025, co-located with ICSE 2025)*, 2025. [pdf](https://drive.google.com/file/d/18Ucph-Uwkq57my8Qo_t2ulIOYIJ9K9xr/view?usp=sharing) [doi](https://ieeexplore.ieee.org/document/11052713)
  Summary: Simulated ADS tests should, in theory, never be flaky. An empirical study with CARLA and MetaDrive found that 31.3% of CARLA benchmark scenarios were potentially flaky because the simulation was nondeterministic, while MetaDrive showed none. Key implication: simulation-based ADS testing must account for simulator nondeterminism.
- **Software Testing for Extended Reality Applications: A Systematic Mapping Study**. Ruizhen Gu, José Miguel Rojas, Donghwan Shin. *Automated Software Engineering 32 (56), Springer*, 2025. [doi](https://link.springer.com/article/10.1007/s10515-025-00523-7)
  Summary: XR applications are harder to test than 2D GUI apps because of 6-DoF interaction. This first systematic mapping study on XR software testing classifies 34 primary studies by landscape, test facets and evaluation methods, and collects the datasets and tools they use. Key implication: it gives researchers a map of the field, its open challenges and future directions.
- **Taxonomy of Software Log Smells**. Nyyti Saarimäki, Donghwan Shin, Domenico Bianculli. *arXiv preprint*, 2025. [arxiv](https://arxiv.org/abs/2412.09284)
  Summary: Logging is error-prone, yet log-related smells have received little study. From a review of 61 papers, the authors build a taxonomy of nine log smells, validate it against real open-source logging issues, and map smells to existing repair tools. Key implication: both logging code and log files suffer many smells, and several have no tool support yet.

### 2024

- **Impact of Log Parsing on Deep Learning-based Anomaly Detection**. Zanis Ali Khan, Donghwan Shin, Domenico Bianculli, Lionel Briand. *Empirical Software Engineering 29 (139)*, 2024. [doi](https://doi.org/10.1007/s10664-024-10533-w)
  Summary: Deep learning anomaly detectors usually parse logs first, assuming better parsing means better detection. A study of 13 parsers, 7 detectors and 3 datasets finds no strong correlation between parsing accuracy and detection accuracy. Key implication: what matters is whether parsing keeps normal and abnormal logs distinguishable, not parsing accuracy itself.
- **Systematic Evaluation of Deep Learning Models for Log-based Failure Prediction**. Fatemeh Hadadi, Joshua H Dawes, Donghwan Shin, Domenico Bianculli, Lionel Briand. *Empirical Software Engineering 29 (105)*, 2024. [doi](https://link.springer.com/article/10.1007/s10664-024-10501-4)
  Summary: Prior studies compared only a few deep learning models for predicting failures from logs. This work systematically evaluates RNN, CNN and transformer encoders with different log embeddings on 360 synthetic datasets. Key implication: a CNN with Logkey2vec embeddings performs best, and predictions are accurate once datasets exceed about 350 sequences or 7.5% failures.
- **Virtual Environment Model Generation for CPS Goal Verification using Imitation Learning**. Yong-Jun Shin, Donghwan Shin, Doo-Hwan Bae. *ACM Transactions on Embedded Computing Systems 23 (1)*, 2024. [doi](https://dl.acm.org/doi/10.1145/3633804)
  Summary: Verifying a cyber-physical system's goals needs many costly field tests. This work learns a virtual environment model from a small set of field-test logs using imitation learning, so verification can run in simulation. Key implication: in a lane-keeping case study, the generated environment models were accurate enough for goal verification at low cost.
- **Mutation-based Consistency Testing for Evaluating the Code Understanding Capability of LLMs**. Ziyu Li, Donghwan Shin. *International Conference on AI Engineering (CAIN 2024, co-located with ICSE 2024)*, 2024. [doi](https://ieeexplore.ieee.org/document/10556138) [slides](https://drive.google.com/file/d/1H2hXvvH-1a0qUbaEjGuFwtt_tn4pslcy/view?usp=sharing)
  Summary: Code-generation benchmarks do not show whether LLMs really understand code. Mutation-based Consistency Testing makes small semantic changes to code so it no longer matches its description, then asks the LLM to spot the mismatch. Key implication: GPT-3.5 and GPT-4 differ markedly across mutation types and six programming languages, revealing uneven code understanding.
- **Autonomous Driving System Testing: Traffic Density Does Matter**. Guannan Lou, Donghwan Shin, Neil Walkinshaw, Robert M Hierons. *International Conference on Testing Software and Systems (ICTSS 2024)*, 2024. [doi](https://link.springer.com/chapter/10.1007/978-3-031-80889-0_21)
  Summary: Most ADS testing uses scenarios with few other road users. Using two open-source ADSs in a high-fidelity simulator, this study varies traffic density systematically. Key implication: traffic density significantly affects ADS behaviour and exposes different kinds of safety violations, so it should be part of ADS testing.
- **Tuning the feedback controller gains is a simple way to improve autonomous driving performance**. Wenyu Liang, Ross Drummond, Pablo Baldivieso Monasterios, Donghwan Shin. *UKACC International Conference on Control (Control 2024)*, 2024. [doi](https://ieeexplore.ieee.org/document/10531819)
  Summary: Autonomous driving research focuses on neural networks while controller gains are copied between papers. Simply retuning the PID gains of TCP's speed controller raised its CARLA driving score from 73.21 to 77.38, especially in rain and at night. Key implication: neural networks and feedback controllers should be designed together.
- **Towards Simplification of Failure Scenarios for Machine Learning-Enabled Autonomous Systems**. Donghwan Shin, Sanjeetha Pennada. *IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C 2024)*, 2024. [doi](https://ieeexplore.ieee.org/abstract/document/10726945)
  Summary: As testing finds more realistic failures, failure scenarios become too complex to diagnose. This short vision paper proposes search-based and surrogate-assisted optimisation to simplify failure scenarios of ML-enabled autonomous systems. Key implication: simpler failure scenarios make root causes easier to find (the basis of the SimpliFaiS project).
- **Toward Automated Compliance Checking of Fund Activities Using Runtime Verification Techniques**. Marcello Ceci, Nicolas Sannier, Donghwan Shin, Sallam Abualhaija, Domenico Bianculli, Michael Halling. *Workshop on Software Engineering Challenges in Financial Firms (FinanSE 2024)*, 2024. [pdf](https://drive.google.com/file/d/1H-rFoPS6fol4V4Np3VkArtrqBWMuhssN/view?usp=sharing)
  Summary: Fund activities such as subscriptions, redemptions and fee management must comply with complex regulations. This position paper outlines extracting monitoring rules automatically from legislation and fund documents and checking fund activities with runtime verification. Key implication: continuous, automated compliance monitoring for fund managers and regulators.

### 2023

- **Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems**. Fitash Ul Haq, Donghwan Shin, Lionel Briand. *International Conference on Software Engineering (ICSE 2023)*, 2023. [doi](https://doi.org/10.1109/ICSE48619.2023.00155)
  Summary: Online testing of DNN-enabled systems must find scenarios that break several safety requirements, while the environment changes during a run. MORLOT combines reinforcement learning, which changes the environment step by step, with many-objective search that targets requirements not yet violated. Key implication: on Transfuser in CARLA it was significantly more effective and efficient than alternatives, with a large effect size.
- **Identifying the Hazard Boundary of ML-enabled Autonomous Systems Using Cooperative Co-Evolutionary Search**. Sepehr Sharifi, Donghwan Shin, Lionel C Briand, Nathan Aschbacher. *IEEE Transactions on Software Engineering*, 2023. [doi](https://doi.ieeecomputersociety.org/10.1109/TSE.2023.3327575) [arxiv](https://arxiv.org/pdf/2301.13807)
  Summary: Knowing where an ML component's behaviour becomes hazardous is essential for safety, but the search space is huge and simulations are costly and nondeterministic. MLCSHE uses cooperative co-evolutionary search to split the problem and estimate a probabilistic hazard boundary. Key implication: on an autonomous vehicle case study it outperformed standard genetic algorithms and random search.
- **Rigorous Assessment of Model Inference Accuracy using Language Cardinality**. Donato Clun, Donghwan Shin, Antonio Filieri, Domenico Bianculli. *ACM Transactions on Software Engineering and Methodology 33 (4)*, 2023. [doi](https://dl.acm.org/doi/10.1145/3640332) [arxiv](https://arxiv.org/pdf/2211.16587)
  Summary: Inferred behavioural models are usually evaluated with randomly sampled traces, which introduces uncertainty and bias. This work uses analytic combinatorics (language cardinality) to measure model inference accuracy deterministically. Key implication: more rigorous, reproducible comparison of model inference tools on specification-mining benchmarks.
- **Towards Log Slicing**. Joshua Dawes, Donghwan Shin, Domenico Bianculli. *International Conference on Fundamental Approaches to Software Engineering (FASE 2023)*, 2023. [doi](https://doi.org/10.1007/978-3-031-30826-0_14)
  Summary: When something goes wrong, engineers need the log messages related to an error, not the whole log. This paper defines the log slicing problem and proposes an initial approach based on static backward slicing from the logging statement. Key implication: it sets out the problem and the key open issues for future research.
- **An extensible modeling method supporting ontology-based scenario specification and domain-specific extension**. Young-Min Baek, Esther Cho, Donghwan Shin, Doo-Hwan Bae. *International Journal of Software Engineering and Knowledge Engineering*, 2023. [doi](https://doi.org/10.1142/S021819402350047X)
  Summary: Scenario-based methods lack a common reference framework for scenario concepts. This work proposes a Conceptual Scenario Framework and an extensible scenario modelling method and language that supports domain-specific extensions. Key implication: case studies in automated driving show it is expressive and extensible enough to serve as a reference framework.

### 2022

- **Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and Many-Objective Optimization**. Fitash Ul Haq, Donghwan Shin, Lionel Briand. *International Conference on Software Engineering (ICSE 2022)*, 2022. ACM SIGSOFT Distinguished Paper Award. [pdf](https://dl.acm.org/doi/pdf/10.1145/3510003.3510188)
  Summary: Testing a DNN-enabled driving system in a high-fidelity simulator is slow, so few scenarios can be tried. SAMOTA trains cheap surrogate models of the simulator and uses them inside many-objective search to find safety requirement violations. Key implication: on Pylot in CARLA it was significantly more effective and efficient than state-of-the-art search and random testing.
- **Guidelines for Assessing the Accuracy of Log Message Template Identification Techniques**. Zanis Ali Khan, Donghwan Shin, Domenico Bianculli, Lionel Briand. *International Conference on Software Engineering (ICSE 2022)*, 2022. [pdf](https://dl.acm.org/doi/pdf/10.1145/3510003.3510101)
  Summary: Prior evaluations of log template identification used inconsistent metrics and error-prone ground truth. This work gives guidelines (template-level metrics, oracle correction, analysis of incorrect templates) and re-evaluates 14 techniques. Key implication: existing techniques perform much worse than previously reported, and their rankings depend on the metric.
- **PRINS: Scalable Model Inference for Component-Based System Logs**. Donghwan Shin, Domenico Bianculli, Lionel Briand. *Empirical Software Engineering 27 (4)*, 2022. [doi](https://link.springer.com/article/10.1007/s10664-021-10111-4)
  Summary: Model inference from logs does not scale to large component-based systems. PRINS splits logs by component, infers component models in parallel, then stitches and determinises them into a system model. Key implication: it is much faster than the state-of-the-art MINT on large logs without significant loss of accuracy.
- **Literature Review to Collect Conceptual Variables of Scenario Methods for Establishing a Conceptual Scenario Framework**. Young-Min Baek, Esther Cho, Donghwan Shin, Doo-Hwan Bae. *arXiv:2205.08290*, 2022. [arxiv](https://arxiv.org/pdf/2205.08290)
  Summary: Scenarios lack a shared definition and scenario methods lack a conceptual foundation. A semi-systematic literature review collects recurring scenario variables into a four-level meta-model (method, suite, scenario, event). Key implication: applied to five automated-driving scenario methods, it helps explain and compare them.

### 2021

- **Automatic Test Suite Generation for Key-Points Detection DNNs using Many-Objective Search**. Fitash Ul Haq, Donghwan Shin, Lionel C Briand, Thomas Stifter, Jun Wang. *ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2021)*, 2021. [pdf](https://dl.acm.org/doi/pdf/10.1145/3460319.3464802)
  Summary: Key-point detection DNNs, such as facial landmark detectors in cars, predict many points at once and are hard to test. Many-objective search generates test suites that target each key-point. Key implication: generated suites caused severe mispredictions for over 93% of key-points versus 41% for random search, and learned conditions (head pose, skin colour) support risk analysis.
- **Log-based Slicing for System-level Test Cases**. Salma Messaoudi, Donghwan Shin, Annibale Panichella, Domenico Bianculli, Lionel C Briand. *ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2021)*, 2021. [pdf](https://dl.acm.org/doi/pdf/10.1145/3460319.3464824)
  Summary: Complex system test cases often bundle several scenarios and are slow to run. DS3 decomposes them into smaller tests by combining static slicing with execution logs that reveal hidden dependencies on shared resources. Key implication: the resulting tests ran 3.56 times faster on average without significant loss of fault detection.
- **Digital Twins Are Not Monozygotic – Cross-Replicating ADAS Testing in Two Industry-Grade Automotive Simulators**. Markus Borg, Raja Ben Abdessalem, Shiva Nejati, François-Xavier Jegeden, Donghwan Shin. *IEEE Conference on Software Testing, Verification and Validation (ICST 2021)*, 2021. [arxiv](https://arxiv.org/pdf/2012.06822)
  Summary: Can results from one driving simulator be trusted in another? Replicating search-based ADAS testing in PreScan and Pro-SiVIC found critical scenarios in both, but the specific violations and behaviours differed. Key implication: use multiple simulators and simulator-independent test objectives.
- **A Theoretical Framework for Understanding the Relationship Between Log Parsing and Anomaly Detection**. Donghwan Shin, Zanis Ali Khan, Domenico Bianculli, Lionel Briand. *International Conference on Runtime Verification (RV 2021)*, 2021. [pdf](https://orbilu.uni.lu/bitstream/10993/47846/1/main.pdf)
  Summary: Log parsing precedes most anomaly detection, but what makes a parse good for detection was never defined. This paper formalises ideal parsing results through distinguishability and minimality. Key implication: engineers can tell whether poor detection comes from parsing or from detection, and spot log messages irrelevant to detection.
- **Can Offline Testing of Deep Neural Networks Replace Their Online Testing?**. Fitash Ul Haq, Donghwan Shin, Shiva Nejati, Lionel Briand. *Empirical Software Engineering 26 (5)*, 2021. [doi](https://link.springer.com/article/10.1007/s10664-021-09982-4)
  Summary: Can offline testing of DNNs on fixed datasets replace closed-loop online testing? An extended study with steering DNNs in a simulator shows offline and online results often disagree, and offline testing is more optimistic because errors accumulate over time in closed loop. Key implication: online testing is needed to find safety violations of DNN-enabled systems.

### 2020

- **Comparing Offline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study**. Fitash Ul Haq, Donghwan Shin, Shiva Nejati, Lionel C Briand. *IEEE Conference on Software Testing, Verification and Validation (ICST 2020)*, 2020. [arxiv](https://arxiv.org/pdf/1912.00805)
  Summary: Offline testing measures prediction error on datasets, while online testing runs the DNN in closed loop. For steering DNNs, simulator-generated data gave errors similar to real data, but offline testing missed many safety violations that online testing found. Key implication: offline testing is not sufficient for DNN-enabled driving systems.
- **Effective Removal of Operational Log Messages: an Application to Model Inference**. Donghwan Shin, Domenico Bianculli, Lionel Briand. *arXiv:2004.07194*, 2020. [arxiv](https://arxiv.org/pdf/2004.07194)
  Summary: Operational log messages such as heartbeats distort models inferred from logs. LogCleaner removes them using periodicity, dependency scoring and clustering, removing about 98% of operational messages while keeping about 81% of transactional ones. Key implication: faster model inference and more accurate inferred models.

### 2019

- **Empirical Evaluation of Mutation-based Test Case Prioritization Techniques**. Donghwan Shin, Shin Yoo, Mike Papadakis, Doo-Hwan Bae. *Software Testing, Verification and Reliability 29 (1-2)*, 2019. [arxiv](https://arxiv.org/pdf/1709.04631)
  Summary: Mutation-based test prioritisation usually rewards killing mutants. This study compares killing with distinguishing mutants across greedy, hybrid and multi-objective strategies on 352 real faults and 553,477 tests, and introduces the Mutant Distinguishment Graph. Key implication: no single technique wins for all faults, and the graph explains when each criterion falls short.
- **Formal Modeling and Verification of a Federated Byzantine Agreement Algorithm for Blockchain Platforms**. Junghun Yoo, Youlim Jung, Donghwan Shin, Minhyo Bae, Eunkyoung Jee. *IEEE International Workshop on Blockchain Oriented Software Engineering (IWBOSE 2019)*, 2019.
  Summary: Consensus algorithms are the core of blockchain platforms and must be correct. This work formally models a federated Byzantine agreement algorithm and verifies its properties with model checking. Key implication: formal verification can expose design issues in blockchain consensus before deployment.
- **Scalable Inference of System-level Models from Component Logs**. Donghwan Shin, Salma Messaoudi, Domenico Bianculli, Annibale Panichella, Lionel Briand, Raimondas Sasnauskas. *arXiv:1908.02329*, 2019. [arxiv](https://arxiv.org/pdf/1908.02329)
  Summary: Inferring a system-level model from very large component logs is beyond standard tools. SCALER infers a model per component and merges them using architectural dependencies and communication-related log entries. Key implication: on industrial satellite ground-control logs it handled much larger logs than MINT and produced more complete models.

### 2018

- **Are Mutation Scores Correlated with Real Fault Detection? A Large Scale Empirical Study on the Relationship Between Mutants and Real Faults**. Mike Papadakis, Donghwan Shin, Shin Yoo, Doo-Hwan Bae. *International Conference on Software Engineering (ICSE 2018)*, 2018. [pdf](https://dl.acm.org/doi/pdf/10.1145/3180155.3180183)
  Summary: Studies use mutants as stand-ins for real faults because mutation scores seem to correlate with fault detection. This large study on CoREBench and Defects4J shows the correlation becomes weak once test suite size is controlled, yet suites with high mutation scores still detect more real faults than random suites of the same size. Key implication: mutants are useful guidance, but researchers must control for suite size.
- **ABC+: Extended Action-Benefit-Cost Modeling with Knowledge-based Decision-making and Interaction Model for System of Systems Simulation**. Mingyu Jin, Donghwan Shin, Doo-Hwan Bae. *ACM Symposium on Applied Computing (SAC 2018)*, 2018.
  Summary: Simulating systems of systems needs models of autonomous constituent systems that make decisions and interact. ABC+ extends Action-Benefit-Cost modelling with stateful decision-making and message-based interaction. Key implication: it can express and simulate SoS interaction patterns that the original ABC model cannot.

### 2017

- **A Theoretical and Empirical Study of Diversity-aware Mutation Adequacy Criterion**. Donghwan Shin, Shin Yoo, Doo-Hwan Bae. *IEEE Transactions on Software Engineering*, 2017.
  Summary: The classic mutation criterion only asks whether mutants are killed, so one test killing every mutant counts as adequate. The distinguishing criterion requires tests to tell all mutants apart by their kill patterns, and is proven to subsume the classic one. Key implication: on 352 Defects4J faults, adequate suites detected 1.33 times more faults with 1.56 times more tests.
- **Mutation Analysis for System of Systems Policy Testing**. Wonkyung Yun, Donghwan Shin, Doo-Hwan Bae. *Joint Workshop on SE for Systems-of-Systems (JSOS 2017)*, 2017.
  Summary: Testers of system-of-systems policies have no way to judge how good their test cases are. This work applies mutation analysis to SoS policies with new operators and a simulation-based kill checker, evaluated on a traffic-congestion SoS in SUMO. Key implication: mutation scores correlated strongly with real-fault detection, and mutant sampling keeps the cost reasonable.

### 2016

- **A Theoretical Framework for Understanding Mutation-based Testing Methods**. Donghwan Shin, Doo-Hwan Bae. *IEEE International Conference on Software Testing, Verification and Validation (ICST 2016)*, 2016. [arxiv](https://arxiv.org/pdf/1601.06466)
  Summary: Mutation-based testing lacked a theoretical foundation. This framework treats testing as measuring differences between programs, using test differentiators and d-vectors to represent behavioural differences. Key implication: it gives a common basis for analysing techniques such as mutation-based fault localisation and mutant set minimisation.
- **Comprehensive Analysis of FBD Test Coverage Criteria using Mutants**. Donghwan Shin, Eunkyoung Jee, Doo-Hwan Bae. *Software & Systems Modeling*, 2016.
  Summary: How do fault detection, test suite size and coverage level relate for function block diagram (FBD) models used in safety-critical PLC software? Using many test suites and mutants, the study finds detection rises with coverage and beats random testing. Key implication: FBD coverage criteria are effective, though weak at detecting wrong constants.
- **Diversity-Aware Mutation Adequacy Criterion for Improving Fault Detection Capability**. Donghwan Shin, Shin Yoo, Doo-Hwan Bae. *International Workshop on Mutation Analysis (Mutation 2016)*, 2016.
  Summary: Mutation adequacy normally ignores which tests kill which mutants. This criterion requires tests to distinguish all mutants by their kill patterns. Key implication: on real faults it raised the fault detection success rate by up to 76.8 percentage points over traditional mutation-adequate suites.
- **Modeling and Verification for Different Types of System of Systems using PRISM**. Dongwon Seo, Donghwan Shin, Young-Min Baek, Jiyoung Song, Wonkyung Yun, Junho Kim, Eunkyoung Jee, Doo-Hwan Bae. *International Workshop on Software Engineering for Systems-of-Systems*, 2016.
  Summary: Directed, acknowledged and collaborative systems of systems give the SoS different authority over its constituents. Using a mass casualty incident response example, the authors model each type probabilistically in PRISM and apply statistical model checking. Key implication: SoS managers can compare how well each type achieves the SoS goal.
- **A Runtime Verification Framework for Dynamically Adaptive Multi-Agent Systems**. Yoo Jin Lim, Gwangui Hong, Donghwan Shin, Eunkyoung Jee, Doo-Hwan Bae. *International Conference on Big Data and Smart Computing (BigComp 2016)*, 2016.
  Summary: Dynamically adaptive multi-agent systems change behaviour at runtime and collaborate, which static verification does not capture. DAMS-RV verifies each adaptation at runtime through an adaptive feedback loop. Key implication: a case study suggests runtime verification is a feasible way to ensure such adaptations stay correct.

### 2015

- **Efficient Testing of Self-Adaptive Behaviors in Collective Adaptive Systems**. Yoo Jin Lim, Eunkyoung Jee, Donghwan Shin, Doo-Hwan Bae. *IEEE Annual Computer Software and Applications Conference (COMPSAC 2015)*, 2015.
  Summary: Collective adaptive systems change their behaviour in response to their environment, which makes testing expensive. This work proposes a more efficient way to test self-adaptive behaviours in such systems. Key implication: adaptive behaviours can be tested at lower cost.
- **Human Resource Allocation in Software Project with Practical Considerations**. Jihun Park, Dongwon Seo, Gwangui Hong, Donghwan Shin, Jimin Hwa, Doo-Hwan Bae. *International Journal of Software Engineering and Knowledge Engineering*, 2015.
  Summary: Existing staff allocation methods for software projects minimise cost but ignore practical issues. Based on expert input, this genetic algorithm also handles task relevance, multitasking and workload balance. Key implication: practitioners rated these practical considerations as important as cost, and found the approach helpful.

### 2014

- **Practical Human Resource Allocation in Software Projects Using Genetic Algorithm**. Jihun Park, Dongwon Seo, Gwangui Hong, Donghwan Shin, Jimin Hwa, Doo-Hwan Bae. *International Conference on Software Engineering and Knowledge Engineering (SEKE 2014)*, 2014.
  Summary: Allocating developers to tasks is complex on large projects, and cost-only methods ignore practical constraints. This genetic algorithm accounts for issues such as multitasking, task relevance and balanced staffing. Key implication: more realistic project staffing plans.

### 2012

- **Empirical Evaluation on FBD Model-based Test Coverage Criteria using Mutation Analysis**. Donghwan Shin, Eunkyoung Jee, Doo-Hwan Bae. *International Conference on Model Driven Engineering Languages and Systems (MODELS 2012)*, 2012.
  Summary: Function block diagrams are used for safety-critical PLC software such as nuclear reactor protection systems. Using 1,800 test suites and over 600 mutants of industrial FBD models, this study evaluates how well FBD coverage criteria detect faults. Key implication: it identifies the criteria's strengths and weaknesses and suggests improvements.

## Projects

- **MF-Drive: Scalable and Sustainable Simulation-based Testing of Autonomous Vehicles using Multi-Fidelity Optimisation**. PI, NVIDIA Academic Grant Program, Mar 2026 – Feb 2027. A multi-fidelity optimisation framework combining Software-in-the-Loop and Hardware-in-the-Loop validation to find safety violations and hardware-induced failures in autonomous driving systems.
- **Simulation-Based Testing for Mobility Cyber-Physical Systems of Systems**. Co-PI, IITP (South Korea), with KAIST Web Engineering Lab, Apr 2025 – Dec 2026. Testing and debugging approaches for systems of multiple independent autonomous vehicles sharing the same roads.
- **SimpliFaiS: Simplification of Failure Scenarios for Machine Learning-enabled Autonomous Systems**. PI, EPSRC New Investigator Award, Jan 2024 – Jun 2026. Automated methods to identify the minimal set of entities that cause failures in critical driving scenarios.

## PhD students

- Olek Osikowicz (2023–): Efficient simulation-based testing for autonomous vehicles using multi-fidelity fusion
- Nathan Shaw (2024–): Testing and debugging autonomous vehicles in simulated environments
- Guannan Lou (2022–, second supervisor): Simulation-based ADS testing with traffic densities and gradient optimisation
- Zalán Lévai (2023–, second supervisor): Mutation testing for Rust
- Rimsha Chaudhry (2025–, second supervisor): Test flakiness

## Service

- ACM SIGSOFT Distinguished Reviewer Award, ISSTA 2025
- IEEE Distinguished Reviewer Award, ICSE 2024
- Student Volunteer Co-chair, ICST 2026
- Co-organiser, DeepTest 2023 (ICSE 2023)
- Co-organiser, Mutation 2022 and 2023 (ICST)
- Student Volunteer Co-chair, ISSTA 2022
- Social Media Co-chair, ICSE 2020
- PC 2027: ICSE, ICST
- PC 2026: ISSTA, ASE, ITSC, DeepTest, ICST Industry, ICST Tools & Data, APSEC Doctoral Symposium
- PC 2025: ISSTA, ICSE Demo, ASE Tutorials, Mutation, ITEQS, SESoS, ICST Tools & Data
- PC 2024: ICSE, ICST, SCAM, FSE Posters, ICSE Demo, SSBSE HOP, QUATIC, TestADS, Mutation, SESoS
- PC 2023: ASE, ICST, APSEC ERA
- PC 2022: ICST, ISSRE
- PC 2019–2021: ICSE SEIP, Mutation, SEA4DQ, IBF