Senior Lecturer (Associate Professor) in Software Engineering
School of Computer Science, University of Sheffield, UK
I am a Senior Lecturer (Associate Professor in the American system) in the School of Computer Science at the University of Sheffield, a member of the AI x Software Engineering & Testing (ASET) group, and a PI of the Intelligent Testing Lab. I received my PhD, MS and BS in Computer Science from KAIST, advised by Prof. Doo-Hwan Bae, and then spent four years as a research associate and research scientist in Prof. Lionel Briand’s SVV group at the SnT Centre, University of Luxembourg.
d.shin@sheffield.ac.uk · Google Scholar · GitHub · LinkedIn · CV · Book a meeting
I am looking for motivated PhD students interested in testing AI-enabled autonomous systems. Please email me with your CV.
Making AI-enabled autonomous systems, such as self-driving cars, safe and reliable through simulation-based, search-based and multi-fidelity testing, failure simplification, and debugging.
Key papers: ISSTA'26 ASE'26 ASE'26 Demo IV'26 AUSE'26 ASE'25 NIER ICTSS'24 ICSE'23 TSE'23 ICSE'22 ISSTA'21 EMSE'21
Understanding whether artificial faults can stand in for real ones, and making mutation analysis effective and scalable, from theory to efficient tools for Rust and safety-aware mutation for ADS.
Key papers: TOSEM'26 ICST'26 ICST'26 Demo arXiv'26 CAIN'24 STVR'19 ICSE'18 TSE'17 ICST'16
Using software execution logs to solve software engineering problems, including model inference, log parsing, anomaly detection, and failure prediction.
Key papers: arXiv'25 EMSE'24 EMSE'24 TOSEM'23 FASE'23 ICSE'22 EMSE'22 ISSTA'21 RV'21
Realistic driving simulations take minutes to hours each, which limits simulation-based ADS testing. MFBO-Drive treats critical scenario generation as multi-fidelity optimisation, choosing both which scenario to run and at what fidelity. Key implication: across more than 100,000 MetaDrive scenarios it improved cost-effectiveness by 16.8% over state-of-the-art Bayesian optimisation with comparable test quality.
Online testing of DNN-enabled systems must find scenarios that break several safety requirements, while the environment changes during a run. MORLOT combines reinforcement learning, which changes the environment step by step, with many-objective search that targets requirements not yet violated. Key implication: on Transfuser in CARLA it was significantly more effective and efficient than alternatives, with a large effect size.
Testing a DNN-enabled driving system in a high-fidelity simulator is slow, so few scenarios can be tried. SAMOTA trains cheap surrogate models of the simulator and uses them inside many-objective search to find safety requirement violations. Key implication: on Pylot in CARLA it was significantly more effective and efficient than state-of-the-art search and random testing.
Studies use mutants as stand-ins for real faults because mutation scores seem to correlate with fault detection. This large study on CoREBench and Defects4J shows the correlation becomes weak once test suite size is controlled, yet suites with high mutation scores still detect more real faults than random suites of the same size. Key implication: mutants are useful guidance, but researchers must control for suite size.
The classic mutation criterion only asks whether mutants are killed, so one test killing every mutant counts as adequate. The distinguishing criterion requires tests to tell all mutants apart by their kill patterns, and is proven to subsume the classic one. Key implication: on 352 Defects4J faults, adequate suites detected 1.33 times more faults with 1.56 times more tests.
Fault localisation for the planning module of autonomous driving systems is hard because faults show up only over time in a driving scenario. This work localises faults at the level of code branches by analysing coverage over time during failing and passing runs. Key implication: developers can trace an ADS planning failure back to specific branches instead of whole modules.Written from the title only; please check.
@inproceedings{woo2026issta26adsfl,
title = {Branch-Level Fault Localization in ADS Planning via Temporal Coverage Analysis},
author = {Sangmin Woo and Dohyun Kim and Donghwan Shin and Yongdae Kim},
booktitle = {ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2026)},
year = {2026},
url = {https://dl.acm.org/doi/10.1145/3832135}
}ML components change through retraining and fine-tuning, so a single update can hurt some tests and help others. GBV-PD represents each test by a compact projection of its loss gradient and checks how the model's parameter update aligns with it, so tests can be prioritised without running the updated model. Key implication: regression testing of evolving ML systems becomes cheaper, with cached vectors reused across updates.
@inproceedings{cho2026ase26rtp,
title = {Evolution-Aware Regression Test Prioritization of ML-Enabled Systems Using Gradient-Based Behavior Vectors},
author = {Eunho Cho and Donghwan Shin and In-Young Ko},
booktitle = {IEEE/ACM International Conference on Automated Software Engineering (ASE 2026)},
year = {2026},
url = {https://dl.acm.org/doi/10.1145/3832783.3834368}
}Simulators such as CARLA can give different results on each run, which makes ADS test outcomes unreliable. DT-Drive records a scenario, separates the ego vehicle from it, and replays it deterministically, so different driving agents can be injected into the same scenario or conditions such as weather can be changed. Key implication: it gave 100% deterministic results on 128 CARLA benchmark scenarios, enabling fair comparison and counterfactual debugging.
@inproceedings{pennada2026dtdrive,
title = {DT-Drive: A Tool for Deterministic Replay-Based Testing and Debugging of Autonomous Driving Systems},
author = {Sanjeetha Pennada and Matthew Leach and Olek Osikowicz and Donghwan Shin},
booktitle = {ASE 2026 Tools and Demonstrations Track},
year = {2026},
url = {https://dl.acm.org/doi/10.1145/3832783.3834601}
}Mutation analysis is rarely used for Rust because valid mutants are hard to generate and running them is slow. This work evaluates many non-interacting mutations together in the same process (mutation batching), with fast approximation algorithms for the NP-hard batching problem, implemented in mutest-rs. Key implication: across 22 Rust programs, batching cut mutation analysis runtime by up to 52.3%.
@article{lévai2026tosem26batching,
title = {A Comprehensive Empirical and Theoretical Analysis of Batching Algorithms for Efficient, Safe, Parallel Mutation Analysis in Rust},
author = {Zalán Lévai and Donghwan Shin and Phil McMinn},
journal = {ACM Transactions on Software Engineering and Methodology},
year = {2026},
url = {https://dl.acm.org/doi/10.1145/3787851}
}Static mutation batching leaves threads idle while the slowest mutation in a batch finishes. Dynamic Mutation Scheduling instead assigns compatible mutations, those touching distinct parts of the program, to workers as soon as they become free. Key implication: on 25 Rust programs it reduced wasted time and evaluation runtime by up to 75.6% compared with static batching.
@inproceedings{lévai2026icst26dms,
title = {Dynamic Mutation Scheduling: Highly Parallel, Efficient Evaluation of Mutations for Rust Programs through Program Splitting},
author = {Zalán Lévai and Donghwan Shin and Phil McMinn},
booktitle = {IEEE International Conference on Software Testing, Verification and Validation (ICST 2026)},
year = {2026}
}No mature mutation analysis tool existed for Rust. mutest-rs integrates with the rustc compiler and uses extensive static analysis to generate valid mutants, plus Rust-specific meta-mutants for efficiency. Key implication: Rust developers get a practical, open-source way to measure how thorough their tests are (mutest.rs).
@inproceedings{lévai2026mutestrs,
title = {mutest-rs: Flexible, Efficient Mutation Analysis Tool for Rust Programs, using Extensive Static Analysis},
author = {Zalán Lévai and Donghwan Shin and Phil McMinn},
booktitle = {ICST 2026 Tools and Demo Track},
year = {2026}
}Realistic driving simulations take minutes to hours each, which limits simulation-based ADS testing. MFBO-Drive treats critical scenario generation as multi-fidelity optimisation, choosing both which scenario to run and at what fidelity. Key implication: across more than 100,000 MetaDrive scenarios it improved cost-effectiveness by 16.8% over state-of-the-art Bayesian optimisation with comparable test quality.
@inproceedings{osikowicz2026iv26mfbo,
title = {Multi-Fidelity Bayesian Optimization for Simulation Based Autonomous Driving Systems Testing},
author = {Olek Osikowicz and Phil McMinn and Wei Xing and Donghwan Shin},
booktitle = {IEEE Intelligent Vehicles Symposium (IV 2026)},
year = {2026}
}Simulation-based ADS testing assumes the simulator is deterministic. This study reruns benchmark scenarios in CARLA and MetaDrive and traces where executions diverge. Key implication: 43.6% of CARLA scenarios were flaky (versus 3 of 1,000 in MetaDrive), mostly due to scenario initialisation, so ADS test results must be checked for nondeterminism before they are trusted.
@article{osikowicz2026ause26flaky,
title = {Simulation Nondeterminism and Its Impact on Flaky Tests for Autonomous Driving Systems},
author = {Olek Osikowicz and Phil McMinn and Donghwan Shin},
journal = {Automated Software Engineering (Springer), accepted},
year = {2026}
}Existing VR testing tools handle simple point-and-click events, not realistic 3D interactions such as grabbing or socketing objects. The authors identify four common interaction types, model them as an Interaction Flow Graph, and build XRintTest to exercise them automatically, evaluated on a new 456-interaction benchmark. Key implication: XRintTest covered 93% of interactions versus 8% for random input and found real interaction bugs.
@article{gu2026ause26vr,
title = {Automated Testing of Prevalent 3D User Interactions in Virtual Reality Applications},
author = {Ruizhen Gu and José Miguel Rojas and Donghwan Shin},
journal = {Automated Software Engineering (Springer)},
year = {2026},
url = {https://link.springer.com/article/10.1007/s10515-026-00620-1}
}ADS failures often emerge from interactions between perception, planning and control rather than from one module. CREAD coordinates a perception fuzzer, a metamorphic validator and an orchestrator agent through a shared blackboard. Key implication: in HighwayEnv it found about 2.1 times more failures per 100 scenarios than a single-agent baseline (work in progress).
@inproceedings{gu2026icra26ws,
title = {Collaborative Multi-Agent Testing for Emergent Failure Discovery in Autonomous Driving Systems},
author = {Ruizhen Gu and Konstantinos Koufos and Donghwan Shin and Vahid Garousi and Mehrdad Dianati},
booktitle = {1st Workshop on Generalization in Autonomous Driving (co-located with ICRA 2026)},
year = {2026}
}ADS testing lacks a principled answer to when testing is enough, and coverage metrics miss the component interactions behind many crashes. This vision paper proposes injecting time-limited faults into the messages between driving modules, with mutation rules derived from STPA safety analysis. Key implication: mutation testing could give ADS a safety-grounded adequacy criterion and guide scenario generation and repair.
@misc{shin2026arxiv26safetymutation,
title = {Towards Safety-Aware Mutation Testing for Autonomous Driving Systems},
author = {Donghwan Shin},
note = {arXiv preprint},
year = {2026},
url = {https://arxiv.org/abs/2606.26456}
}Testing of autonomous driving systems is well studied, but debugging after a test fails is not. This systematic mapping study surveys fault localisation and repair approaches for ADS, identifies gaps, and proposes common terminology. Key implication: the field is fragmented but promising, and the study gives a map and a research agenda for ADS debugging.
@misc{shaw2026arxiv26adsdebug,
title = {A Systematic Mapping Study on the Debugging of Autonomous Driving Systems},
author = {Nathan Shaw and Sanjeetha Pennada and Robert M Hierons and Donghwan Shin},
note = {arXiv preprint},
year = {2026},
url = {https://arxiv.org/abs/2601.04293}
}Developers often bundle several unrelated changes into one commit, which makes history hard to understand. This work treats detecting the concerns in a tangled commit as multi-label classification and evaluates small language models that can run locally. Key implication: a fine-tuned small model beat a state-of-the-art LLM, and adding commit messages improved accuracy by up to 31% at negligible cost.
@misc{koh2026arxiv26tangled,
title = {Detecting Multiple Semantic Concerns in Tangled Code Commits},
author = {Beomsu Koh and Neil Walkinshaw and Donghwan Shin},
note = {arXiv preprint},
year = {2026},
url = {https://arxiv.org/abs/2601.21298}
}Causal testing works well for complex, nondeterministic systems, but only when all relevant variables are observed and do not interact. This work adds effect modification and instrumental-variable methods to a causal testing tool and evaluates it on three requirements of the CARLA simulator. Key implication: reliable test outcomes are possible even with hidden and interacting variables, without heavy instrumentation or tightly controlled data.
@inproceedings{foster2025ease25causal,
title = {Using Causal Inference to Test Systems with Hidden and Interacting Variables: An Evaluative Case Study},
author = {Michael Foster and Robert Hierons and Donghwan Shin and Neil Walkinshaw and Chris Wild},
booktitle = {International Conference on Evaluation and Assessment in Software Engineering (EASE 2025)},
year = {2025},
url = {https://dl.acm.org/doi/10.1145/3756681.3756967}
}AI-powered XR glasses combine spatial computing with foundation models, creating a new kind of embodied software. This vision paper frames them as a software paradigm and identifies key challenges: security and privacy, validating spatial capabilities, and explainability. Key implication: it sets a research agenda for building reliable, trustworthy XR systems in the foundation-model era.
@inproceedings{gu2025aiware25xr,
title = {On the Promises and Challenges of AI-Powered XR Glasses as Embodied Software},
author = {Ruizhen Gu and Jingqiong Zhang and José Miguel Rojas and Donghwan Shin},
booktitle = {ACM International Conference on AI-Powered Software (AIware 2025)},
year = {2025},
url = {https://ieeexplore.ieee.org/document/11334291}
}Detecting inputs unlike the training data is critical for autonomous driving, but output-based detectors suffer from overconfidence. Routing Entropy uses how uncertainly a Mixture-of-Experts model routes an input among its experts as an out-of-distribution signal. Key implication: on an MoE segmentation model for driving perception, it works on its own and significantly improves detection when combined with output-based methods.
@inproceedings{lee2025ase25moe,
title = {Unseen Data Detection using Routing Entropy in Mixture-of-Experts for Autonomous Vehicles},
author = {Sang In Lee and Donghwan Shin and Jihun Park},
booktitle = {ASE 2025 New Ideas and Emerging Results Track},
year = {2025},
url = {https://dl.acm.org/doi/10.1109/ASE63991.2025.00332}
}XR apps rely on 6-DoF controller interactions that existing tools cannot simulate. XRintTest builds an interaction graph of a Unity XR scene and uses it to explore the scene and trigger interactions automatically. Key implication: it reached 97% coverage of trigger and grab interactions, about 9 times more effective and 5 times more efficient than random exploration.
@inproceedings{gu2025xrinttest,
title = {XRintTest: An Automated Framework for User Interaction Testing in Extended Reality Applications},
author = {Ruizhen Gu and José Miguel Rojas and Donghwan Shin},
booktitle = {ASE 2025 Tool Demonstration Track},
year = {2025},
url = {https://dl.acm.org/doi/10.1109/ASE63991.2025.00363}
}Can automated test generation and program repair help grade complex student projects, not just small exercises? A case study on a real Java project with 296 incorrect submissions found generated tests missed bugs in over half of them, and repair fully fixed only 2.1%. Key implication: current tools are not yet reliable for automated assessment of realistic programming projects.
@inproceedings{gu2025icst25edu,
title = {Can Test Generation and Program Repair Inform Automated Assessment of Programming Projects?},
author = {Ruizhen Gu and José Miguel Rojas and Donghwan Shin},
booktitle = {ICST 2025 Education Track},
year = {2025},
url = {https://ieeexplore.ieee.org/document/10988955}
}Simulated ADS tests should, in theory, never be flaky. An empirical study with CARLA and MetaDrive found that 31.3% of CARLA benchmark scenarios were potentially flaky because the simulation was nondeterministic, while MetaDrive showed none. Key implication: simulation-based ADS testing must account for simulator nondeterminism.
@inproceedings{osikowicz2025ftw25,
title = {Empirically Evaluating Flaky Tests for Autonomous Driving Systems in Simulated Environments},
author = {Olek Osikowicz and Phil McMinn and Donghwan Shin},
booktitle = {International Flaky Tests Workshop (FTW 2025, co-located with ICSE 2025)},
year = {2025},
url = {https://ieeexplore.ieee.org/document/11052713}
}XR applications are harder to test than 2D GUI apps because of 6-DoF interaction. This first systematic mapping study on XR software testing classifies 34 primary studies by landscape, test facets and evaluation methods, and collects the datasets and tools they use. Key implication: it gives researchers a map of the field, its open challenges and future directions.
@article{gu2025ause25xrsms,
title = {Software Testing for Extended Reality Applications: A Systematic Mapping Study},
author = {Ruizhen Gu and José Miguel Rojas and Donghwan Shin},
journal = {Automated Software Engineering 32 (56), Springer},
year = {2025},
url = {https://link.springer.com/article/10.1007/s10515-025-00523-7}
}Logging is error-prone, yet log-related smells have received little study. From a review of 61 papers, the authors build a taxonomy of nine log smells, validate it against real open-source logging issues, and map smells to existing repair tools. Key implication: both logging code and log files suffer many smells, and several have no tool support yet.
@misc{saarimäki2025arxiv25logsmells,
title = {Taxonomy of Software Log Smells},
author = {Nyyti Saarimäki and Donghwan Shin and Domenico Bianculli},
note = {arXiv preprint},
year = {2025},
url = {https://arxiv.org/abs/2412.09284}
}Deep learning anomaly detectors usually parse logs first, assuming better parsing means better detection. A study of 13 parsers, 7 detectors and 3 datasets finds no strong correlation between parsing accuracy and detection accuracy. Key implication: what matters is whether parsing keeps normal and abnormal logs distinguishable, not parsing accuracy itself.
@article{khan2024emse24logparsing,
title = {Impact of Log Parsing on Deep Learning-based Anomaly Detection},
author = {Zanis Ali Khan and Donghwan Shin and Domenico Bianculli and Lionel Briand},
journal = {Empirical Software Engineering 29 (139)},
year = {2024},
url = {https://doi.org/10.1007/s10664-024-10533-w}
}Prior studies compared only a few deep learning models for predicting failures from logs. This work systematically evaluates RNN, CNN and transformer encoders with different log embeddings on 360 synthetic datasets. Key implication: a CNN with Logkey2vec embeddings performs best, and predictions are accurate once datasets exceed about 350 sequences or 7.5% failures.
@article{hadadi2024emse24failpred,
title = {Systematic Evaluation of Deep Learning Models for Log-based Failure Prediction},
author = {Fatemeh Hadadi and Joshua H Dawes and Donghwan Shin and Domenico Bianculli and Lionel Briand},
journal = {Empirical Software Engineering 29 (105)},
year = {2024},
url = {https://link.springer.com/article/10.1007/s10664-024-10501-4}
}Verifying a cyber-physical system's goals needs many costly field tests. This work learns a virtual environment model from a small set of field-test logs using imitation learning, so verification can run in simulation. Key implication: in a lane-keeping case study, the generated environment models were accurate enough for goal verification at low cost.
@article{shin2024tecs24,
title = {Virtual Environment Model Generation for CPS Goal Verification using Imitation Learning},
author = {Yong-Jun Shin and Donghwan Shin and Doo-Hwan Bae},
journal = {ACM Transactions on Embedded Computing Systems 23 (1)},
year = {2024},
url = {https://dl.acm.org/doi/10.1145/3633804}
}Code-generation benchmarks do not show whether LLMs really understand code. Mutation-based Consistency Testing makes small semantic changes to code so it no longer matches its description, then asks the LLM to spot the mismatch. Key implication: GPT-3.5 and GPT-4 differ markedly across mutation types and six programming languages, revealing uneven code understanding.
@inproceedings{li2024cain24llmmutation,
title = {Mutation-based Consistency Testing for Evaluating the Code Understanding Capability of LLMs},
author = {Ziyu Li and Donghwan Shin},
booktitle = {International Conference on AI Engineering (CAIN 2024, co-located with ICSE 2024)},
year = {2024},
url = {https://ieeexplore.ieee.org/document/10556138}
}Most ADS testing uses scenarios with few other road users. Using two open-source ADSs in a high-fidelity simulator, this study varies traffic density systematically. Key implication: traffic density significantly affects ADS behaviour and exposes different kinds of safety violations, so it should be part of ADS testing.
@inproceedings{lou2024ictss24,
title = {Autonomous Driving System Testing: Traffic Density Does Matter},
author = {Guannan Lou and Donghwan Shin and Neil Walkinshaw and Robert M Hierons},
booktitle = {International Conference on Testing Software and Systems (ICTSS 2024)},
year = {2024},
url = {https://link.springer.com/chapter/10.1007/978-3-031-80889-0_21}
}Autonomous driving research focuses on neural networks while controller gains are copied between papers. Simply retuning the PID gains of TCP's speed controller raised its CARLA driving score from 73.21 to 77.38, especially in rain and at night. Key implication: neural networks and feedback controllers should be designed together.
@inproceedings{liang2024control24,
title = {Tuning the feedback controller gains is a simple way to improve autonomous driving performance},
author = {Wenyu Liang and Ross Drummond and Pablo Baldivieso Monasterios and Donghwan Shin},
booktitle = {UKACC International Conference on Control (Control 2024)},
year = {2024},
url = {https://ieeexplore.ieee.org/document/10531819}
}As testing finds more realistic failures, failure scenarios become too complex to diagnose. This short vision paper proposes search-based and surrogate-assisted optimisation to simplify failure scenarios of ML-enabled autonomous systems. Key implication: simpler failure scenarios make root causes easier to find (the basis of the SimpliFaiS project).
@inproceedings{shin2024qrsc24,
title = {Towards Simplification of Failure Scenarios for Machine Learning-Enabled Autonomous Systems},
author = {Donghwan Shin and Sanjeetha Pennada},
booktitle = {IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C 2024)},
year = {2024},
url = {https://ieeexplore.ieee.org/abstract/document/10726945}
}Fund activities such as subscriptions, redemptions and fee management must comply with complex regulations. This position paper outlines extracting monitoring rules automatically from legislation and fund documents and checking fund activities with runtime verification. Key implication: continuous, automated compliance monitoring for fund managers and regulators.
@inproceedings{ceci2024finanse24,
title = {Toward Automated Compliance Checking of Fund Activities Using Runtime Verification Techniques},
author = {Marcello Ceci and Nicolas Sannier and Donghwan Shin and Sallam Abualhaija and Domenico Bianculli and Michael Halling},
booktitle = {Workshop on Software Engineering Challenges in Financial Firms (FinanSE 2024)},
year = {2024}
}Online testing of DNN-enabled systems must find scenarios that break several safety requirements, while the environment changes during a run. MORLOT combines reinforcement learning, which changes the environment step by step, with many-objective search that targets requirements not yet violated. Key implication: on Transfuser in CARLA it was significantly more effective and efficient than alternatives, with a large effect size.
@inproceedings{haq2023morlot,
title = {Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems},
author = {Fitash Ul Haq and Donghwan Shin and Lionel Briand},
booktitle = {International Conference on Software Engineering (ICSE 2023)},
year = {2023},
url = {https://doi.org/10.1109/ICSE48619.2023.00155}
}Knowing where an ML component's behaviour becomes hazardous is essential for safety, but the search space is huge and simulations are costly and nondeterministic. MLCSHE uses cooperative co-evolutionary search to split the problem and estimate a probabilistic hazard boundary. Key implication: on an autonomous vehicle case study it outperformed standard genetic algorithms and random search.
@article{sharifi2023tse23hazard,
title = {Identifying the Hazard Boundary of ML-enabled Autonomous Systems Using Cooperative Co-Evolutionary Search},
author = {Sepehr Sharifi and Donghwan Shin and Lionel C Briand and Nathan Aschbacher},
journal = {IEEE Transactions on Software Engineering},
year = {2023},
url = {https://doi.ieeecomputersociety.org/10.1109/TSE.2023.3327575}
}Inferred behavioural models are usually evaluated with randomly sampled traces, which introduces uncertainty and bias. This work uses analytic combinatorics (language cardinality) to measure model inference accuracy deterministically. Key implication: more rigorous, reproducible comparison of model inference tools on specification-mining benchmarks.
@article{clun2023tosem23,
title = {Rigorous Assessment of Model Inference Accuracy using Language Cardinality},
author = {Donato Clun and Donghwan Shin and Antonio Filieri and Domenico Bianculli},
journal = {ACM Transactions on Software Engineering and Methodology 33 (4)},
year = {2023},
url = {https://dl.acm.org/doi/10.1145/3640332}
}When something goes wrong, engineers need the log messages related to an error, not the whole log. This paper defines the log slicing problem and proposes an initial approach based on static backward slicing from the logging statement. Key implication: it sets out the problem and the key open issues for future research.
@inproceedings{dawes2023fase23,
title = {Towards Log Slicing},
author = {Joshua Dawes and Donghwan Shin and Domenico Bianculli},
booktitle = {International Conference on Fundamental Approaches to Software Engineering (FASE 2023)},
year = {2023},
url = {https://doi.org/10.1007/978-3-031-30826-0_14}
}Scenario-based methods lack a common reference framework for scenario concepts. This work proposes a Conceptual Scenario Framework and an extensible scenario modelling method and language that supports domain-specific extensions. Key implication: case studies in automated driving show it is expressive and extensible enough to serve as a reference framework.
@article{baek2023ijseke23,
title = {An extensible modeling method supporting ontology-based scenario specification and domain-specific extension},
author = {Young-Min Baek and Esther Cho and Donghwan Shin and Doo-Hwan Bae},
journal = {International Journal of Software Engineering and Knowledge Engineering},
year = {2023},
url = {https://doi.org/10.1142/S021819402350047X}
}Testing a DNN-enabled driving system in a high-fidelity simulator is slow, so few scenarios can be tried. SAMOTA trains cheap surrogate models of the simulator and uses them inside many-objective search to find safety requirement violations. Key implication: on Pylot in CARLA it was significantly more effective and efficient than state-of-the-art search and random testing.
@inproceedings{haq2022samota,
title = {Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and Many-Objective Optimization},
author = {Fitash Ul Haq and Donghwan Shin and Lionel Briand},
booktitle = {International Conference on Software Engineering (ICSE 2022)},
year = {2022}
}Prior evaluations of log template identification used inconsistent metrics and error-prone ground truth. This work gives guidelines (template-level metrics, oracle correction, analysis of incorrect templates) and re-evaluates 14 techniques. Key implication: existing techniques perform much worse than previously reported, and their rankings depend on the metric.
@inproceedings{khan2022icse22logtemplate,
title = {Guidelines for Assessing the Accuracy of Log Message Template Identification Techniques},
author = {Zanis Ali Khan and Donghwan Shin and Domenico Bianculli and Lionel Briand},
booktitle = {International Conference on Software Engineering (ICSE 2022)},
year = {2022}
}Model inference from logs does not scale to large component-based systems. PRINS splits logs by component, infers component models in parallel, then stitches and determinises them into a system model. Key implication: it is much faster than the state-of-the-art MINT on large logs without significant loss of accuracy.
@article{shin2022prins,
title = {PRINS: Scalable Model Inference for Component-Based System Logs},
author = {Donghwan Shin and Domenico Bianculli and Lionel Briand},
journal = {Empirical Software Engineering 27 (4)},
year = {2022},
url = {https://link.springer.com/article/10.1007/s10664-021-10111-4}
}Scenarios lack a shared definition and scenario methods lack a conceptual foundation. A semi-systematic literature review collects recurring scenario variables into a four-level meta-model (method, suite, scenario, event). Key implication: applied to five automated-driving scenario methods, it helps explain and compare them.
@misc{baek2022arxiv22scenario,
title = {Literature Review to Collect Conceptual Variables of Scenario Methods for Establishing a Conceptual Scenario Framework},
author = {Young-Min Baek and Esther Cho and Donghwan Shin and Doo-Hwan Bae},
note = {arXiv:2205.08290},
year = {2022},
url = {https://arxiv.org/pdf/2205.08290}
}Key-point detection DNNs, such as facial landmark detectors in cars, predict many points at once and are hard to test. Many-objective search generates test suites that target each key-point. Key implication: generated suites caused severe mispredictions for over 93% of key-points versus 41% for random search, and learned conditions (head pose, skin colour) support risk analysis.
@inproceedings{haq2021issta21kp,
title = {Automatic Test Suite Generation for Key-Points Detection DNNs using Many-Objective Search},
author = {Fitash Ul Haq and Donghwan Shin and Lionel C Briand and Thomas Stifter and Jun Wang},
booktitle = {ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2021)},
year = {2021}
}Complex system test cases often bundle several scenarios and are slow to run. DS3 decomposes them into smaller tests by combining static slicing with execution logs that reveal hidden dependencies on shared resources. Key implication: the resulting tests ran 3.56 times faster on average without significant loss of fault detection.
@inproceedings{messaoudi2021issta21logslice,
title = {Log-based Slicing for System-level Test Cases},
author = {Salma Messaoudi and Donghwan Shin and Annibale Panichella and Domenico Bianculli and Lionel C Briand},
booktitle = {ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2021)},
year = {2021}
}Can results from one driving simulator be trusted in another? Replicating search-based ADAS testing in PreScan and Pro-SiVIC found critical scenarios in both, but the specific violations and behaviours differed. Key implication: use multiple simulators and simulator-independent test objectives.
@inproceedings{borg2021icst21dt,
title = {Digital Twins Are Not Monozygotic – Cross-Replicating ADAS Testing in Two Industry-Grade Automotive Simulators},
author = {Markus Borg and Raja Ben Abdessalem and Shiva Nejati and François-Xavier Jegeden and Donghwan Shin},
booktitle = {IEEE Conference on Software Testing, Verification and Validation (ICST 2021)},
year = {2021}
}Log parsing precedes most anomaly detection, but what makes a parse good for detection was never defined. This paper formalises ideal parsing results through distinguishability and minimality. Key implication: engineers can tell whether poor detection comes from parsing or from detection, and spot log messages irrelevant to detection.
@inproceedings{shin2021rv21,
title = {A Theoretical Framework for Understanding the Relationship Between Log Parsing and Anomaly Detection},
author = {Donghwan Shin and Zanis Ali Khan and Domenico Bianculli and Lionel Briand},
booktitle = {International Conference on Runtime Verification (RV 2021)},
year = {2021}
}Can offline testing of DNNs on fixed datasets replace closed-loop online testing? An extended study with steering DNNs in a simulator shows offline and online results often disagree, and offline testing is more optimistic because errors accumulate over time in closed loop. Key implication: online testing is needed to find safety violations of DNN-enabled systems.
@article{haq2021emse21offline,
title = {Can Offline Testing of Deep Neural Networks Replace Their Online Testing?},
author = {Fitash Ul Haq and Donghwan Shin and Shiva Nejati and Lionel Briand},
journal = {Empirical Software Engineering 26 (5)},
year = {2021},
url = {https://link.springer.com/article/10.1007/s10664-021-09982-4}
}Offline testing measures prediction error on datasets, while online testing runs the DNN in closed loop. For steering DNNs, simulator-generated data gave errors similar to real data, but offline testing missed many safety violations that online testing found. Key implication: offline testing is not sufficient for DNN-enabled driving systems.
@inproceedings{haq2020icst20,
title = {Comparing Offline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study},
author = {Fitash Ul Haq and Donghwan Shin and Shiva Nejati and Lionel C Briand},
booktitle = {IEEE Conference on Software Testing, Verification and Validation (ICST 2020)},
year = {2020}
}Operational log messages such as heartbeats distort models inferred from logs. LogCleaner removes them using periodicity, dependency scoring and clustering, removing about 98% of operational messages while keeping about 81% of transactional ones. Key implication: faster model inference and more accurate inferred models.
@misc{shin2020arxiv20,
title = {Effective Removal of Operational Log Messages: an Application to Model Inference},
author = {Donghwan Shin and Domenico Bianculli and Lionel Briand},
note = {arXiv:2004.07194},
year = {2020},
url = {https://arxiv.org/pdf/2004.07194}
}Mutation-based test prioritisation usually rewards killing mutants. This study compares killing with distinguishing mutants across greedy, hybrid and multi-objective strategies on 352 real faults and 553,477 tests, and introduces the Mutant Distinguishment Graph. Key implication: no single technique wins for all faults, and the graph explains when each criterion falls short.
@article{shin2019stvr19,
title = {Empirical Evaluation of Mutation-based Test Case Prioritization Techniques},
author = {Donghwan Shin and Shin Yoo and Mike Papadakis and Doo-Hwan Bae},
journal = {Software Testing, Verification and Reliability 29 (1-2)},
year = {2019}
}Consensus algorithms are the core of blockchain platforms and must be correct. This work formally models a federated Byzantine agreement algorithm and verifies its properties with model checking. Key implication: formal verification can expose design issues in blockchain consensus before deployment.Written from the title only; please check.
@inproceedings{yoo2019iwbose19,
title = {Formal Modeling and Verification of a Federated Byzantine Agreement Algorithm for Blockchain Platforms},
author = {Junghun Yoo and Youlim Jung and Donghwan Shin and Minhyo Bae and Eunkyoung Jee},
booktitle = {IEEE International Workshop on Blockchain Oriented Software Engineering (IWBOSE 2019)},
year = {2019}
}Inferring a system-level model from very large component logs is beyond standard tools. SCALER infers a model per component and merges them using architectural dependencies and communication-related log entries. Key implication: on industrial satellite ground-control logs it handled much larger logs than MINT and produced more complete models.
@misc{shin2019arxiv19,
title = {Scalable Inference of System-level Models from Component Logs},
author = {Donghwan Shin and Salma Messaoudi and Domenico Bianculli and Annibale Panichella and Lionel Briand and Raimondas Sasnauskas},
note = {arXiv:1908.02329},
year = {2019},
url = {https://arxiv.org/pdf/1908.02329}
}Studies use mutants as stand-ins for real faults because mutation scores seem to correlate with fault detection. This large study on CoREBench and Defects4J shows the correlation becomes weak once test suite size is controlled, yet suites with high mutation scores still detect more real faults than random suites of the same size. Key implication: mutants are useful guidance, but researchers must control for suite size.
@inproceedings{papadakis2018icse18mutants,
title = {Are Mutation Scores Correlated with Real Fault Detection? A Large Scale Empirical Study on the Relationship Between Mutants and Real Faults},
author = {Mike Papadakis and Donghwan Shin and Shin Yoo and Doo-Hwan Bae},
booktitle = {International Conference on Software Engineering (ICSE 2018)},
year = {2018}
}Simulating systems of systems needs models of autonomous constituent systems that make decisions and interact. ABC+ extends Action-Benefit-Cost modelling with stateful decision-making and message-based interaction. Key implication: it can express and simulate SoS interaction patterns that the original ABC model cannot.
@inproceedings{jin2018sac18,
title = {ABC+: Extended Action-Benefit-Cost Modeling with Knowledge-based Decision-making and Interaction Model for System of Systems Simulation},
author = {Mingyu Jin and Donghwan Shin and Doo-Hwan Bae},
booktitle = {ACM Symposium on Applied Computing (SAC 2018)},
year = {2018}
}The classic mutation criterion only asks whether mutants are killed, so one test killing every mutant counts as adequate. The distinguishing criterion requires tests to tell all mutants apart by their kill patterns, and is proven to subsume the classic one. Key implication: on 352 Defects4J faults, adequate suites detected 1.33 times more faults with 1.56 times more tests.
@article{shin2017dmutation,
title = {A Theoretical and Empirical Study of Diversity-aware Mutation Adequacy Criterion},
author = {Donghwan Shin and Shin Yoo and Doo-Hwan Bae},
journal = {IEEE Transactions on Software Engineering},
year = {2017}
}Testers of system-of-systems policies have no way to judge how good their test cases are. This work applies mutation analysis to SoS policies with new operators and a simulation-based kill checker, evaluated on a traffic-congestion SoS in SUMO. Key implication: mutation scores correlated strongly with real-fault detection, and mutant sampling keeps the cost reasonable.
@inproceedings{yun2017jsos17,
title = {Mutation Analysis for System of Systems Policy Testing},
author = {Wonkyung Yun and Donghwan Shin and Doo-Hwan Bae},
booktitle = {Joint Workshop on SE for Systems-of-Systems (JSOS 2017)},
year = {2017}
}Mutation-based testing lacked a theoretical foundation. This framework treats testing as measuring differences between programs, using test differentiators and d-vectors to represent behavioural differences. Key implication: it gives a common basis for analysing techniques such as mutation-based fault localisation and mutant set minimisation.
@inproceedings{shin2016icst16,
title = {A Theoretical Framework for Understanding Mutation-based Testing Methods},
author = {Donghwan Shin and Doo-Hwan Bae},
booktitle = {IEEE International Conference on Software Testing, Verification and Validation (ICST 2016)},
year = {2016}
}How do fault detection, test suite size and coverage level relate for function block diagram (FBD) models used in safety-critical PLC software? Using many test suites and mutants, the study finds detection rises with coverage and beats random testing. Key implication: FBD coverage criteria are effective, though weak at detecting wrong constants.
@article{shin2016sosym16,
title = {Comprehensive Analysis of FBD Test Coverage Criteria using Mutants},
author = {Donghwan Shin and Eunkyoung Jee and Doo-Hwan Bae},
journal = {Software & Systems Modeling},
year = {2016}
}Mutation adequacy normally ignores which tests kill which mutants. This criterion requires tests to distinguish all mutants by their kill patterns. Key implication: on real faults it raised the fault detection success rate by up to 76.8 percentage points over traditional mutation-adequate suites.
@inproceedings{shin2016mutation16,
title = {Diversity-Aware Mutation Adequacy Criterion for Improving Fault Detection Capability},
author = {Donghwan Shin and Shin Yoo and Doo-Hwan Bae},
booktitle = {International Workshop on Mutation Analysis (Mutation 2016)},
year = {2016}
}Directed, acknowledged and collaborative systems of systems give the SoS different authority over its constituents. Using a mass casualty incident response example, the authors model each type probabilistically in PRISM and apply statistical model checking. Key implication: SoS managers can compare how well each type achieves the SoS goal.
@inproceedings{seo2016sesos16,
title = {Modeling and Verification for Different Types of System of Systems using PRISM},
author = {Dongwon Seo and Donghwan Shin and Young-Min Baek and Jiyoung Song and Wonkyung Yun and Junho Kim and Eunkyoung Jee and Doo-Hwan Bae},
booktitle = {International Workshop on Software Engineering for Systems-of-Systems},
year = {2016}
}Dynamically adaptive multi-agent systems change behaviour at runtime and collaborate, which static verification does not capture. DAMS-RV verifies each adaptation at runtime through an adaptive feedback loop. Key implication: a case study suggests runtime verification is a feasible way to ensure such adaptations stay correct.
@inproceedings{lim2016bigcomp16,
title = {A Runtime Verification Framework for Dynamically Adaptive Multi-Agent Systems},
author = {Yoo Jin Lim and Gwangui Hong and Donghwan Shin and Eunkyoung Jee and Doo-Hwan Bae},
booktitle = {International Conference on Big Data and Smart Computing (BigComp 2016)},
year = {2016}
}Collective adaptive systems change their behaviour in response to their environment, which makes testing expensive. This work proposes a more efficient way to test self-adaptive behaviours in such systems. Key implication: adaptive behaviours can be tested at lower cost.Written from the title only; please check.
@inproceedings{lim2015compsac15,
title = {Efficient Testing of Self-Adaptive Behaviors in Collective Adaptive Systems},
author = {Yoo Jin Lim and Eunkyoung Jee and Donghwan Shin and Doo-Hwan Bae},
booktitle = {IEEE Annual Computer Software and Applications Conference (COMPSAC 2015)},
year = {2015}
}Existing staff allocation methods for software projects minimise cost but ignore practical issues. Based on expert input, this genetic algorithm also handles task relevance, multitasking and workload balance. Key implication: practitioners rated these practical considerations as important as cost, and found the approach helpful.
@article{park2015ijseke15,
title = {Human Resource Allocation in Software Project with Practical Considerations},
author = {Jihun Park and Dongwon Seo and Gwangui Hong and Donghwan Shin and Jimin Hwa and Doo-Hwan Bae},
journal = {International Journal of Software Engineering and Knowledge Engineering},
year = {2015}
}Allocating developers to tasks is complex on large projects, and cost-only methods ignore practical constraints. This genetic algorithm accounts for issues such as multitasking, task relevance and balanced staffing. Key implication: more realistic project staffing plans.
@inproceedings{park2014seke14,
title = {Practical Human Resource Allocation in Software Projects Using Genetic Algorithm},
author = {Jihun Park and Dongwon Seo and Gwangui Hong and Donghwan Shin and Jimin Hwa and Doo-Hwan Bae},
booktitle = {International Conference on Software Engineering and Knowledge Engineering (SEKE 2014)},
year = {2014}
}Function block diagrams are used for safety-critical PLC software such as nuclear reactor protection systems. Using 1,800 test suites and over 600 mutants of industrial FBD models, this study evaluates how well FBD coverage criteria detect faults. Key implication: it identifies the criteria's strengths and weaknesses and suggests improvements.
@inproceedings{shin2012models12,
title = {Empirical Evaluation on FBD Model-based Test Coverage Criteria using Mutation Analysis},
author = {Donghwan Shin and Eunkyoung Jee and Doo-Hwan Bae},
booktitle = {International Conference on Model Driven Engineering Languages and Systems (MODELS 2012)},
year = {2012}
}TOSEM, TSE, EMSE, CSUR, JSS, STVR; EPSRC Peer Review College