Cognitive Diagnosis Achieves Robustness Against Noise with One-Shot Neural Architecture Search

The demand for increasingly precise personalised learning within intelligent tutoring systems is driving innovation in cognitive diagnosis, the process of identifying a learner’s understanding of specific concepts. Ziwen Wang, Shangshang Yang, and Xingyi Zhang, all from the School of Computer Science and Technology at Anhui University, alongside colleagues Xiaoshan Yu and Haiping Ma, have tackled a critical limitation in current cognitive diagnosis models: their vulnerability to noisy data and reliance on manual architectural design. Their research addresses the fact that existing models often prioritise performance gains without accounting for real-world data imperfections, hindering practical application. By introducing OSCD, an evolutionary One-Shot neural architecture search method, the team demonstrates a significant step towards robust and efficient cognitive diagnosis, unlocking the potential of model architectures beyond traditional, manually designed structures. This new approach promises to improve the accuracy and reliability of assessing learner proficiency, even when faced with imperfect data.

Researchers recognised that existing CD models often struggle with noisy real-world data and rely heavily on manual architectural design, hindering their potential. To overcome these challenges, the team engineered a system capable of automatically discovering robust and effective model architectures tailored for CD tasks. The core of this work lies in a two-stage process: training and searching. During training, scientists constructed a search space encompassing a diverse range of architectural combinations, then trained a weight-sharing supernet using a complete binary tree topology.

This innovative approach allows for comprehensive exploration of potential architectures, moving beyond the constraints of manual design and significantly reducing computational cost. The searching stage formulates the optimal architecture search as a multi-objective optimization problem, specifically designed to perform well under heterogeneous noise scenarios. The team developed an optimization framework integrating a Pareto-optimal solution search strategy with cross-scenario performance evaluation, achieving robustness by identifying architectures that perform consistently well across different types of noise, including log miss, exercise confusion, Q-matrix confusion, and log flip. Extensive experiments utilising real-world educational datasets validated the effectiveness of OSCD. The discovered architectures demonstrated superior performance and robustness compared to existing CD models, particularly in the presence of noisy data. By systematically modifying a widely recognised NCD model and observing divergent performance change rates across noise environments, the research confirms that architectural design significantly impacts a model’s noise tolerance, establishing a new paradigm for CD model development.

OSCD Optimises Cognitive Diagnosis Under Noise

Scientists achieved a significant breakthrough in cognitive diagnosis (CD) through the development of OSCD, an evolutionary multi-objective One-Shot neural architecture search method. This work addresses limitations in existing intelligent tutoring systems by focusing on robustly assessing learner proficiency amidst noisy data and overcoming reliance on manual model design. The team constructed a search space encompassing diverse architectural combinations, training a weight-sharing supernet based on a complete binary tree topology to facilitate comprehensive exploration. Experiments revealed that OSCD effectively formulates optimal architecture search under heterogeneous noise scenarios as a multi-objective optimization problem.

The optimization framework integrates a Pareto-optimal solution search strategy with cross-scenario performance evaluation, allowing for robust model discovery. Researchers generated perturbed validation sets by applying distinct noise perturbations to original validation data, simulating real-world data imperfections, including partial response log miss, confusion of response exercises, confusion of the Q-matrix, and flipping of response logs. Data shows the OSCD model accurately estimates learner proficiency on specific knowledge concepts given learner-exercise response records and a predefined Q-matrix. The study defines the core cognitive diagnosis task as accurately estimating a learner’s knowledge of concepts, while the robust architecture search aims to identify models maintaining stable performance across both original and perturbed validation sets.

Measurements confirm the model accepts learner, exercise, and concept-related feature embeddings as input, outputting an estimated probability of a correct response. The research team established a general framework where the model, F, calculates ˆrij, the estimated probability of student si answering exercise ej correctly, using the equation ˆrij= F (hs, he, hc). Here, hs represents the learner-related feature embedding, he the exercise-related embedding, and hc the concept-related embedding, all within a defined embedding dimension d. Tests prove that OSCD’s two-stage process delivers optimal architectures for CD tasks, paving the way for more adaptable and reliable intelligent tutoring systems.

OSCD Evolves Robust Architectures for Cognitive Diagnosis This

This work introduces OSCD, an evolutionary multi-objective One-Shot neural architecture search framework specifically designed for cognitive diagnosis. The researchers addressed limitations in existing cognitive diagnosis models by moving beyond manually designed architectures and accounting for noise inherent in learner response data. OSCD operates in two stages, first training a comprehensive weight-sharing supernet and then searching for optimal architectures using a Pareto-optimal strategy evaluated across multiple noisy scenarios. Experiments utilising real-world educational datasets demonstrate that the architectures discovered by OSCD consistently achieve both effective and robust performance in assessing learner proficiency. The identified architectures favour operations that satisfy the Lipschitz condition, suggesting a mechanism for enhancing robustness by limiting sensitivity to input perturbations. While acknowledging the complexity of real-world data, the authors note a limitation in the scope of noise scenarios considered, suggesting future research could explore extending the search space and investigating the transferability of discovered architectures to different educational contexts and datasets.