Temporal Preference-Consistent Multi-Timescale Credit Assignment for Digital Twin-Enabled Reinforcement Learning in Smart Manufacturing

References

[1] Corallo A, Del Vecchio V, Lezzi M, Morciano P. Shop floor digital twin in smart manufacturing: a systematic literature review. Sustainability. 2021;13(23):12987.

[2] Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, et al. Human-level control through deep reinforcement learning. Nature. 2015;518:529–33.

[3] Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O. Proximal policy optimization algorithms. arXiv:1707.06347 [Preprint]. 2017.

[4] Tobin J, Fong R, Ray A, Schneider J, Zaremba W, Abbeel P. Domain randomization for transferring deep neural networks from simulation to the real world. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway (NJ): IEEE; 2017. p. 23–30.

[5] Torne M, Simeonov A, Li Z, Chan A, Chen T, et al. Reconciling reality through simulation: a real-to-sim-to-real approach for robust manipulation. arXiv:2403.03949 [Preprint]. 2024.

[6] Liu C, Xu X, Hu D. Multiobjective reinforcement learning: a comprehensive overview. IEEE Trans Syst Man Cybern Syst. 2014.

[7] Lee S, Lee M, Moon J. Weight vector selection methods by hypervolume maximization in the Pareto front for single policy multi-objective reinforcement learning. Expert Syst Appl. 2026.

[8] Yang J, Hsu C, Chen K, Hsieh P. From reward-free representations to preferences: rethinking offline preference-based reinforcement learning. In: International Conference on Machine Learning. 2026.

[9] Cheng R, Ma H, Wang W, Duan R, Liu J, et al. Inverse reinforcement learning with dynamic reward scaling for LLM alignment. In: International Conference on Learning Representations. 2026.

[10] Tanner B, Sutton RS. TD(λ) networks: temporal-difference networks with eligibility traces. In: Proceedings of the 22nd International Conference on Machine Learning. 2005.

[11] Mousavi Y, Mousavi R, Mousavi A, Tavasoli M, et al. FracTrace: long-horizon credit assignment in deep reinforcement learning via fractional eligibility traces. Neurocomputing. 2026.

[12] Li T, Liu X, Zhu F. Counterfactual generative adversarial network for data augmentation. Lect Notes Comput Sci. 2026.

[13] Uehara M, Shi C, Kallus N. A review of off-policy evaluation in reinforcement learning. Stat Sci. 2026.

[14] Altman E. Constrained Markov decision processes with total cost criteria: Lagrangian approach and dual linear program. Math Methods Oper Res. 1998.

[15] Agrawal A, Sreenath K. Discrete control barrier functions for safety-critical control of discrete systems with application to bipedal robot navigation. In: Robotics: Science and Systems. 2017.

[16] Lee D, Kang Y, Noh S. Digital twin-driven deep reinforcement learning for real-time optimisation in dynamic AGV systems. Int J Prod Res. 2026.

[17] Turgay S, Başar Ş, Ceran M, et al. A cross-method optimisation framework for automated cheese manufacturing based on digital twin-driven reinforcement learning. Int J Manag Decis Mak. 2026.

[18] Zhou S, et al. Variance-adaptive distributional soft actor-critic with expected-value targets and dual-priority replay. Neurocomputing. 2026.

@article{1724292,
      author = {Avinash Nath Tiwari},
      title = {Temporal Preference-Consistent Multi-Timescale Credit Assignment for Digital Twin-Enabled Reinforcement Learning in Smart Manufacturing},
      journal = {Iconic Research And Engineering Journals},
      year = {2026},
      volume = {10},
      number = {4},
      pages = {4259-4272},
      issn = {2456-8880},
      url = {https://www.irejournals.com/formatedpaper/1724292.pdf},
      abstract = {Reinforcement learning agents deployed in digital twin-enabled smart manufacturing environments frequently encounter non-stationary operational conditions and multi-objective trade-offs that conventional static reward formulations cannot adequately address. We propose a temporal preference-consistent multi-timescale credit assignment framework that fundamentally restructures how reward signals are generated and how policy updates are performed within the standard deep reinforcement learning loop. A recurrent preference encoder, regularized for temporal smoothness, dynamically generates context-dependent preference vectors from operational metadata such as energy prices, production queue status, and machine health indicators, thereby preventing abrupt shifts in the reward landscape. Furthermore, a drift detection mechanism monitors discrepancies between digital twin predictions and observed physical states, triggering preference re-anchoring when regime changes occur. To address delayed consequences of control actions, we employ dual eligibility trace streams that separately capture immediate actuator effects and long-term throughput or quality outcomes. Counterfactual digital twin rollouts refine advantage estimation by simulating alternative trajectories under perturbed dynamics, hence improving robustness against sim-to-real discrepancies. Additionally, a prioritized replay strategy that considers both temporal difference error and preference novelty, combined with a temporal consistency loss during mini-batch training, stabilizes policy learning across evolving operational contexts. The proposed method is expected to yield more adaptive and stable control policies in real-time manufacturing settings, particularly where energy efficiency, product quality, and throughput must be balanced under continuously changing conditions.},
      keywords = {deep reinforcement learning; digital twin; smart manufacturing; multi-objective optimization; credit assignment; eligibility traces},
      month = {October},
  }