Analysis of the Timestep in Reinforcement Learning for Mobile Robots
DOI:
https://doi.org/10.17979/ja-cea.2026.47.13780Keywords:
Deep Reinforcement Learning, Timestep Duration, Mobile Robots, Sim-to-Real Gap, Asynchronous Architectures, Sample Efficiency, Temporal Dynamics, Intelligent RoboticsAbstract
The use of Deep Reinforcement Learning (DRL) in robotics has gained prominence in the last decade, but the impact of its intrinsic temporal components of DRL when applied to physical systems - generally summarized as the timestep duration -, remains underexplored. While some approaches to this issue include timestep duration as part of action space, they offer a limited analysis, overlooking key elements such as software architecture asynchrony or system-induced latencies. These factors cause deviations between nominal and actual timesteps, negatively impacting learning. This article proposes a set of navigation tasks for mobile robots specifically designed to analyze the effects of timestep duration, supported by over 2000 hours of simulation. To our knowledge, this constitutes the first multimetric evaluation of the influence of step time duration on DRL-based robotic tasks. The findings confirm the existence of optimal step times that maximize learning efficiency and task completion.
References
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., Zheng, X., 2015. TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow.org. URL: https://www.tensorflow.org/ .
Chen, X., Wang, J., 2022. Inhomogeneous deep q-network for time sensitive applications. Artificial Intelligence 312, 103757.
de Jesus, J. C., Kich, V. A., Kolling, A. H., Grando, R. B., de Souza Leite Cuadros, M. A., Gamarra, D. F. T., 2021. Soft actor-critic for navigation of mobile robots. Journal of Intelligent & Robotic Systems 102.
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., Levine, S., 2019. Soft actor-critic algorithms and applications.
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., Levine, S., 2021. How to train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Robotics Research 40, 698 – 721.
IMECH.UMA, 2025. University Research Institute in Mechatronics Engineering and Cyber-Physical Systems, University of Málaga,Spain. https:
//www.imech-uma.es/, accessed on July 14, 2025.
Kuo, P.-H., Huang, C.-T., Chang, C.-W., Feng, P.-H., Lin, Y.-S., 2025. Design and implementation of a soft actor–critic controller for a robotic arm. Engineering Applications of Artificial Intelligence 151, 110589.
Le, H., Saeedvand, S., Hsu, C.-C., 2024. A comprehensive review of mobile robot navigation using deep reinforcement learning algorithms in crowded environments. Journal of Intelligent & Robotic Systems 110 (4), 158.
Lee, J., 2013. Turtlebot 2: The new standard hardware reference platform. In: ROSCon 2013. Open Robotics. URL: http://dx.doi.org/10.36288/roscon2013-899053
Mahmood, A. R., Korenkevych, D., Komer, B., Bergstra, J., 2018. Setting up a reinforcement learning task with a real-world robot. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 4635–4640.
Puterman, M. L., 2014. Markov Decision Processes: Discrete Stochastic Dynamic Programming, 1st Edition. John Wiley & Sons.
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., Dormann, N., 2021. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research 22 (268), 1–8.
Robotics, C., 2024. Coppeliasim. https://www.coppeliarobotics.com/ , accessed: 2025-05-08.
Salvato, E., Fenu, G., Medvet, E., Pellegrino, F. A., 2021. Crossing the reality gap: a survey on sim-to-real transferability of robot controllers in reinforcement learning. IEEE Access PP, 1–1.
Towers, M., Kwiatkowski, A., Terry, J., Balis, J. U., De Cola, G., Deleu, T., Goulão, M., Kallinteris, A., Krimmel, M., KG, A., et al., 2024. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032.
UNCORE-Team, 2025. rl spin decoupler: A library for decoupling agent and environment control loops in RL. https://github.com/uncore-team/rl_spin_decoupler/tree/main , accessed: 2025-05-13.
Wang, D., Beltrame, G., 2024. Variable time step reinforcement learning for robotic applications.
Yuan, Y., Mahmood, A. R., 2022. Asynchronous reinforcement learning for real-time control of physical robots.
Zhou, H., Huang, A., Azizzadenesheli, K., Childers, D., Lipton, Z., 2024. Timing as an action: Learning when to observe and act. In: Dasgupta, S., Mandt, S., Li, Y. (Eds.), Proceedings of The 27th International Conference on Artificial Intelligence and Statistics. Vol. 238 of Proceedings of Machine Learning Research. PMLR, pp. 3979–3987.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Adrián Bañuls Arias, Juan-Antonio Fernández-Madrigal, Ana Cruz-Martín, Vicente Arévalo-Espejo, Cipriano Galindo, Juan-Manuel Gandarias
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
