The central claim of the paper is that the development and control of Cyber-Physical Production Systems (CPPS) requires a systematic approach to handle and include explicit ethical considerations. Since the contribution of artificial intelligence (AI) technologies, and of agent-based models in particular, was instrumental in the evolution of CPPSs, approaches of ethical AI should be endorsed in CPPS development by design. The paper discusses recent advances for ethical AI and suggests a pathway from ethical norms towards standards. As it is argued, taking the responsible AI approach is promising when tackling the main ethic-related challenges of Cyber-Physical Production Systems. We expose a number of dilemmas to be resolved so that AI systems incorporated in CPPS cause no damages either in humans, equipment or in the environment and increase the trust in the users of current and future AI technologies.