Why AI in test automation must never become a black box
The IT landscape is shifting under our feet. Autonomous AI agents, code-generating LLMs and smart integrations are rapidly changing how we build and maintain applications. For testers and test automation engineers, this brings a fundamental shift. How do you make sure this acceleration doesn't come at the expense of reliability? We use AI as an accelerator, with one fixed principle: quality always comes before quantity. And quality starts with knowing exactly what the AI did.
The new reality
The impact of Artificial Intelligence on our daily work is enormous. As AI increasingly takes over application development, from requirements to code and unit tests, it's only logical that test automation moves along with it. Where we used to set up frameworks by hand, we now use AI across the board: from generating test scenarios and test data at high speed to creating the accompanying test scripts and documentation.
That doesn't release us from our critical eye. Quite the opposite. Because AI drastically increases development speed, the risk of introducing errors grows just as fast. Blindly generating large volumes of automated tests is not the answer. What matters is using AI in a targeted way and continuously testing exactly what the AI does. Our consultants understand how this technology affects the work and use it to get to the heart of quality issues faster.
Innovating by building
Staying ahead in a market that changes every week takes more than reading the occasional article. You have to build the technology yourself and master it. At Cerios, our innovation arm, Cerios Solutions, has already made significant progress here. The team continuously explores how we can apply AI smartly in our own daily work, while integrating these latest technologies directly into our own software solutions. That makes abstract technology directly and easily applicable in day-to-day practice.
Hand in hand with this product development is our active AI Community. Where Cerios Solutions pushes the boundaries of the software, our test consultants translate that into the best ways of working. The community is constantly experimenting, sparring and training. That combination is what makes the difference: we develop our own AI products, and we have the consultants in house who know how to apply this up-to-date knowledge successfully at our clients.
Traceability as the foundation: JOSF
Our low-code test automation platform JOSF proves that we don't just talk about AI, we actually build it. As AI models increasingly take over the writing of complex code, the challenge in our field shifts to maintaining oversight and validation. JOSF will act as the essential layer for AI test context, traceability and auditability: we harness the reasoning power of AI in the background, but record every step transparently. That way, the entire testing process on the front end is not only readable, but above all 100% traceable and verifiable. This keeps test automation from becoming an impenetrable black box and ensures that teams always retain insight and control.
With the new JOSF AI module, we make this theory tangible for testers and developers. Through smart LLM integrations, for example via Azure AI Foundry, or directly with OpenAI and Anthropic, JOSF changes the daily practice of test automation engineers:
Autonomous exploration
JOSF can navigate an application on its own, record actions and turn them straight into a robust test case, complete with automatically generated descriptions and documentation.
Smart composition
Test cases are assembled intelligently from reusable blocks, which keeps management and maintenance to a minimum.
Smart failure analysis
A test case fails? The AI analyzes the cause and suggests a targeted fix, which you can then implement with a single click.
Instant answers
Ask the AI directly about JOSF documentation and specific test scenarios for instant context.
This not only saves a huge amount of time when creating tests, it also produces well-documented scenarios that are easy to hand over. The next step is leaving the design of test scenarios, script generation and execution entirely to AI, with JOSF guaranteeing that every autonomous decision remains 100% traceable and accountable. Read more about the platform on the JOSF page.
Help build the future
We're only at the beginning of what AI can mean for software quality. The question isn't whether you'll use AI for test automation, but how effectively you do it.