Home
Jobs
Artmac
Senior AI Test Automation Engineer - LLM & Agentic Systems

Senior AI Test Automation Engineer - LLM & Agentic Systems

On site

San jose, United states

Senior

Freelance

06-03-2026

Share this job:

Score my CV

Skills

Python Java JavaScript Test Selenium Test Automation Product Management Machine Learning Regression AWS api testing Langchain

Job Specifications

Who We Are

Artmac Soft is a technology consulting and service-oriented IT company dedicated to providing innovative technology solutions and services to customers.

Job Description

Job Title : Senior AI Test Automation Engineer – LLM & Agentic Systems

Job Type : W2/C2C

Experience : 10-15 Years

Location : San Jose, California

Responsibilities

Minimum 8 years of experience in software QA / test engineering.
Strong understanding of testing principles: functional, regression, integration, performance.
Hands-on experience testing AI or ML systems such as LLM chatbots or recommendation engines.
Hands-on experience with AI and Machine Learning (ML) techniques to enhance test automation, such as generating test cases, predicting potential defects, or updating test code for user interface changes.
Hands-on experience with test automation frameworks and tools like Selenium, Playwright, or Cucumber using scripting languages like Python, Java, or JavaScript, and leveraging AI to improve productivity.
Familiarity with key concepts of multi-agentic systems, agentic frameworks (e.g., LangChain), and AWS AgentCore.
Scripting skills in Python and experience with REST API testing.
Design and run intelligent test strategies for LLM and agent-based systems.
Combining classic QA and test automation skills with a deep understanding of agentic automation and conversational AI,
Design and execute end-to-end test plans for UX / Uber orchestrator/ Sub Agents.
Build and maintain agentic AI test suites validating autonomous.
Develop test data and synthetic scenarios for edge cases, ambiguous queries, and multilingual requests.
Evaluate agent behavior through reasoning traces, tool use decisions, and memory usage.
Implement and improve automated testing frameworks leveraging AI for conversational flows, APIs, and integrations.
Collaborate with Product management / cross-functional teams to refine requirements and acceptance criteria.
Document defects, test results, and insights clearly for stakeholders.

Qualification

Bachelor's degree or equivalent combination of education and experience.

About the Company

Artmac provides Digital, consulting and Management IT services to clients globally - As your trusted partner, let's reimagine how your business gets done. Merging your vision and our expertise, we help you achieve powerful results.It's our mission to help our customers achieve success. That can only be accomplished through strong, lasting relationships. Artmac offers a wide array of solutions for a range of key verticals and horizontals.Starting from the strategy consulting right through to implementing IT solutions for cli... Know more